Review refactoring

README.md

@@ -2,6 +2,9 @@
 
 ![Test ElasticStack](https://github.com/NETWAYS/ansible-collection-elasticstack/actions/workflows/test_full_stack.yml/badge.svg)
 
+> [!CAUTION]
+> The current main branch has some breaking changes. Be careful using it! Everything after commit 3d6673f6d526afeb0fd7ba382d067d76bd10bbd6 is affected!
+
 This collection installs and manages the Elastic Stack. It provides roles for every component of the Stack. Furthermore, it is possible to differentiate between Enterprise or OSS releases.
 
 Every role is documented with all variables, please refer to the documentation found in **[Getting-Started](./docs/getting-started.md)**
@@ -10,6 +13,9 @@ Every role is documented with all variables, please refer to the documentation f
 
 ## Roles documentation
 
+> [!NOTE]
+> Some roles have fixed requirements that you must observe. Please have a look at the [requirements](docs/01-requirements.md) before using the collection. (There is a high probability that some of them will be refactored soon)
+
 * [Beats](docs/role-beats.md)
 * [Elasticsearch](docs/role-elasticsearch.md)
 * [Kibana](docs/role-kibana.md)

docs/01-requirements.md

@@ -0,0 +1,11 @@
+# Requirements
+
+There are some requirements that the user have to fulfill while using the collection. Some of them will be refactored and disappear from the list soon.
+
+**Inventory names**
+
+The collection provides roles for several tools (elasticsearch, kibana, ...). In some tasks hosts from the specific inventory group are fetched. The group names are customizable via variables. But using different inventory names than provided inside the variables will result in errors.
+
+**elasticstack_ca_host**
+
+This is a mandatory variable. It is used to define the host that will be used as "CA host". Per default it is already defined (first host inside the group `elasticstack_elasticsearch_group_name`). In case you are using different group names, this variable wont be set with a defualt value.

docs/role-elasticsearch.md

@@ -60,7 +60,7 @@ The following variable was only integrated to speed up upgrades of non-productio
 
 These variables are identical over all our elastic related roles, hence the different naming schemes.
 
-* *elasticstack_ca*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)
+* *elasticstack_ca_host*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)
 * *elasticstack_ca_name*: Distinguished name of the CA. (default: `CN=Elastic Certificate Tool Autogenerated CA`)
 * *elasticstack_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
 * *elasticstack_ca_validity_period*: number of days that the generated CA are valid (default: 1095).

docs/role-kibana.md

@@ -32,7 +32,7 @@ These variables are identical over all our elastic related roles, hence the diff
 * *kibana_cert_will_expire_soon*: Set it to true to renew kibana certificate (default: `false`), Or run the playbook with `--tags renew_kibana_cert` to do that.
 * *elasticstack_kibana_host*: Hostname users use to connect to Kibana (default: FQDN of the host the role is executed on)
 * *elasticstack_kibana_port*: Port Kibana webinterface is listening on (default: `5601`)
-* *elasticstack_ca*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)
+* *elasticstack_ca_host*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)
 * *elasticstack_ca_dir*: Directory where on the Elasticsearch CA host certificates are stored. This is only useful in connection with out other Elastic Stack related roles. (default: `/opt/es-ca`)
 * *elasticstack_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
 * *elasticstack_initial_passwords*: Path to file with initical elasticsearch passwords (default: `/usr/share/elasticsearch/initial_passwords`)

roles/beats/tasks/beats-security.yml

@@ -1,5 +1,4 @@
 ---
-
 - name: Ensure beats certificate exists
   ansible.builtin.stat:
     path: "/etc/beats/certs/{{ inventory_hostname }}-beats.crt"
@@ -37,7 +36,7 @@
       register: beats_check_cert_path
 
     - name: Move cert directory
       ansible.builtin.copy:
         src: /etc/beats/certs
         dest: "/etc/beats/certs_{{ ansible_date_time.iso8601_micro }}"
         mode: preserve
@@ -53,9 +52,9 @@
         state: absent
       when: beats_move_cert_directory.changed
 
-- name: Backup beats certs on elasticstack_ca host then remove
+- name: Backup beats certs on elasticstack_ca_host then remove
   when: "'renew_beats_cert' in ansible_run_tags or 'renew_ca' in ansible_run_tags or beats_cert_will_expire_soon | bool"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   tags:
     - renew_ca
     - renew_beats_cert
@@ -134,7 +133,7 @@
     --pass {{ beats_tls_key_passphrase }}
     --pem
     --out {{ elasticstack_ca_dir }}/{{ ansible_hostname }}-beats.zip
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   args:
     creates: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}-beats.zip"
   tags:
@@ -147,7 +146,7 @@
     src: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}-beats.zip"
     dest: "{{ lookup('config', 'DEFAULT_LOCAL_TMP') | dirname }}/{{ ansible_hostname }}-beats.zip"
     flat: yes
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   tags:
     - certificates
     - renew_ca
@@ -204,7 +203,7 @@
     src: "{{ elasticstack_ca_dir }}/ca.crt"
     dest: "{{ lookup('config', 'DEFAULT_LOCAL_TMP') | dirname }}/ca.crt"
     flat: yes
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   tags:
     - certificates
     - renew_ca
@@ -234,4 +233,4 @@
   register: beats_writer_password
   changed_when: false
   no_log: "{{ elasticstack_no_log }}"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"

roles/beats/tasks/main.yml

@@ -3,6 +3,7 @@
 - name: Include global role
   ansible.builtin.import_role:
     name: netways.elasticstack.elasticstack
+  when: not hostvars[inventory_hostname]._elasticstack_role_imported | default(false)
 
 - name: Update apt cache.
   ansible.builtin.apt:

roles/elasticsearch/tasks/elasticsearch-security.yml

@@ -1,28 +1,28 @@
 ---
 
 - name: Ensure ca exists
   ansible.builtin.stat:
     path: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
   register: elasticstack_ca_exists
-  when: inventory_hostname == elasticstack_ca
+  when: inventory_hostname == elasticstack_ca_host
 
 - name: Get CA informations
   cert_info:
     path: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
     passphrase: "{{ elasticstack_ca_pass | default(omit, true) }}"
   register: elasticstack_ca_infos
-  when: inventory_hostname == elasticstack_ca and elasticstack_ca_exists.stat.exists | bool
+  when: inventory_hostname == elasticstack_ca_host and elasticstack_ca_exists.stat.exists | bool
 
 - name: Set the ca expiration date in days
   ansible.builtin.set_fact:
     elasticstack_ca_expiration_days: "{{ ((elasticstack_ca_infos.not_valid_after | to_datetime()) - (ansible_date_time.date | to_datetime('%Y-%m-%d'))).days }}"
-  when: inventory_hostname == elasticstack_ca and elasticstack_ca_infos.skipped is not defined
+  when: inventory_hostname == elasticstack_ca_host and elasticstack_ca_infos.skipped is not defined
 
 - name: Set ca will expire soon to true
   ansible.builtin.set_fact:
     elasticstack_ca_will_expire_soon: true
   when: >
-    inventory_hostname == elasticstack_ca and
+    inventory_hostname == elasticstack_ca_host and
     elasticstack_ca_expiration_days is defined and
     elasticstack_ca_expiration_days | int <= elasticstack_ca_expiration_buffer | int
 
@@ -32,7 +32,7 @@
       Your ca will expire in {{ elasticstack_ca_expiration_days }} days.
       Ansible will renew it and all elastic stack certificates
   when: >
-    inventory_hostname == elasticstack_ca and
+    inventory_hostname == elasticstack_ca_host and
     elasticstack_ca_expiration_days is defined and
     elasticstack_ca_expiration_days | int <= elasticstack_ca_expiration_buffer | int
 
@@ -48,7 +48,7 @@
     - groups[elasticstack_logstash_group_name] is defined
 
 - name: Backup ca directory on elasticsearch ca host then remove
-  when: (inventory_hostname == elasticstack_ca) and ('renew_ca' in "ansible_run_tags" or elasticstack_ca_will_expire_soon | bool)
+  when: (inventory_hostname == elasticstack_ca_host) and ('renew_ca' in "ansible_run_tags" or elasticstack_ca_will_expire_soon | bool)
   tags:
     - renew_ca
   block:
@@ -149,19 +149,19 @@
         state: absent
       when: elasticsearch_move_cert_directory.changed
 
-- name: Backup elasticsearch certs on elasticstack_ca host then remove
+- name: Backup elasticsearch certs on elasticstack_ca_host host then remove
   when: "'renew_es_cert' in ansible_run_tags or 'renew_ca' in ansible_run_tags or elasticsearch_cert_will_expire_soon | bool"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   tags:
     - renew_ca
     - renew_es_cert
   block:
-    - name: Check if cert file exists on elasticstack_ca host
+    - name: Check if cert file exists on elasticstack_ca_host host
       ansible.builtin.stat:
         path: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}.p12"
       register: elasticsearch_check_cert_file
 
-    - name: Move cert file on elasticstack_ca host
+    - name: Move cert file on elasticstack_ca_host
       ansible.builtin.copy:
         src: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}.p12"
         dest: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}.p12_{{ ansible_date_time.iso8601_micro }}"
@@ -170,7 +170,7 @@
       when: elasticsearch_check_cert_file.stat.exists
       register: elasticsearch_move_cert_file
 
-    - name: Remove cert file on elasticstack_ca host
+    - name: Remove cert file on elasticstack_ca_host
       ansible.builtin.file:
         path: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}.p12"
         state: absent
@@ -217,14 +217,14 @@
 - name: Import Tasks elasticsearch-keystore.yml
   ansible.builtin.import_tasks: elasticsearch-keystore.yml
 
-- name: Create ca and certificates on elasticstack_ca host
-  when: inventory_hostname == elasticstack_ca
+- name: Create ca and certificates on elasticstack_ca_host
+  when: inventory_hostname == elasticstack_ca_host
   tags:
     - certificates
     - renew_ca
     - renew_es_cert
   block:
-    - name: Configure ca on elasticstack_ca host
+    - name: Configure ca on elasticstack_ca_host
       ansible.builtin.command: >
         /usr/share/elasticsearch/bin/elasticsearch-certutil ca
         --ca-dn {{ elasticstack_ca_name }}
@@ -236,7 +236,7 @@
         creates: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
       no_log: "{{ elasticstack_no_log }}"
 
-    - name: Create node certificates on elasticstack_ca host
+    - name: Create node certificates on elasticstack_ca_host
       ansible.builtin.command: >
         /usr/share/elasticsearch/bin/elasticsearch-certutil cert
         --ca {{ elasticstack_ca_dir }}/elastic-stack-ca.p12
@@ -266,7 +266,7 @@
     src: "{{ elasticstack_ca_dir }}/ca.crt"
     dest: "{{ lookup('config', 'DEFAULT_LOCAL_TMP') | dirname }}/ca.crt"
     flat: yes
-  when: inventory_hostname == elasticstack_ca
+  when: inventory_hostname == elasticstack_ca_host
   tags:
     - certificates
     - renew_ca
@@ -277,7 +277,7 @@
     src: "{{ elasticstack_ca_dir }}/{{ ansible_hostname }}.p12"
     dest: "{{ lookup('config', 'DEFAULT_LOCAL_TMP') | dirname }}/{{ ansible_hostname }}.p12"
     flat: yes
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   tags:
     - certificates
     - renew_ca
@@ -361,7 +361,7 @@
 - name: Check for passwords being set
   ansible.builtin.stat:
     path: "{{ elasticstack_initial_passwords }}"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   register: elasticsearch_passwords_file
 
 - name: Setting elasticsearch_http_protocol
@@ -404,7 +404,7 @@
   retries: 5
   delay: 10
 
 - name: Fetch Elastic password # noqa: risky-shell-pipe
   ansible.builtin.shell: >
     if test -n "$(ps -p $$ | grep bash)"; then set -o pipefail; fi;
     grep "PASSWORD elastic" {{ elasticstack_initial_passwords }} |
@@ -412,7 +412,7 @@
   register: elasticstack_password
   changed_when: false
   no_log: "{{ elasticstack_no_log }}"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   when: elasticsearch_passwords_file.stat.exists | bool
 
 - name: Check for API availability with elastic password
@@ -499,7 +499,7 @@
     if test -n "$(ps -p $$ | grep bash)"; then set -o pipefail; fi;
     /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto -b >
     {{ elasticstack_initial_passwords }}
-  when: inventory_hostname == elasticstack_ca
+  when: inventory_hostname == elasticstack_ca_host
   no_log: "{{ elasticstack_no_log }}"
   args:
     creates: "{{ elasticstack_initial_passwords }}"
@@ -512,6 +512,6 @@
     owner: root
     group: root
     mode: 0600
-  when: inventory_hostname == elasticstack_ca
+  when: inventory_hostname == elasticstack_ca_host
 
 # Maybe make sure that Elasticsearch is using the right protocol http(s) to connect, even in newly setup clusters

roles/elasticsearch/tasks/main.yml

@@ -10,6 +10,7 @@
 - name: Include global role
   ansible.builtin.import_role:
     name: netways.elasticstack.elasticstack
+  when: not hostvars[inventory_hostname]._elasticstack_role_imported | default(false)
 
 - name: Update apt cache.
   ansible.builtin.apt:
@@ -274,7 +275,7 @@
     - name: Check for cluster status without security
       ansible.builtin.uri:
 # kics-scan ignore-line
         url: "http://{{ elasticsearch_api_host }}:{{ elasticstack_elasticsearch_http_port }}/_cluster/health?pretty"
       register: elasticsearch_cluster_status
       ignore_errors: "{{ ansible_check_mode }}"
       until: elasticsearch_cluster_status.json.status == "green"
@@ -314,8 +315,8 @@
 
 - name: Show hint about passwords
   ansible.builtin.debug:
-    msg: "Remember, your temporary passwords can be found on {{ elasticstack_ca }} in {{ elasticstack_initial_passwords }}"
+    msg: "Remember, your temporary passwords can be found on {{ elasticstack_ca_host }} in {{ elasticstack_initial_passwords }}"
   when:
     - elasticsearch_security | bool
     - elasticstack_variant == "elastic"
-    - inventory_hostname == elasticstack_ca
+    - inventory_hostname == elasticstack_ca_host

roles/elasticstack/defaults/main.yml

@@ -5,6 +5,7 @@ elasticstack_logstash_group_name: logstash
 elasticstack_kibana_group_name: kibana
 
 elasticstack_beats_port: 5044
+elasticstack_ca_host: "{{ groups[elasticstack_elasticsearch_group_name][0] }}"
 elasticstack_ca_dir: /opt/es-ca
 elasticstack_ca_expiration_buffer: 30
 elasticstack_ca_name: "CN=Elastic Certificate Tool Autogenerated CA"

roles/elasticstack/tasks/elasticstack-passwords.yml

@@ -3,7 +3,7 @@
 - name: Check for passwords being set
   ansible.builtin.stat:
     path: "{{ elasticstack_initial_passwords }}"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   register: elasticsearch_passwords_file
 
 - name: Fetch Elastic password # noqa: risky-shell-pipe
@@ -14,5 +14,5 @@
   register: elasticstack_password
   changed_when: false
   no_log: "{{ elasticstack_no_log }}"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   when: elasticsearch_passwords_file.stat.exists | bool

roles/elasticstack/tasks/elasticstack-versions.yml

@@ -7,7 +7,7 @@
 - name: Set target version to Elasticsearch on CA host
   ansible.builtin.set_fact:
     elasticstack_version: "{{ ansible_facts.packages['elasticsearch'][0].version }}"
-  delegate_to: "{{ elasticstack_ca }}"
+  delegate_to: "{{ elasticstack_ca_host }}"
   when:
     - ansible_facts.packages['elasticsearch'][0].version is defined
     - elasticstack_version is undefined

roles/elasticstack/tasks/main.yml

@@ -1,34 +1,40 @@
 ---
+- name: Include elasticstack role
+  when: not hostvars[inventory_hostname]._elasticstack_role_imported | default(false) # Ensure the global meta role is only called once for each host
+  block:
+    - name: Include OS specific vars
+      ansible.builtin.include_vars: '{{ item }}'
+      with_first_found:
+        - '{{ ansible_os_family }}_{{ ansible_distribution_major_version }}.yml'
+        - '{{ ansible_os_family }}.yml'
 
-- name: Include OS specific vars
-  ansible.builtin.include_vars: '{{ item }}'
-  with_first_found:
-    - '{{ ansible_os_family }}_{{ ansible_distribution_major_version }}.yml'
-    - '{{ ansible_os_family }}.yml'
+    - name: Set elasticstack_ca_host variable if not already done by user
+      ansible.builtin.set_fact:
+        elasticstack_ca: "{{ groups[elasticstack_elasticsearch_group_name][0] }}"
+      when:
+        - elasticstack_ca_host is undefined
+        - groups[elasticstack_elasticsearch_group_name][0] is defined
 
-- name: Set elasticstack_ca variable if not already done by user
-  ansible.builtin.set_fact:
-    elasticstack_ca: "{{ groups[elasticstack_elasticsearch_group_name][0] }}"
-  when:
-    - elasticstack_ca is undefined
-    - groups[elasticstack_elasticsearch_group_name][0] is defined
+    - name: Set elasticstack_ca_host variable if not already set to Elasticsearch server
+      ansible.builtin.set_fact:
+        elasticstack_ca_host: "{{ groups[elasticstack_logstash_group_name][0] }}"
+      when:
+        - elasticstack_ca_host is undefined
+        - groups[elasticstack_logstash_group_name][0] is defined
 
-- name: Set elasticstack_ca variable if not already set to Elasticsearch server
-  ansible.builtin.set_fact:
-    elasticstack_ca: "{{ groups[elasticstack_logstash_group_name][0] }}"
-  when:
-    - elasticstack_ca is undefined
-    - groups[elasticstack_logstash_group_name][0] is defined
+    - name: Set versions for components
+      ansible.builtin.import_tasks: elasticstack-versions.yml
 
-- name: Set versions for components
-  ansible.builtin.import_tasks: elasticstack-versions.yml
+    - name: Fetch passwords if passwords are initialized
+      ansible.builtin.import_tasks: elasticstack-passwords.yml
 
-- name: Fetch passwords if passwords are initialized
-  ansible.builtin.import_tasks: elasticstack-passwords.yml
+    - name: Set elasticstack_globals_set for other roles to skip this role
+      ansible.builtin.set_fact:
+        elasticstack_globals_set: true
 
-- name: Set elasticstack_globals_set for other roles to skip this role
-  ansible.builtin.set_fact:
-    elasticstack_globals_set: true
+    - name: Install common packages and dependencies
+      ansible.builtin.import_tasks: packages.yml
 
-- name: Install common packages and dependencies
-  ansible.builtin.import_tasks: packages.yml
+    - name: Register elasticstack role has run for that host
+      ansible.builtin.set_fact:
+        _elasticstack_role_imported: true