diff --git a/doc/source/configuration/ci-cd.rst b/doc/source/configuration/ci-cd.rst new file mode 100644 index 000000000..6e495c2e8 --- /dev/null +++ b/doc/source/configuration/ci-cd.rst @@ -0,0 +1,183 @@ +===== +CI/CD +===== + +Concepts +======== + +The CI/CD system developed for managing Kayobe based OpenStack clouds is composed of three main components; workflows, runners and kayobe automation. +Firstly, the workflows are files which describe a series of tasks to be performed in relation to the deployed cloud. +These workflows are executed on request, on schedule or in response to an event such as a pull request being opened. +The workflows are designed to carry out various day-to-day activites such as; running Tempest tests, configuring running services or displaying the change to configuration files if a pull request is merged. +Secondly, in order for the workflows to run against a cloud we would need private runners present within the cloud positioned in such a way they can reach the internal network and public API. +Deployment of private runners is supported by all major providers with the use of community developed Ansible roles. +Finally, due to the requirement that we support various different platforms tooling in the form of `Kayobe automation `__ was developed. +This tooling is not tied to any single CI/CD platform as all tasks are a series of shell script and Ansible playbooks which are designed to run in a purpose build kayobe container. +This is complemented by the use of an Ansible collection known as `stackhpc.kayobe_workflows `__ which aims to provide users with a quick and easy way of customising all workflows to fit within a customer's cloud. + +Currently we support the creation and deployment of workflows for GitHub with Gitlab support being actively worked upon. + +Kayobe Automation +----------------- + +`Kayobe automation `__ is a collection of scripts and tools that automate kayobe operations. +It is deployed and controlled by CI/CD platforms such as GitHub actions and GitLab pipelines. +Kayobe automation provides users with an easy process to perform tasks such as: overcloud service deploy, config-diff, tempest testing, and many more. +With it being integrated into platforms such as GitHub or GitLab it builds a close relationship between the contents of the deployments kayobe configuration and what is currently deployed. +This is because operations such as opening a pull request will trigger a config diff to be generated providing insight on what impact it might have on services or a tempest test that could be scheduled to run daily providing knowledge of faults earlier than before. + +Workflows +--------- + +Kayobe automation has been designed to be independent of any CI/CD platform with all tasks running inside of a purpose built Kayobe container. +However, platform specific workflows need to be deployed to bridge the gap between the contents of Kayobe Automation and these CI/CD platforms. +Workflows are templated for each Kayobe configuration repository, ensuring appropriate workflow input parameters are defined, and any necessary customisations can be applied. +The templating of workflows is offered through the `stackhpc.kayobe_workflows `__ collection which currently supports GitHub workflows. + +Runners +------- + +Runners are purpose built services tied to a particular service vendor such as GitHub Actions or GitLab CI. +These services will listen for jobs which have been tagged appropriately and dispatched to these specific runners. +The runners will need to be deployed using existing roles and playbooks whereby the binary/package is downloaded and registered using a special token. +In some deployments runner hosts can be shared between environments however this is not always true and dedicated hosts will need to be used for each environment you intend to deploy kayobe automation within. + +GitHub Actions +================= + +To enable CI/CD where GitHub Actions is used please follow the steps described below starting with the deployment of the runners. + +Runner Deployment +----------------- + +1. Identify a suitable host for hosting the runners. + GitHub runners need to be deployed on a host which has not had Docker deployed using kolla. + This is because GitHub runners cannot provide `network options when running in a container `__. + + Ideally an Infra VM could be used here or failing that the control host. + Wherever it is deployed the host will need access to the :code:`admin_network`, :code:`public_network` and the :code:`pulp registry` on the seed. + +2. Edit the environment's :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/groups` to add the predefined :code:`github-runners` group to :code:`infra-vms` + +.. code-block:: ini + + [infra-vms:children] + github-runners + +3. Edit the environment's :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/hosts` to define the host(s) that will host the runners. + +.. code-block:: ini + + [github-runners] + prod-runner-01 + +4. Provide all the relevant Kayobe :code:`group_vars` for :code:`github-runners` under :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/group_vars/github-runners` + * `infra-vms` ensuring all required `infra_vm_extra_network_interfaces` are defined + * `network-interfaces` + * `python-interpreter.yml` ensuring that `ansible_python_interpreter: /usr/bin/python3` has been set + +5. Edit the ``${KAYOBE_CONFIG_PATH}/inventory/group_vars/github-runners/runners.yml`` file which will contain the variables required to deploy a series of runners. + Below is a core set of variables that will require consideration and modification for successful deployment of the runners. + The number of runners deployed can be configured by removing and extending the dict :code:`github-runners`. + As for how many runners present three is suitable number as this would prevent situations where long running jobs could halt progress other tasks whilst waiting for a free runner. + You might want to increase the number of runners if usage demands it or new workflows make use of multiple parallel jobs. + + Note :code:`github_registry` and the elements of the dict control the registry settings for pulling and pushing container images used by the workflows. + In the example below the registry settings have been adapted to demonstrate what a shared registry between environments might look like. + This values maybe suitable for your deployment providing all environments can reach the same registry. + If the all of the environments use their own registry and nothing is shared between them then :code:`github_registry` can omitted from the file and the template will expect environment specific secrets and variables to be added to the repository settings. + This is discussed further in the next section. + +.. code-block:: yaml + + --- + runner_user: VM_USER_NAME_HERE + github_account: ORG_NAME_HERE + github_repo: KAYOBE_CONFIG_REPO_NAME_HERE + access_token: "{{ secrets_github_access_token }}" + + default_runner_labels: + - kayobe + - openstack + - "{{ kayobe_environment | default(omit) }}" + + github_registry: + url: pulp.example.com + username: admin + password: ${{ secrets.REGISTRY_PASSWORD }} + share: true + + github_runners: + runner_01: {} + runner_02: {} + runner_03: {} + +6. Obtain a personal access token that would enable the registration of GitHub runners against the `github_account` and `github_repo` defined above. + This token ideally should be `fine-grained personal access token `__ which may require the organisation to enable such tokens beforehand. + Steps can be found `here `__. + The repository permissions for a fine-grained personal access token should be; :code:`Actions: R/W, Administration: R/W, Metadata: R` + Once the key has been obtained, add it to :code:`secrets.yml` under :code:`secrets_github_access_token` + +7. If the host is an actual Infra VM then please refer to upstream `Infrastructure VMs `__ documentation for additional configuration and steps. + +8. Run :code:`kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/deploy-github-runner.yml` + +9. Check runners have registered properly by visiting the repository's :code:`Action` tab -> :code:`Runners` -> :code:`Self-hosted runners` + +10. Repeat the above steps for each environment you intend to deploy runners within. + You can share the fine-grained access token between environments. + +Workflow Deployment +------------------- + +1. Edit :code:`${KAYOBE_CONFIG_PATH}/inventory/group_vars/github-writer/writer.yml` in the base configuration making the appropriate changes to your deployments specific needs. See documentation for `stackhpc.kayobe_workflows.github `__. + +2. Run :code:`kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/write-github-workflows.yml` + +3. Add all required secrets and variables to repository either via the GitHub UI or GitHub CLI (may require repository owner) + ++----------------------------------------------------------------------------------+ +| Secrets | ++===================================+==============================================+ +| Single Environment | Multiple Environments | ++-----------------------------------+----------------------------------------------+ +| KAYOBE_AUTOMATION_SSH_PRIVATE_KEY | _KAYOBE_AUTOMATION_SSH_PRIVATE_KEY | ++-----------------------------------+----------------------------------------------+ +| KAYOBE_VAULT_PASSWORD | _KAYOBE_VAULT_PASSWORD | ++-----------------------------------+----------------------------------------------+ +| REGISTRY_PASSWORD | _REGISTRY_PASSWORD | ++-----------------------------------+----------------------------------------------+ +| TEMPEST_OPENRC | _TEMPEST_OPENRC | ++-----------------------------------+----------------------------------------------+ + + +-------------------------------------------------------+ + | VARIABLES | + +====================+==================================+ + | Single Environment | Multiple Environments | + +--------------------+----------------------------------+ + | REGISTRY_URL | _REGISTRY_URL | + +--------------------+----------------------------------+ + | REGISTRY_USERNAME | _REGISTRY_USERNAME | + +--------------------+----------------------------------+ + +Note the above tables shows the secrets and variables one may need to add to GitHub for a successful deployment. +When adding secrets and variables make sure to adhere to the naming standards and ensure the :code:`` is replaced with all supported kayobe environments in uppercase. + +4. Commit and push all newly generated workflows found under :code:`.github/workflows` + +Final Steps +----------- + +Some final steps include the following: running config-diff will require that :code:`.automation.conf/config.sh` contains a list :code:`KAYOBE_CONFIG_VAULTED_FILES_PATHS_EXTRA` of all vaulted files contained within the config. +All such files can be found with :code:`grep -r "$ANSIBLE_VAULT;1.1;AES256" .` though make sure NOT to include `kolla/passwords.yml` and `secrets.yml` +Also make sure tempest has been configured appropriately in :code:`.automation.conf/config.sh` to meet the limitations of a given deployment such as not using a too high of :code:`TEMPEST_CONCURRENCY` value and that overrides and load/skips lists are correct. +Finally, once all the workflows and configuration has been pushed and reviewed you can build a kayobe image using the `Build Kayobe Docker Image` workflow. Once it is successfully built and pushed to a container registry, other workflows can be used. + +Sometimes the kayobe docker image must be rebuilt the reasons for this include but are not limited to the following; + + * Change :code:`$KAYOBE_CONFIG_PATH/ansible/requirements.yml` + * Change to requirements.txt + * Update Kayobe + * Update kolla-ansible + * UID/GID collision when deploying workflows to a new environment + * Prior to deployment of new a OpenStack release diff --git a/doc/source/configuration/index.rst b/doc/source/configuration/index.rst index 8c283481d..bb0e1a9fe 100644 --- a/doc/source/configuration/index.rst +++ b/doc/source/configuration/index.rst @@ -18,4 +18,5 @@ the various features provided. wazuh vault magnum-capi + ci-cd security-hardening diff --git a/doc/source/operations/index.rst b/doc/source/operations/index.rst index e3a387abd..94880ba6a 100644 --- a/doc/source/operations/index.rst +++ b/doc/source/operations/index.rst @@ -11,3 +11,4 @@ This guide is for operators of the StackHPC Kayobe configuration project. rabbitmq octavia hotfix-playbook + secret-rotation diff --git a/doc/source/operations/secret-rotation.rst b/doc/source/operations/secret-rotation.rst new file mode 100644 index 000000000..7912530fb --- /dev/null +++ b/doc/source/operations/secret-rotation.rst @@ -0,0 +1,527 @@ +=============== +Secret Rotation +=============== + +General notes +============= + +This guide covers secret rotation in Kayobe and Kolla-Ansible for most services +in a standard deployment. It does not cover every secret. A full list of +passwords that have been successfully rotated is available at the bottom of +this page (:ref:`link`). + +Many of the secrets can simply be deleted from your ``passwords.yml`` and will +be automatically regenerated with a ``kayobe overcloud service deploy``. + +Some secrets require manual input from the operator to change. + +Following this process, there may be a few seconds of network downtime for +running VMs when Neutron is reconfigured when using ML2/OVS. + +There will be API downtime for all services. The main reason for the outage is +that RabbitMQ must be completely stopped to change the secrets it uses. The +services must all be reconfigured to use the new RabbitMQ cluster. Each service +will come back once it has been reconfigured. The outage time for each service +is therefore equal to the time between starting a ``kayobe overcloud service +deploy``, and that service being reconfigured. + +Some secrets currently have to be regenerated by hand. Make sure you use a +reliable tool and match the formatting (length, character set etc) of the +existing secret. ``pwgen`` is recommended and used as an example throughout +this guide. Installation: + +.. code:: bash + + sudo apt/dnf install pwgen + + +As of writing, there are three upstream patches in the works to make this +process easier. + +#. A change to Kolla, to automate :ref:`this` step to change the + extended start for the ``nova-api`` container. + + The upstream patch can be found `here + `__. + + This was previously mitigated with a change to the StackHPC fork of + Kolla-Ansible, which has since been reverted due to an unforeseen issue. See + `here ` for more + details. + +#. A change to Nova, to automate :ref:`this` step to change the + nova cell0 database connection string. + + The upstream patch can be found `here + `__. + +#. A change to Kolla-Ansible, to automate :ref:`this` step to + update service keystone user passwords. + + The upstream patch can be found `here + `__. + + +Full method +=========== + +.. warning:: + + You **must** back up your ``passwords.yml`` before making changes. You will + need to refer back to it later + +1. Run a Tempest ``refstack`` & check Kibana/OpenSearch Dashboards to check + the state of the cloud before any changes are made + +2. Edit your Kolla-Ansible checkout to include changes not yet included + upstream. + +.. _kolla-change: + + 1. Add this line within the ``kolla_docker`` dict in + ``ansible/roles/nova/tasks/bootstrap_service.yml`` See `here + `__ + for an example. + + .. code:: + + command: bash -c 'sudo -E kolla_set_configs && nova-manage api_db sync && nova-manage db sync --local_cell' + + This change will break new deployments and should be reverted once this + process is complete + +.. _k-a-change: + + 2. Cherry-pick `this patch + `__ + + .. code:: bash + + git fetch https://review.opendev.org/openstack/kolla-ansible refs/changes/78/903178/2 && git cherry-pick FETCH_HEAD + + 3. Re-install Kolla-Ansible from source in your Kolla-Ansible Python + environment + + +3. Navigate to the directory containing your ``passwords.yml`` file + (``kayobe-config/etc/kolla/passwords.yml`` OR + ``kayobe-config/etc/kayobe/environments/envname/kolla/passwords.yml``) + +4. Create a file called ``deletelist.txt`` and populate it with this content + (including all whitespace): + + .. code:: + + _keystone_password + _database_password + ^keystone_admin_password + ^memcache_secret_key + ^designate_rndc_key + ^docker_registry_password + ^keepalived_password + ^kibana_password + ^libvirt_sasl_password + ^metadata_secret + ^opensearch_dashboards_password + ^osprofiler_secret + ^prometheus_alertmanager_password + ^qdrouterd_password + ^redis_master_password + ^memcache_secret_key + _ssh_key + + private_key + public_key + ^$ + rabbitmq + ^haproxy_password + + +5. Decrypt your ``passwords.yml`` file with ``ansible-vault`` + +6. Delete all the passwords in the deletion list + + .. code:: bash + + grep -vf deletelist.txt passwords.yml > new-passwords.yml + +7. Check the new file for basic formatting errors. If it looks correct, + replace the existing ``passwords.yml`` file with ``new-passwords.yml`` + + .. code:: bash + + rm passwords.yml && mv new-passwords.yml passwords.yml + +8. Use the ``rekey-hosts.yml`` playbook to rotate your SSH keys for hosts + across the cloud. The playbook should exist under + ``kayobe-config/etc/kayobe/ansible/`` if not, merge the latest + ``stackhpc-kayobe-config`` + + 1. Run the playbook to generate a new keypair and add it to the authorised + keys of your hosts. + + .. code:: bash + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/rekey-hosts.yml + + 2. Ensure you can SSH to other nodes using the new keypair + + 3. Re-run the playbook with arguments to remove the old keypair. + + .. code:: bash + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/rekey-hosts.yml -t remove-key -e rekey_remove_existing_key=true + +9. Update the Pulp password + + 1. Generate a new Pulp password + + .. code:: bash + + pwgen -s 25 1 + + 2. Update ``secrets_pulp_password`` (usually found in ``secrets.yml``) + + 3. Deploy changes + + .. code:: bash + + kayobe seed service deploy -t seed-deploy-containers -kt none + + (note you will need to skip Docker registry login since the password will + now be ‘incorrect’ e.g. ``-e deploy_containers_registry_attempt_login=false``) + +10. Rotate ``horizon_secret_key`` + + 1. Generate a new secret: + + .. code:: bash + + pwgen -s 40 1 + + 2. Add it to the ``passwords.yml`` file, along with the old secret, in this + exact format (including quotes in the middle): + + .. code:: bash + + horizon_secret_key: newsecret' 'oldsecret + + This will allow both the old and new secrets to be used at the same + time, resulting in no interruption to service. The key is mainly used + for generating login and password reset tokens. The old secret can be + deleted & redeployed at a later date once all users have closed & + reopened their sessions. + +11. Update ``grafana_admin_password`` + + 1. Generate a new Grafana Admin password + + .. code:: bash + + pwgen -s 40 1 + + 2. Update the value of ``grafana_admin_password`` in ``passwords.yml`` + + 3. Exec into the Grafana container on a controller + + .. code:: bash + + sudo docker exec -it grafana bash + + 4. Run the password reset command, then enter the new password + + .. code:: bash + + grafana-cli admin reset-admin-password --password-from-stdin + +12. Update the MariaDB database password + + 1. Generate a new secret: + + .. code:: bash + + pwgen -s 40 1 + + 2. Update ``database_password`` in ``passwords.yml`` with your new + password. Make a note of the old password. + + 3. Exec into the MariaDB container on a controller + + .. code:: bash + + sudo docker exec -it mariadb bash + + 4. Log in to the database. You will be prompted for the password. Use the + old value of ``database_password`` + + .. code:: bash + + mysql -uroot -p + + 5. Check the current state of the ``root`` user + + .. code:: bash + + SELECT Host,User,Password FROM mysql.user WHERE User='root'; + + 6. Update the password for the ``root`` user + + .. code:: bash + + SET PASSWORD FOR 'root'@'%' = PASSWORD('newpassword'); + + 7. Check that the password hash has changed in the user list + + .. code:: bash + + SELECT Host,User,Password FROM mysql.user WHERE User='root'; + + 8. If there are any remaining root users with the old password e.g. + ``root@localhost``, change the password for them too + +.. _nova-change: + +13. Update the Nova Database password + + .. warning:: + + From this point onward, service may be disrupted + + #. Create a new ``nova_database_password`` and store it in + ``passwords.yml`` + + .. code:: bash + + pwgen -s 40 1 + + #. Exec into the ``nova_conductor`` container + + .. code:: bash + + sudo docker exec -it nova_conductor bash + + #. List the cells + + .. code:: bash + + nova-manage cell_v2 list_cells --verbose + + #. Find the entry for ``cell0``, copy the Database Connection value, + replace the password in the string with the new value, and update it + with the following command: + + .. code:: bash + + nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection "CONNECTION WITH NEW PASSWORD HERE" --transport-url "none:///" + + (If the ``cell_uuid`` for cell0 is not + ``00000000-0000-0000-0000-000000000000``, change the above command + accordingly) + +14. Re-encrypt your ``passwords.yml`` file + +15. Stop all OpenStack services + + .. code:: bash + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/stop-openstack-services.yml + +16. Flush the Memcached data on all controllers (any old data will now be + inaccessible) + + #. Install Telnet (on one of the controllers) + + .. code:: bash + + sudo apt -y install telnet + + #. Check the config for the IP and port used by Memcached (on every + controller) + + .. code:: bash + + sudo grep command /etc/kolla/memcached/config.json + + The IP and port will be printed after ``-l`` and ``-p`` respectively + + #. For each controller start a Telnet session, clear all data, then + exit + + .. code:: bash + + telnet + flush_all + quit + +17. Nuke RabbitMQ + + .. code:: bash + + kayobe overcloud host command run -l controllers --become --command "docker stop rabbitmq && docker rm rabbitmq && docker volume rm rabbitmq" + +19. Reconfigure Overcloud services to apply changes + + .. warning:: + + VMs should continue running, but connections to them will briefly be + disrupted when Neutron is redeployed when using ML2/OVS + + .. code:: bash + + kayobe overcloud service deploy + +20. Manually update ``heat_domain_admin_password`` + + #. TODO: Instructions + This has not been tested yet + +21. Re-run Tempest to make sure everything has come back + +22. Inform other users of the steps they’ll need to take now that the secrets + have been rotated: + + 1. SSH keys have been rotated, so the new key will have to be distributed + if individual user accounts are used + + 2. Any existing ``openrc`` files generated by Kolla Ansible will need to be + re-generated or edited to use the new Keystone admin password + +23. Create a PR to merge the new secrets into your main Kayobe configuration + branch + + .. warning:: + + Unless you **really** enjoyed this process, RE-ENCRYPT + ``passwords.yml`` BEFORE COMMITTING + +24. Approximately 1 week after deploying, remove the old horizon secret key + from ``passwords.yml`` and reconfigure horizon + + +.. _full-password-list: + +Full password list +------------------- + +:: + + aodh_database_password + aodh_keystone_password + blazar_database_password + blazar_keystone_password + caso_keystone_password + ceilometer_database_password + ceilometer_keystone_password + cinder_database_password + cinder_keystone_password + barbican_database_password + barbican_keystone_password + cloudkitty_database_password + cloudkitty_keystone_password + congress_database_password + congress_keystone_password + cyborg_database_password + cyborg_keystone_password + designate_database_password + designate_keystone_password + freezer_database_password + freezer_keystone_password + glance_database_password + glance_keystone_password + gnocchi_database_password + gnocchi_keystone_password + heat_database_password + heat_keystone_password + horizon_database_password + ironic_database_password + ironic_inspector_database_password + ironic_inspector_keystone_password + ironic_keystone_password + karbor_database_password + karbor_keystone_password + keystone_database_password + magnum_database_password + manila_database_password + mariadb_backup_database_password + masakari_database_password + mistral_database_password + monasca_database_password + murano_database_password + neutron_database_password + nova_api_database_password + nova_database_password + octavia_database_password + panko_database_password + placement_database_password + prometheus_mysql_exporter_database_password + qinling_database_password + rally_database_password + sahara_database_password + senlin_database_password + solum_database_password + tacker_database_password + trove_database_password + vitrage_database_password + watcher_database_password + zun_database_password + keystone_admin_password + kuryr_keystone_password + magnum_keystone_password + manila_keystone_password + masakari_keystone_password + mistral_keystone_password + monasca_keystone_password + murano_keystone_password + neutron_keystone_password + nova_keystone_password + octavia_keystone_password + panko_keystone_password + rabbitmq_cluster_cookie + rabbitmq_monitoring_password + rabbitmq_password + database_password + heat_domain_admin_password + horizon_secret_key + placement_keystone_password + qinling_keystone_password + sahara_keystone_password + searchlight_keystone_password + senlin_keystone_password + solum_keystone_password + swift_keystone_password + tacker_keystone_password + trove_keystone_password + vitrage_keystone_password + watcher_keystone_password + zun_keystone_password + ceph_rgw_keystone_password + designate_rndc_key + keepalived_password + kibana_password + libvirt_sasl_password + metadata_secret + opensearch_dashboards_password + osprofiler_secret + prometheus_alertmanager_password + qdrouterd_password + grafana_admin_password + docker_registry_password + secrets_pulp_password + redis_master_password + haproxy_password + keystone_ssh_key + private_key + public_key + neutron_ssh_key + private_key + public_key + nova_ssh_key + private_key + public_key + octavia_amp_ssh_key + private_key + public_key + bifrost_ssh_key + private_key + public_key + diff --git a/etc/kayobe/ansible/deploy-github-runner.yml b/etc/kayobe/ansible/deploy-github-runner.yml new file mode 100644 index 000000000..47c8211a1 --- /dev/null +++ b/etc/kayobe/ansible/deploy-github-runner.yml @@ -0,0 +1,34 @@ +--- +- name: Deploy GitHub Runner + hosts: github-runners + become: yes + roles: + - role: geerlingguy.pip + - role: geerlingguy.docker + tasks: + - name: Deploy runners + ansible.builtin.include_role: + role: monolithprojects.github_actions_runner + vars: + runner_name: "{{ ansible_facts.nodename }}-{{ runner.key }}" + runner_dir: "{{ base_runner_dir }}/{{ runner.key }}" + runner_labels: "{{ runner.value.labels | default(default_runner_labels) }}" + runner_state: "{{ runner.value.state | default('started') }}" + with_dict: + "{{ github_runners }}" + loop_control: + loop_var: runner + + # FIXME: Sometimes the runner service is not running at the end of the role. + # Start the service manually. + - name: Ensure runner service is running + ansible.builtin.service: + name: actions.runner.{{ github_account }}-{{ github_repo }}.{{ ansible_facts.nodename }}-{{ runner.key }}.service + state: started + enabled: true + become: true + when: runner_state | default('started') == 'started' + with_dict: + "{{ github_runners }}" + loop_control: + loop_var: runner diff --git a/etc/kayobe/ansible/requirements.yml b/etc/kayobe/ansible/requirements.yml index bfb2d51b9..fd685f1dc 100644 --- a/etc/kayobe/ansible/requirements.yml +++ b/etc/kayobe/ansible/requirements.yml @@ -10,6 +10,8 @@ collections: version: 0.5.2 - name: stackhpc.hashicorp version: 2.4.0 + - name: stackhpc.kayobe_workflows + version: 1.0.2 roles: - src: stackhpc.vxlan - name: ansible-lockdown.ubuntu22_cis @@ -25,3 +27,9 @@ roles: - name: wazuh-ansible src: https://github.com/stackhpc/wazuh-ansible version: stackhpc + - name: geerlingguy.pip + version: 2.2.0 + - name: monolithprojects.github_actions_runner + version: 1.18.5 + - src: https://github.com/stackhpc/ansible-role-docker.git + name: geerlingguy.docker diff --git a/etc/kayobe/ansible/stop-openstack-services.yml b/etc/kayobe/ansible/stop-openstack-services.yml new file mode 100644 index 000000000..08f4e964a --- /dev/null +++ b/etc/kayobe/ansible/stop-openstack-services.yml @@ -0,0 +1,29 @@ +--- +# Stops containers running OpenStack services + +- name: Stop OpenStack services + hosts: overcloud + become: true + gather_facts: false + vars: + - stop_service_list: + - "blazar" + - "barbican" + - "cinder" + - "cloudkitty" + - "designate" + - "glance" + - "heat" + - "horizon" + - "ironic" + - "keystone" + - "magnum" + - "manila" + - "neutron" + - "nova" + - "octavia" + - "placement" + tasks: + - name: Stop OpenStack services + shell: >- + docker ps -a | egrep '({{ stop_service_list | join('|') }})' | awk '{ print $NF }' | xargs docker stop diff --git a/etc/kayobe/ansible/wazuh-manager.yml b/etc/kayobe/ansible/wazuh-manager.yml index fedd1ad93..6865d3b6f 100644 --- a/etc/kayobe/ansible/wazuh-manager.yml +++ b/etc/kayobe/ansible/wazuh-manager.yml @@ -41,7 +41,9 @@ dest: "/var/ossec/etc/shared/default/" owner: wazuh group: wazuh - when: custom_sca_policies.files | length > 0 + when: + - custom_sca_policies_folder.stat.exists + - custom_sca_policies.files | length > 0 - name: Add custom policy definition(s) to the shared Agent config blockinfile: @@ -61,7 +63,9 @@ {% endfilter %} - when: custom_sca_policies.files | length > 0 + when: + - custom_sca_policies_folder.stat.exists + - custom_sca_policies.files | length > 0 notify: - Restart wazuh diff --git a/etc/kayobe/ansible/write-github-workflows.yml b/etc/kayobe/ansible/write-github-workflows.yml new file mode 100644 index 000000000..f45319833 --- /dev/null +++ b/etc/kayobe/ansible/write-github-workflows.yml @@ -0,0 +1,7 @@ +--- +- name: Write Kayobe Automation Workflows for GitHub + hosts: github-writer + vars: + github_output_directory: "{{ kayobe_config_path }}/../../.github/workflows" + roles: + - stackhpc.kayobe_workflows.github diff --git a/etc/kayobe/inventory/group_vars/github-runners/runners.yml b/etc/kayobe/inventory/group_vars/github-runners/runners.yml new file mode 100644 index 000000000..1d2d5235e --- /dev/null +++ b/etc/kayobe/inventory/group_vars/github-runners/runners.yml @@ -0,0 +1,38 @@ +--- +runner_user: "{{ kayobe_ansible_user }}" +github_account: "{{ undef(hint='Name of customer's GitHub account') }}" +github_repo: "{{ undef(hint='Name of customer's kayobe config repository') }}" +access_token: "{{ secrets_github_access_token }}" + +base_runner_dir: /opt/actions-runner + +default_runner_labels: + - kayobe + - openstack + - "{{ kayobe_environment }}" + +# Dictionary of GitHub runners to be deployed to each runner host. +# Each dict item can be provided with optional attributes +# * labels - provide a list of labels for a specific runner +# overriding the contents of `default_runner_labels` +# * state - either `started`` or `absent`. By default it will +# be started if however the runner needs to be removed +# then setting it to `absent` will unregister the runner with +# GitHub and remove it from the system. +# Example +# github_runners: +# runner_01: {} +# runner_02: +# labels: ['foo', 'bar', 'baz'] +# runner_03: +# state: absent +github_runners: + runner_01: {} + runner_02: {} + runner_03: {} + +docker_users: + - "{{ runner_user }}" + +pip_install_packages: + - name: docker diff --git a/etc/kayobe/inventory/group_vars/github-writer/writer.yml b/etc/kayobe/inventory/group_vars/github-writer/writer.yml new file mode 100644 index 000000000..4c536087f --- /dev/null +++ b/etc/kayobe/inventory/group_vars/github-writer/writer.yml @@ -0,0 +1,8 @@ +--- +# Configuration of GitHub workflows generated with stackhpc.kayobe_workflows.github should go here. +# See documentation for more information +# https://github.com/stackhpc/ansible-collection-kayobe-workflows/blob/main/roles/github/README.md + +############################################################################### +# Dummy variable to allow Ansible to accept this file. +workaround_ansible_issue_8743: yes diff --git a/etc/kayobe/inventory/groups b/etc/kayobe/inventory/groups index fa0a55a1c..d368b1bb2 100644 --- a/etc/kayobe/inventory/groups +++ b/etc/kayobe/inventory/groups @@ -31,6 +31,12 @@ seed-hypervisor seed overcloud +[github-runners] +# Empty group to provide declaration of github-runner group. + +[github-writer] +localhost + ############################################################################### # Overcloud groups. diff --git a/etc/kayobe/kolla/config/haproxy/services.d/os_exporter.cfg b/etc/kayobe/kolla/config/haproxy/services.d/os_exporter.cfg index 1292f38bd..e40c27a38 100644 --- a/etc/kayobe/kolla/config/haproxy/services.d/os_exporter.cfg +++ b/etc/kayobe/kolla/config/haproxy/services.d/os_exporter.cfg @@ -1,3 +1,4 @@ +{% if stackhpc_enable_os_capacity | bool %} {% raw %} frontend os_capacity_frontend mode http @@ -17,3 +18,4 @@ backend os_capacity_backend server {{ host_name }} {{ host_ip }}:9000 check inter 2000 rise 2 fall 5 {% endfor %} {% endraw %} +{% endif %} diff --git a/etc/kayobe/kolla/config/prometheus/prometheus.yml.d/70-oscapacity.yml.j2 b/etc/kayobe/kolla/config/prometheus/prometheus.yml.d/70-oscapacity.yml similarity index 73% rename from etc/kayobe/kolla/config/prometheus/prometheus.yml.d/70-oscapacity.yml.j2 rename to etc/kayobe/kolla/config/prometheus/prometheus.yml.d/70-oscapacity.yml index 1c0c25c15..659c26047 100644 --- a/etc/kayobe/kolla/config/prometheus/prometheus.yml.d/70-oscapacity.yml.j2 +++ b/etc/kayobe/kolla/config/prometheus/prometheus.yml.d/70-oscapacity.yml @@ -1,3 +1,6 @@ +# yamllint disable-file +--- +{% if stackhpc_enable_os_capacity | bool %} {% raw %} scrape_configs: - job_name: os-capacity @@ -7,3 +10,4 @@ scrape_configs: scrape_interval: 15m scrape_timeout: 10m {% endraw %} +{% endif %} diff --git a/etc/kayobe/stackhpc-monitoring.yml b/etc/kayobe/stackhpc-monitoring.yml index 43f1f309f..b48646e79 100644 --- a/etc/kayobe/stackhpc-monitoring.yml +++ b/etc/kayobe/stackhpc-monitoring.yml @@ -9,3 +9,9 @@ alertmanager_low_memory_threshold_gib: 5 ############################################################################### +# Exporter configuration + +# Whether the OpenStack Capacity exporter is enabled. +# Enabling this flag will result in HAProxy configuration and Prometheus scrape +# targets being templated during deployment. +stackhpc_enable_os_capacity: false diff --git a/releasenotes/notes/add-github-kayobe-automation-support-33dc85bbc8746d82.yaml b/releasenotes/notes/add-github-kayobe-automation-support-33dc85bbc8746d82.yaml new file mode 100644 index 000000000..e18490003 --- /dev/null +++ b/releasenotes/notes/add-github-kayobe-automation-support-33dc85bbc8746d82.yaml @@ -0,0 +1,9 @@ +--- +features: + - | + Adds support for deploying GitHub runners and creating GitHub workflows + for use within Kayobe Automation. Two playbooks and their requirements + have been added to `ansible/` in addition to the relevant groups defined + with some useful default variables where appropriate. Finally, + documentation has been added to cover how to deploy these runners and + workflows. diff --git a/releasenotes/notes/stop-services-playbook-b85b53d1a7571009.yaml b/releasenotes/notes/stop-services-playbook-b85b53d1a7571009.yaml new file mode 100644 index 000000000..6516f8ec4 --- /dev/null +++ b/releasenotes/notes/stop-services-playbook-b85b53d1a7571009.yaml @@ -0,0 +1,5 @@ +--- +features: + - | + Added the ``stop-openstack-services.yml`` playbook, which can be used to + stop OpenStack services across the overcloud. diff --git a/tox.ini b/tox.ini index 19b554ab4..c11c6c5d9 100644 --- a/tox.ini +++ b/tox.ini @@ -13,8 +13,8 @@ deps = commands = yamllint etc/kayobe reno lint - doc8 README.rst doc/source --ignore D001 - + # secret-rotation must be skipped because it includes purposeful whitespace + doc8 README.rst doc/source --ignore D001 --ignore-path-errors doc/source/operations/secret-rotation.rst;D002 # StackHPC Kayobe configuration release notes: [testenv:releasenotes] allowlist_externals = rm