diff --git a/.ansible-lint b/.ansible-lint index 9ce20e42..743bf739 100644 --- a/.ansible-lint +++ b/.ansible-lint @@ -2,12 +2,14 @@ # Collection wide lint-file # DO NOT CHANGE exclude_paths: + - .ansible/ - .cache/ - .github/ #- docs/ + - changelogs/ # Changelog files are missing '---' required in normal yml files. + - roles/sap_vm_preconfigure # Role is WIP + # TODO: Remove when ansible-lint issues are resolved (Issue #101). - roles/sap_hypervisor_node_preconfigure - #- roles/sap_vm_provision - - roles/sap_vm_preconfigure enable_list: - yaml diff --git a/.gitignore b/.gitignore index 3469fb09..b2722776 100644 --- a/.gitignore +++ b/.gitignore @@ -62,3 +62,6 @@ __pycache__/ *.tfstate *.tfstate.* .terraform.lock.hcl + +# Ignore ansible workspace +.ansible diff --git a/CHANGELOG.rst b/CHANGELOG.rst index abaffc59..12f07399 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -4,6 +4,22 @@ community.sap\_infrastructure Release Notes .. contents:: Topics +v1.2.0 +====== + +Release Summary +--------------- + +Various enhancements and readme update + +Minor Changes +-------- + +- collection - Readme update to align with project readme update (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/104) +- sap_hypervisor_node_preconfigure - Readme update to align with project readme update (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/109) +- sap_hypervisor_node_preconfigure - Updated redhat_ocpv (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/106) +- sap_vm_provision - Updated kubevirt_vm (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/100) + v1.1.3 ====== diff --git a/README.md b/README.md index 3fb6980d..7df9c436 100644 --- a/README.md +++ b/README.md @@ -2,36 +2,128 @@ ![Ansible Lint](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint.yml/badge.svg?branch=main) -This Ansible Collection executes various SAP Infrastructure related tasks, creating resources needed for hosts of SAP Systems. +## Description +This Ansible Collection provides a set of Ansible Roles designed to automate various infrastructure-related tasks for SAP systems. It focuses on creating and configuring the necessary resources on different infrastructure platforms, including cloud hyperscalers and hypervisors. -These Ansible Roles are often run first and combined with other Ansible Collections to provide end-to-end automation. +These roles are typically used as a foundational step in end-to-end automation workflows, often in conjunction with other Ansible Collections that handle higher-level configurations, such as SAP application deployments. -Various Infrastructure Platforms (Cloud Hyperscalers and Hypervisors) are compatible and tested with this Ansible Collection. +The included roles cover a range of tasks, such as: +- Provisioning Virtual Machines on target infrastructure platforms, using `Ansible` or `Terraform`. + - This also includes provisioning of High Availability resources (Routing, Load Balancers, etc.), where applicable. +- Assigning temporary Virtual IP Addresses for application installation, before they are managed by a cluster. +- Pre-configuring hypervisor nodes for hosting virtual machines for SAP systems. +- Pre-configuring virtual machines (`Work in Progress`). +- Verifying provisioned virtual machines (`Work in Progress`). -**Please read the [full documentation](./docs#readme) for how-to guidance, requirements, and all other details. Summary documentation is below:** +## Requirements +**Please read the detailed documentation for each Ansible Role to understand their specific requirements.** +Always follow official [Ansible Documentation](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix) for compatibility matrix between Control and Managed nodes. -## Contents +### Control Nodes +Supported Operating systems: +- Any operating system with required Python and Ansible versions. + +Component versions: +| Component | Version | +| --- | --- | +| Python | 3.11 or higher | +| ansible-core | 2.16 or higher | + +**NOTE:** We recommend using the latest version of components.
+Each minor version of `ansible-core` can bring Security fixes (CVE) that can affect functionality. Examples: +- `CVE-2023-5764` changed `assert` functionality in `2.14.12`, `2.15.8` and `2.16.1`. +- `CVE-2024-11079` changed `hostvars` functionality in `2.16.14`, `2.17.7` and `2.18.1`. + +### Managed Nodes +Supported Operating systems: +- SUSE Linux Enterprise Server for SAP applications (SLE4SAP): 15 SP5-SP7 and 16 +- Red Hat Enterprise Linux for SAP Solutions (RHEL4SAP): 8.x, 9.x and 10.x + +**NOTE: Operating system needs to have access to required package repositories either directly or via a subscription registration.** + +Component versions: +| Component | Version | +| --- | --- | +| Python | 3.6 or higher | + + +## Installation Instructions + +### Installation +Install this collection with Ansible Galaxy command: +```console +ansible-galaxy collection install community.sap_infrastructure +``` + +Optionally you can include collection in requirements.yml file and include it together with other collections using: `ansible-galaxy collection install -r requirements.yml`.
+**NOTE: This is not recommended for this collection, because you will need only specific subset of collections for your chosen Infrastructure Platform.**
+ +Requirements file need to be maintained in following format: +```yaml +collections: + - name: community.sap_infrastructure +``` + +### Upgrade +Installed Ansible Collection will not be upgraded automatically when Ansible package is upgraded. + +To upgrade the collection to the latest available version, run the following command: +```console +ansible-galaxy collection install community.sap_infrastructure --upgrade +``` + +You can also install a specific version of the collection if you encounter issues with the latest version. Please report such issues in the affected Role repository. +For example, to install version 1.1.0: +``` +ansible-galaxy collection install community.sap_infrastructure:==1.1.0 +``` + +See [Installing collections](https://docs.ansible.com/ansible/latest/collections_guide/collections_installing.html) for more details on installation methods. -Within this Ansible Collection, there are various Ansible Roles and no custom Ansible Modules. ### Ansible Roles +All included roles can be executed independently or as part of [ansible.playbooks_for_sap](https://github.com/sap-linuxlab/ansible.playbooks_for_sap) playbooks. | Name | Summary | | :--- | :--- | -| [sap_hypervisor_node_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_hypervisor_node_preconfigure)
`Beta` | Vendor-specific configuration preparation tasks for Hypervisor nodes hosting Virtual Machines running SAP Systems | -| ~~[sap_vm_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_preconfigure)~~
`WIP` | ~~Vendor-specific configuration preparation tasks for Virtual Machines running SAP Systems~~ | -| [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) | Provision Virtual Machines to different Infrastructure Platforms; with optional Ansible to Terraform to provision minimal landing zone (partial compatibility via [Terraform Modules for SAP](https://github.com/sap-linuxlab/terraform.modules_for_sap)) | -| [sap_vm_temp_vip](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_temp_vip)
`Beta` | Temporary Virtual IP (VIP) assigned to OS Network Interface prior to Linux Pacemaker ownership | -| ~~[sap_vm_verify](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_verify)~~
`WIP` | ~~Verification of Virtual Machine state and readiness to perform SAP Software installation~~ | +| [sap_hypervisor_node_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_hypervisor_node_preconfigure)`Beta` | Vendor-specific configuration preparation tasks for Hypervisor nodes hosting Virtual Machines running SAP Systems | +| ~~[sap_vm_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_preconfigure)~~`WIP` | ~~Vendor-specific configuration preparation tasks for Virtual Machines running SAP Systems~~ | +| [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) | Provision Virtual Machines to different Infrastructure Platforms; with optional Ansible to Terraform to provision minimal landing zone. | +| [sap_vm_temp_vip](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_temp_vip)
| Temporary Virtual IP (VIP) assigned to OS Network Interface prior to Linux Pacemaker ownership | +| ~~[sap_vm_verify](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_verify)~~ `WIP` | ~~Verification of Virtual Machine state and readiness to perform SAP Software installation~~ | -## License +## Testing +This Ansible Collection has been tested across different operating systems, SAP products, and scenarios. + +Prior to each release, basic scenarios are executed to confirm functionality is working as expected, including SAP S/4HANA installation. + +**NOTE: It is not possible for the project maintainers to test every combination of Infrastructure Platform, Operating System and SAP Software for every release.** + -- [Apache 2.0](./LICENSE) +## Contributing +For information on how to contribute, please see our [contribution guidelines](https://sap-linuxlab.github.io/initiative_contributions/). ## Contributors +You can find list of Contributors at [/docs/contributors](./docs/CONTRIBUTORS.md). + + +## Support +You can report any issues using [GitHub Issues](https://github.com/sap-linuxlab/community.sap_infrastructure/issues). + -Contributors to the Ansible Roles within this Ansible Collection, are shown within [/docs/contributors](./docs/CONTRIBUTORS.md). +## Release Notes and Roadmap +The release notes for this collection can be found in the [CHANGELOG file](https://github.com/sap-linuxlab/community.sap_infrastructure/blob/main/CHANGELOG.rst). + + +## Further Information + +### Variable Precedence Rules +Please follow [Ansible Precedence guidelines](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable) on how to pass variables when using this collection. + + +## License +[Apache 2.0](https://github.com/sap-linuxlab/community.sap_infrastructure/blob/main/LICENSE) diff --git a/changelogs/changelog.yaml b/changelogs/changelog.yaml index d32c794f..c27eea6a 100644 --- a/changelogs/changelog.yaml +++ b/changelogs/changelog.yaml @@ -90,3 +90,13 @@ releases: release_summary: Bastion SSH Proxy boolean bugfixes: - sap_vm_provision - bastion boolean, to allow Bastion SSH Proxy or direct (e.g. VPN) for all Infrastructure Platforms (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/93) + + 1.2.0: + release_date: '2025-08-12' + changes: + release_summary: Various enhancements and readme update + minor_changes: + - collection - Readme update to align with project readme update (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/104) + - sap_hypervisor_node_preconfigure - Readme update to align with project readme update (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/109) + - sap_hypervisor_node_preconfigure - Updated redhat_ocpv (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/106) + - sap_vm_provision - Updated kubevirt_vm (https://github.com/sap-linuxlab/community.sap_infrastructure/pull/100) diff --git a/docs/CONTRIBUTORS.md b/docs/CONTRIBUTORS.md index d4e76f06..f3a3c18d 100644 --- a/docs/CONTRIBUTORS.md +++ b/docs/CONTRIBUTORS.md @@ -5,8 +5,12 @@ - **Sean Freeman** - Developer of Ansible Collection and sap_vm_provision Ansible Role - **Red Hat** - Red Hat for SAP CoP - - **Janine Fuchs** - Developer of Ansible parallelisation and OVirt capability for sap_vm_provision Ansible Role + - **Janine Fuchs** - Developer of Ansible parallelization and OVirt capability for sap_vm_provision Ansible Role - **Nils Koenig** - Developer of sap_hypervisor_node_preconfigure and KubeVirt capability for sap_vm_provision Ansible Role +- **SUSE** + - SUSE SAP Emerging Technology Solutions + - **Marcel Mamula** - Developer of Ansible Collection + # New contributors diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index e1c0c9a7..00000000 --- a/docs/README.md +++ /dev/null @@ -1,100 +0,0 @@ -# Documentation of community.sap_infrastructure Ansible Collection - -## Introduction - -The `sap_infrastructure` Ansible Collection executes various SAP Infrastructure related tasks, creating resources needed for hosts of SAP Systems. - -These Ansible Roles are often run first and combined with other Ansible Collections to provide end-to-end automation. - - -## Functionality - -This Ansible Collection provides a variety of tasks related to SAP Infrastructure (networks, storage, compute). The code structure and logic has been separated to support a flexible execution of different steps for various Infrastructure Platforms and hosting options. - -At a high-level, the key functionality of this Ansible Collection includes: - -- Preconfigure Hypervisor nodes ready to host Virtual Machines running SAP Systems -- Preconfigure Virtual Machines with specific tasks for the Infrastructure Platform -- Provision Virtual Machines - - on target Infrastructure Platform, using Ansible or Ansible to Terraform (to perform minimal landing zone setup of an Infrastructure Platform) - - with High Availability resources if required for the Infrastructure Platform (e.g. Routing and Load Balancers on Cloud Hyperscalers) -- Assignment of Temporary Virtual IP required for High Availability installations on selected Infrastructure Platforms - - -Compatibility is available within the Ansible Collection for various Infrastructure Platforms: - -- Cloud Hyperscalers - AWS EC2 VS, GCP CE VM, IBM Cloud VS, IBM Power VS from IBM Cloud, MS Azure VM -- Hypervisors - IBM PowerVM VM, OVirt VM, KubeVirt VM, VMware VM - - -## Execution - -An Ansible Playbook is the file created and executed by an end-user, which imports from Ansible Collections to perform various activities on the target hosts. - -The Ansible Playbook can call either an Ansible Role, or directly call the individual Ansible Modules: - -- **Ansible Roles** (runs multiple Ansible Modules) -- **Ansible Modules** (and adjoining Python/Bash Functions) - -It is strongly recommended to execute these Ansible Roles in accordance to best practice Ansible usage, where an Ansible Playbook is executed from a host and Ansible will login to a target host to perform the activities. - -> If an Ansible Playbook is executed from the target host itself (similar to logging in and running a shell script), this is known as an Ansible Playbook 'localhost execution' and is not recommended as it has limitations on SAP Software installations (particularly installations across multiple hosts). - -At a high-level, complex executions with various interlinked activities are run in parallel or sequentially using the following execution structure: - -``` -Ansible Playbook --> source Ansible Collection --> execute Ansible Task ----> run Ansible Role ------> run Ansible Module (e.g. built-in Ansible Module for Shell) -``` - -### Execution examples - -There are various methods to execute the Ansible Collection, dependent on the use case. - -For more information, see [sample Ansible Playbooks in `/playbooks`](../playbooks/). - - -## Requirements and Dependencies - -### Execution/Controller host - Operating System requirements - -Execution of Ansible Playbooks using this Ansible Collection have been tested with: -- Python 3.9.7 and above (i.e. CPython distribution) -- Ansible Core 2.12.0 and above _(included with optional installation of Ansible Community Edition 5.0 and above)_ -- OS: macOS with Homebrew, RHEL, SLES, and containers in Task Runners (e.g. Azure DevOps) - -#### Ansible Core version - -This Ansible Collection was designed for maximum backwards compatibility, with full compatibility starting from Ansible Core 2.12.0 and above. - -**Note 1:** Ansible 2.9 was the last release before the Ansible project was split into Ansible Core and Ansible Community Edition, and was before Ansible Collections functionality was introduced. This Ansible Collection should execute when Ansible 2.9 is used, but it is not recommended and errors should be expected (and will not be resolved). - -**Note 2:** Ansible Core versions prior to 2.14.12 , 2.15.8 , and 2.16.1 where `CVE-2023-5764` (templating inside `that` statement of `assert` Ansible Tasks) security fix was addressed, will work after `v1.3.4` of this Ansible Collection. Otherwise an error similar to the following will occur: - -```yaml -fatal: [host01]: FAILED! => - msg: 'The conditional check ''13 <= 128'' failed. The error was: Conditional is marked as unsafe, and cannot be evaluated.' -``` - - -## Testing - -Various Infrastructure Platforms and SAP Software solutions have been extensively tested. - -Prior to each release, basic scenarios are executed to confirm functionality is working as expected; including SAP S/4HANA installation. - -Important note: it is not possible for the project maintainers to test every Infrastructure Platform setup and all SAP Software for each OS, if an error is identified please raise a [GitHub Issue](/../../issues/). - - -### Ansible Roles Lint Status - -| Role Name | Ansible Lint Status | -| :--- | :--- | -| [sap_hypervisor_node_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_hypervisor_node_preconfigure) | [![Ansible Lint for sap_hypervisor_node_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_hypervisor_node_preconfigure.yml/badge.svg)](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_hypervisor_node_preconfigure.yml) | -| [sap_vm_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_preconfigure) | [![Ansible Lint for sap_vm_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_preconfigure.yml/badge.svg)](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_preconfigure.yml) | -| [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) | [![Ansible Lint for sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_provision.yml/badge.svg)](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_provision.yml) | -| [sap_vm_temp_vip](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_temp_vip) | [![Ansible Lint for sap_vm_temp_vip](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_temp_vip.yml/badge.svg)](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_temp_vip.yml) | -| [sap_vm_verify](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_verify) | [![Ansible Lint for sap_vm_verify](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_verify.yml/badge.svg)](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_verify.yml) | diff --git a/galaxy.yml b/galaxy.yml index 736e8eec..d925f9f4 100644 --- a/galaxy.yml +++ b/galaxy.yml @@ -10,7 +10,7 @@ namespace: community name: sap_infrastructure # The version of the collection. Must be compatible with semantic versioning -version: 1.1.3 +version: 1.2.0 # The path to the Markdown (.md) readme file. This path is relative to the root of the collection readme: README.md diff --git a/playbooks/sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml b/playbooks/sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml index 5f1d0bb5..c3f0b2b5 100644 --- a/playbooks/sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml +++ b/playbooks/sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml @@ -6,13 +6,21 @@ sap_hypervisor_node_preconfigure_platform: redhat_ocp_virt tasks: - - name: Use kubeconfig file specified in environment variable K8S_AUTH_KUBECONFIG if sap_hypervisor_node_preconfigure_ocp_kubeconfig_path is not defined + - name: Use kubeconfig file specified in environment variable K8S_AUTH_KUBECONFIG | KUBECONFIG if sap_hypervisor_node_preconfigure_ocp_kubeconfig_path is not defined. when: > - sap_hypervisor_node_preconfigure_ocp_kubeconfig_path is not defined or - sap_hypervisor_node_preconfigure_ocp_kubeconfig_path == None or - sap_hypervisor_node_preconfigure_ocp_kubeconfig_path == '' + sap_hypervisor_node_preconfigure_kubeconfig is not defined or + sap_hypervisor_node_preconfigure_kubeconfig == None or + sap_hypervisor_node_preconfigure_kubeconfig == '' ansible.builtin.set_fact: - sap_hypervisor_node_preconfigure_ocp_kubeconfig_path: "{{ lookup('env', 'K8S_AUTH_KUBECONFIG') | default(None) }}" + sap_hypervisor_node_preconfigure_kubeconfig: "{{ lookup('env', 'K8S_AUTH_KUBECONFIG') | default(lookup('env', 'KUBECONFIG'), true) }}" + + - name: Ensure that kubeconfig is set + ansible.builtin.assert: + that: + - sap_hypervisor_node_preconfigure_kubeconfig is defined + - sap_hypervisor_node_preconfigure_kubeconfig is not none + - sap_hypervisor_node_preconfigure_kubeconfig | length > 0 + fail_msg: "sap_hypervisor_node_preconfigure_kubeconfig is required." - name: Create Tempdir on jumphost ansible.builtin.tempfile: @@ -20,77 +28,12 @@ suffix: "_sap_hypervisor_node_preconfigure" register: __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost - - name: Set kubeconfig file variable - ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_kubeconfig: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost.path }}/kubeconfig" - - - name: Read content of kubeconfig file - ansible.builtin.set_fact: - sap_hypervisor_node_preconfigure_ocp_kubeconfig_data: - "{{ lookup('file', sap_hypervisor_node_preconfigure_ocp_kubeconfig_path) | from_yaml }}" - - - name: Read cluster endpoint and CA certificate from kubeconfig if either is not defined - when: sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig - block: - - - name: Set sap_hypervisor_node_preconfigure_ocp_endpoint from kubeconfig - ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_ocp_endpoint: - "{{ sap_hypervisor_node_preconfigure_ocp_kubeconfig_data['clusters'][0]['cluster']['server'] }}" - - - name: Write the certificate-authority-data to temp dir on jumphost - ansible.builtin.copy: - content: "{{ sap_hypervisor_node_preconfigure_ocp_kubeconfig_data['clusters'][0]['cluster']['certificate-authority-data'] | b64decode }}" - dest: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost.path }}/cluster-ca-cert.pem" - mode: "0666" - - - name: Set CA file variable - ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_ca_cert: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost.path }}/cluster-ca-cert.pem" - - - - name: Use predefined CA cert and API endpoint - when: not sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig - - block: - - name: Set predefined OCP API Endpoint - ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_ocp_endpoint: "{{ sap_hypervisor_node_preconfigure_ocp_endpoint }}" - - - name: Set predefined CA file - ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_ca_cert: "{{ sap_hypervisor_node_preconfigure_ocp_ca_cert }}" - - - - name: Log into Red Hat OpenShift cluster (obtain access token) - community.okd.openshift_auth: - host: "{{ __sap_hypervisor_node_preconfigure_register_ocp_endpoint }}" - username: "{{ sap_hypervisor_node_preconfigure_ocp_admin_username }}" - password: "{{ sap_hypervisor_node_preconfigure_ocp_admin_password }}" - ca_cert: "{{ __sap_hypervisor_node_preconfigure_register_ca_cert }}" - register: __sap_vm_provision_register_ocp_auth_results - - - name: Set token in kubeconfig - ansible.builtin.set_fact: - sap_hypervisor_node_preconfigure_ocp_kubeconfig_data: >- - {{ - sap_hypervisor_node_preconfigure_ocp_kubeconfig_data | combine({ - 'users': sap_hypervisor_node_preconfigure_ocp_kubeconfig_data.users | map('combine', [{'user': {'token': __sap_vm_provision_register_ocp_auth_results.openshift_auth.api_key }}] ) - }, recursive=True) - }} - - - name: Write the updated kubeconfig - ansible.builtin.copy: - content: "{{ sap_hypervisor_node_preconfigure_ocp_kubeconfig_data | to_nice_yaml }}" - dest: "{{ __sap_hypervisor_node_preconfigure_register_kubeconfig }}" - mode: "0600" - - name: Invoke role with credentials set as environment variables delegate_to: "{{ inventory_hostname }}" delegate_facts: true environment: - KUBECONFIG: "{{ __sap_hypervisor_node_preconfigure_register_kubeconfig }}" - K8S_AUTH_KUBECONFIG: "{{ __sap_hypervisor_node_preconfigure_register_kubeconfig }}" + KUBECONFIG: "{{ sap_hypervisor_node_preconfigure_kubeconfig }}" + K8S_AUTH_KUBECONFIG: "{{ sap_hypervisor_node_preconfigure_kubeconfig }}" block: - name: Include sap_hypervisor_node_preconfigure Ansible Role diff --git a/playbooks/sample-sap-vm-provision-redhat-ocpv.yml b/playbooks/sample-sap-vm-provision-redhat-ocpv.yml index 279bc203..3425e4a1 100644 --- a/playbooks/sample-sap-vm-provision-redhat-ocpv.yml +++ b/playbooks/sample-sap-vm-provision-redhat-ocpv.yml @@ -8,25 +8,18 @@ sap_vm_provision_iac_platform: kubevirt_vm pre_tasks: # Alternative to executing ansible-playbook with -e for Ansible Extravars file -# - name: Include sample variables for Red Hat Openshift Virtualization -# ansible.builtin.include_vars: ./vars/sample-variables-sap-vm-provision-redhat-ocpv.yml + # - name: Include sample variables for Red Hat Openshift Virtualization + # ansible.builtin.include_vars: ./vars/sample-variables-sap-vm-provision-redhat-ocpv.yml tasks: - name: Save inventory_host as execution_host ansible.builtin.set_fact: sap_vm_provision_execution_host: "{{ inventory_hostname }}" + when: sap_vm_provision_execution_host is not defined - name: Save ansible_user as execution_host user ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_execution_host_user: "{{ ansible_user }}" - - - name: Use kubeconfig file specified in environment variable K8S_AUTH_KUBECONFIG if sap_vm_provision_kubevirt_vm_kubeconfig_path is not defined - when: > - sap_vm_provision_kubevirt_vm_kubeconfig_path is not defined or - sap_vm_provision_kubevirt_vm_kubeconfig_path == None or - sap_vm_provision_kubevirt_vm_kubeconfig_path == '' - ansible.builtin.set_fact: - sap_vm_provision_kubevirt_vm_kubeconfig_path: "{{ lookup('env', 'K8S_AUTH_KUBECONFIG') | default(None) }}" + __sap_vm_provision_kubevirt_vm_register_execution_host_user: "{{ ansible_user | default(lookup('env', 'USER')) }}" - name: Create Tempdir ansible.builtin.tempfile: @@ -34,83 +27,40 @@ suffix: "_sap_vm_provision_kubevirt_vm" register: __sap_vm_provision_kubevirt_vm_register_tmpdir - - name: Set kubeconfig file variable - ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_kubeconfig: "{{ __sap_vm_provision_kubevirt_vm_register_tmpdir.path }}/kubeconfig" - - - name: Read content of kubeconfig file - ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_kubeconfig_data: - "{{ lookup('file', sap_vm_provision_kubevirt_vm_kubeconfig_path) | from_yaml }}" - - - name: Read cluster endpoint and CA certificate from kubeconfig if either is not defined - when: sap_vm_provision_kubevirt_vm_extract_kubeconfig - block: - - - name: Set sap_vm_provision_kubevirt_vm_api_endpoint from kubeconfig - ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_api_endpoint: - "{{ __sap_vm_provision_kubevirt_vm_register_kubeconfig_data['clusters'][0]['cluster']['server'] }}" - - - name: Write the certificate-authority-data to temp dir - ansible.builtin.copy: - content: "{{ __sap_vm_provision_kubevirt_vm_register_kubeconfig_data['clusters'][0]['cluster']['certificate-authority-data'] | b64decode }}" - dest: "{{ __sap_vm_provision_kubevirt_vm_register_tmpdir.path }}/cluster-ca-cert.pem" - mode: "0600" - - - name: Set CA file variable - ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_ca_cert: "{{ __sap_vm_provision_kubevirt_vm_register_tmpdir.path }}/cluster-ca-cert.pem" - - - name: Use predefined CA cert and API endpoint - when: not sap_vm_provision_kubevirt_vm_extract_kubeconfig - block: - - name: Set predefined OCP API Endpoint - ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_api_endpoint: "{{ sap_vm_provision_kubevirt_vm_api_endpoint }}" - - - name: Set predefined CA file - ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_ca_cert: "{{ sap_vm_provision_kubevirt_vm_ca_cert }}" - - - name: Log into Red Hat OpenShift cluster (obtain access token) - community.okd.openshift_auth: - host: "{{ __sap_vm_provision_kubevirt_vm_register_api_endpoint }}" - username: "{{ sap_vm_provision_kubevirt_vm_admin_username }}" - password: "{{ sap_vm_provision_kubevirt_vm_admin_password }}" - ca_cert: "{{ __sap_vm_provision_kubevirt_vm_register_ca_cert }}" - register: __sap_vm_provision_kubevirt_vm_register_kubevirt_vm_auth_results - - - name: Set token in kubeconfig + - name: Use kubeconfig file specified in environment variable K8S_AUTH_KUBECONFIG | KUBECONFIG if sap_vm_provision_kubevirt_vm_kubeconfig_path is not defined + when: > + sap_vm_provision_kubevirt_vm_kubeconfig is not defined or + sap_vm_provision_kubevirt_vm_kubeconfig == None or + sap_vm_provision_kubevirt_vm_kubeconfig == '' ansible.builtin.set_fact: - __sap_vm_provision_kubevirt_vm_register_kubeconfig_data: >- - {{ - __sap_vm_provision_kubevirt_vm_register_kubeconfig_data | combine({ - 'users': __sap_vm_provision_kubevirt_vm_register_kubeconfig_data.users | map('combine', [{'user': {'token': __sap_vm_provision_kubevirt_vm_register_kubevirt_vm_auth_results.openshift_auth.api_key }}] ) - }, recursive=True) - }} + sap_vm_provision_kubevirt_vm_kubeconfig: "{{ lookup('env', 'K8S_AUTH_KUBECONFIG') | default(lookup('env', 'KUBECONFIG'), true) }}" - - name: Write the updated kubeconfig - ansible.builtin.copy: - content: "{{ __sap_vm_provision_kubevirt_vm_register_kubeconfig_data | to_nice_yaml }}" - dest: "{{ __sap_vm_provision_kubevirt_vm_register_kubeconfig }}" - mode: "0600" + - name: Ensure that kubeconfig is set + ansible.builtin.assert: + that: + - sap_vm_provision_kubevirt_vm_kubeconfig is defined + - sap_vm_provision_kubevirt_vm_kubeconfig is not none + - sap_vm_provision_kubevirt_vm_kubeconfig | length > 0 + fail_msg: "sap_vm_provision_kubevirt_vm_kubeconfig is required." - - name: Create dynamic inventory group for Ansible Role sap_vm_provision and provide execution_host and api token + - name: Create dynamic inventory group for Ansible Role sap_vm_provision and provide configuration such as execution_host, kubeconfig, etc. ansible.builtin.add_host: name: "{{ item }}" group: sap_vm_provision_target_inventory_group + sap_vm_provision_iac_type: ansible + sap_vm_provision_iac_platform: kubevirt_vm sap_vm_provision_execution_host: "{{ sap_vm_provision_execution_host }}" __sap_vm_provision_kubevirt_vm_register_execution_host_user: "{{ __sap_vm_provision_kubevirt_vm_register_execution_host_user }}" __sap_vm_provision_kubevirt_vm_register_tmpdir: "{{ __sap_vm_provision_kubevirt_vm_register_tmpdir }}" - __sap_vm_provision_kubevirt_vm_register_kubeconfig: "{{ __sap_vm_provision_kubevirt_vm_register_kubeconfig }}" + sap_vm_provision_kubevirt_vm_kubeconfig: "{{ sap_vm_provision_kubevirt_vm_kubeconfig }}" loop: "{{ sap_vm_provision_kubevirt_vm_host_specifications_dictionary[sap_vm_provision_host_specification_plan].keys() }}" - name: Ansible Play to provision VMs for SAP hosts: sap_vm_provision_target_inventory_group # Ansible Play target hosts pattern, use Inventory Group created by previous Ansible Task (add_host) gather_facts: false environment: - K8S_AUTH_KUBECONFIG: "{{ __sap_vm_provision_kubevirt_vm_register_kubeconfig }}" + K8S_AUTH_KUBECONFIG: "{{ sap_vm_provision_kubevirt_vm_kubeconfig }}" + KUBECONFIG: "{{ sap_vm_provision_kubevirt_vm_kubeconfig }}" tasks: - name: Execute Ansible Role sap_vm_provision diff --git a/playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml b/playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml index 57995756..73fc7cbb 100644 --- a/playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml +++ b/playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml @@ -6,28 +6,11 @@ # Red Hat OpenShift cluster connection details ########################################################### -# Admin username for Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_admin_username: - -# Admin password for Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_admin_password: - -# Path to kubeconfig file Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_kubeconfig_path: - -# If this is set to true, the API endpoint and the -# CA Certificate are extracted from the kubeconfig file. -# If set to false, sap_hypervisor_node_preconfigure_ocp_endpoint and -# sap_hypervisor_node_preconfigure_ocp_ca_cert have to be specified. -sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig: true - -# URL to the API endpoint of Red Hat OpenShift cluster -#sap_hypervisor_node_preconfigure_ocp_endpoint: - -# CA Certificate for Red Hat OpenShift cluster connection -# To extract the CA Cert from the kubeconfig, you can use -# grep certificate-authority-data ${KUBECONFIG} | awk '{ print $2 }' | base64 --decode > client-cert.pem -#sap_hypervisor_node_preconfigure_ocp_ca_cert: +# kubeconfig for Red Hat OpenShift cluster connection. +# Needs to contain a valid API token for trident storage operator to work. +# If not provided, the kubeconfig will be read from the environment variables +# KUBECONFIG or K8S_AUTH_KUBECONFIG +# sap_hypervisor_node_preconfigure_kubeconfig: ########################################################### @@ -47,12 +30,27 @@ sap_hypervisor_node_preconfigure_install_operators: true sap_hypervisor_node_preconfigure_setup_worker_nodes: true +################################################################################### +# Individual worker nodes config details, typically no need to change anything here. +# Requires sap_hypervisor_node_preconfigure_setup_worker_nodes to be set to true. +################################################################################### + +# Should the network of the nodes be configured? +sap_hypervisor_node_preconfigure_configure_worker_node_network: true + +# Should the cpumanager be enabled? This has to be set to true. +sap_hypervisor_node_preconfigure_enable_cpumanager: true + +# Should the kargs be applied to the worker nodes? +sap_hypervisor_node_preconfigure_enable_kargs: true + + ########################################################### # Configuration details ########################################################### # URL of the trident installer package to use -sap_hypervisor_node_preconfigure_install_trident_url: https://github.com/NetApp/trident/releases/download/v24.06.0/trident-installer-24.06.0.tar.gz +sap_hypervisor_node_preconfigure_install_trident_url: https://github.com/NetApp/trident/releases/download/v25.02.1/trident-installer-25.02.1.tar.gz # Allow unsupported NICs to be used for SR-IOV? sap_hypervisor_node_preconfigure_sriov_enable_unsupported_nics: false @@ -72,6 +70,12 @@ sap_hypervisor_node_preconfigure_ocpv_namespace: openshift-cnv # Channel to be used for the Red Hat Openshift Virtualization Operator sap_hypervisor_node_preconfigure_ocpv_subscription_channel: stable +# Namespace to be used for the Red Hat Openshift Nmstate Operator +sap_hypervisor_node_preconfigure_nmstate_namespace: openshift-nmstate + +# Mountpoint for hostpath provisioner +sap_hypervisor_node_preconfigure_hpp_mountpoint: /var/localstorage + ########################################################### # Red Hat OpenShift cluster configuration details @@ -96,6 +100,10 @@ sap_hypervisor_node_preconfigure_cluster_config: password: xxxxx storage_driver: ontap-nas storage_prefix: ocpv_sap_ + nfs_mount_options: nfsvers=3 + export_policy: default + storageclass_name: sapstorage # Name of the storage class (careful, there can only be one SC with this name) + default_storageclass: true # Should this storage class be default? # CPU cores which will be reserved for kubernetes worker_kubernetes_reserved_cpus: "0,1" @@ -103,6 +111,12 @@ sap_hypervisor_node_preconfigure_cluster_config: # Storage device used for host path provisioner as local storage. worker_localstorage_device: /dev/vdb + # Name of the storage class for locastorage + worker_localstorage_storageclass_name: sapstorage + + # Should the local storage class be default? + worker_localstorage_default_storageclass: true + # detailed configuration for every worker that should be configured workers: @@ -124,39 +138,41 @@ sap_hypervisor_node_preconfigure_cluster_config: port: - name: ens1f0 # network IF name - - name: storage # an SR-IOV device - interface: ens2f0 # network IF name - type: sriov - - - name: ens2f0 # Set elevated MTU of 9000 - type: ethernet # on parent interface of - state: up # storagebridge - ipv4: - dhcp: false - enabled: false - mtu: 9000 - - - name: storagebridge # using a bridge - bridge: # another bridge - options: - stp: - enabled: false - port: - - name: ens2f0 # network IF name - description: storage - mtu: 9000 - ipv4: - address: - - ip: 192.168.1.10 # IP config - prefix-length: 24 - auto-dns: false - auto-gateway: false - state: up - type: linux-bridge - - - name: multi # another SR-IOV device - interface: ens2f1 # network IF name - type: sriov +# Additional network interfaces, just shown here for reference +# - name: storage # an SR-IOV device +# interface: ens2f0 # network IF name +# type: sriov + +# - name: ens2f0 # Set elevated MTU of 9000 +# type: ethernet # on parent interface of +# state: up # storagebridge +# ipv4: +# dhcp: false +# enabled: false +# mtu: 9000 + +# - name: storagebridge # using a bridge +# bridge: # another bridge +# options: +# stp: +# enabled: false +# port: +# - name: ens2f0 # network IF name +# description: storage +# mtu: 9000 +# ipv4: +# address: +# - ip: 192.168.1.10 # IP config +# prefix-length: 24 +# auto-dns: false +# auto-gateway: false +# enabled: true +# state: up +# type: linux-bridge +# +# - name: multi # another SR-IOV device +# interface: ens2f1 # network IF name +# type: sriov - name: worker-1 # second worker configuration @@ -177,36 +193,38 @@ sap_hypervisor_node_preconfigure_cluster_config: port: - name: ens1f0 # network IF name - - name: storage # an SR-IOV device - interface: ens2f0 # network IF name - type: sriov - - - name: ens2f0 # Set elevated MTU of 9000 - type: ethernet # on parent interface of - state: up # storagebridge - ipv4: - dhcp: false - enabled: false - mtu: 9000 - - - name: storagebridge # create storage bridge - bridge: - options: - stp: - enabled: false - port: - - name: ens2f0 # network IF name - description: storage - mtu: 9000 - ipv4: - address: - - ip: 192.168.1.11 # IP config - prefix-length: 24 - auto-dns: false - auto-gateway: false - state: up - type: linux-bridge - - - name: multi # another SR-IOV device - interface: ens2f1 # network IF name - type: sriov +# Additional network interfaces, just shown here for reference +# - name: storage # an SR-IOV device +# interface: ens2f0 # network IF name +# type: sriov +# +# - name: ens2f0 # Set elevated MTU of 9000 +# type: ethernet # on parent interface of +# state: up # storagebridge +# ipv4: +# dhcp: false +# enabled: false +# mtu: 9000 +# +# - name: storagebridge # create storage bridge +# bridge: +# options: +# stp: +# enabled: false +# port: +# - name: ens2f0 # network IF name +# description: storage +# mtu: 9000 +# ipv4: +# address: +# - ip: 192.168.1.11 # IP config +# prefix-length: 24 +# auto-dns: false +# auto-gateway: false +# enabled: true +# state: up +# type: linux-bridge +# +# - name: multi # another SR-IOV device +# interface: ens2f1 # network IF name +# type: sriov diff --git a/playbooks/vars/sample-variables-sap-vm-provision-redhat-ocpv.yml b/playbooks/vars/sample-variables-sap-vm-provision-redhat-ocpv.yml index 0dd8a395..4bc07a10 100644 --- a/playbooks/vars/sample-variables-sap-vm-provision-redhat-ocpv.yml +++ b/playbooks/vars/sample-variables-sap-vm-provision-redhat-ocpv.yml @@ -3,6 +3,11 @@ # Red Hat OpenShift Virtualization # ############################################ +# kubeconfig for Red Hat OpenShift cluster connection. +# If not provided, the kubeconfig will be read from the environment variables +# KUBECONFIG or K8S_AUTH_KUBECONFIG +# sap_vm_provision_kubevirt_vm_kubeconfig: /path/to/clusterconfigs/kubeconfig + # Namespace where the VM should be created in sap_vm_provision_kubevirt_vm_target_namespace: sap @@ -12,50 +17,18 @@ sap_vm_provision_kubevirt_vm_os_user: cloud-user # Password for the above user sap_vm_provision_kubevirt_vm_os_user_password: "" -# how to authenticate to the guest vm [password|private_key|private_key_data] -# password: uses provided password in sap_vm_provision_kubevirt_vm_os_user_password, make sure your ssh config allows password authentication -# private_key: use the private ssh key at the location defined by sap_vm_provision_ssh_host_private_key_file_path -# private_key_data: use the private ssh key provided in sap_vm_provision_ssh_host_private_key_data and write it to the location defined in sap_vm_provision_ssh_host_private_key_file_path -sap_vm_provision_kubevirt_vm_guest_ssh_auth_mechanism: private-key - -# Private SSH key file, must be accessible on the ansible controller -# sap_vm_provision_ssh_host_private_key_file_path: - -# private ssh key, make sure the indentation is correct, here it's two spaces at the beginning of every line -# sap_vm_provision_ssh_host_private_key_data: | -# < your key data> - -# Should the CA cert and the API endpoint be extracted from the kubeconfig file? -sap_vm_provision_kubevirt_vm_extract_kubeconfig: true - -# Should an existing VM be overwritten? -sap_vm_provision_kubevirt_vm_overwrite_vm: false - -# Kubeconfig file for cluster where VMs should be created -sap_vm_provision_kubevirt_vm_kubeconfig_path: /path/to/clusterconfigs/kubeconfig - -# In order to use secured communication, provide the CA cert bundle for the cluster. -# This can be extracted from the kubeconfig file with the following command from the -# kubeconfig file: -# grep certificate-authority-data ${KUBECONFIG} | awk '{ print $2 }' | base64 --decode > cluster-ca-cert.pem -# This variable will not be used if sap_vm_provision_kubevirt_vm_extract_kubeconfig = true -# sap_vm_provision_kubevirt_vm_ca_cert: /path/to/clusterconfigs/cluster-ca-cert.pem - -# API endpoint of the cluster -# This variable will not be used if sap_vm_provision_kubevirt_vm_extract_kubeconfig = true -# sap_vm_provision_kubevirt_vm_api_endpoint: https://api.cluster.domain.tld:6443 - -# Admin username for the cluster communication -sap_vm_provision_kubevirt_vm_admin_username: kubeadmin - -# Password for the above admin user -sap_vm_provision_kubevirt_vm_admin_password: AAAAA-BBBBB-CCCCC-DDDDD +# SSH key files, must be accessible on the ansible controller +sap_vm_provision_ssh_host_private_key_file_path: /path/to/id_rsa +sap_vm_provision_ssh_host_public_key_file_path: /path/to/id_rsa.pub # RAM Overhead [GiB] for virt-launcher container, this can be small for VMs < 1 TB and without SRIOV but should be increased to 16 or more for VMs > 1TB sap_vm_provision_kubevirt_vm_container_memory_overhead: 1 # hostname of the ansible controller -sap_vm_provision_kubevirt_vm_ansible_controller: localhost # on AAP, this is localhost +sap_vm_provision_execution_host: localhost # on AAP, this is localhost + +# What's the host specification plan that should be rolled out? +sap_vm_provision_host_specification_plan: example_host_specification_plan sap_vm_provision_kubevirt_vm_host_specifications_dictionary: example_host_specification_plan: @@ -69,11 +42,17 @@ sap_vm_provision_kubevirt_vm_host_specifications_dictionary: # Provide either an existing PVC or a URL for an OS image os_image: # either url or source_pvc_name have to be provided # URL for an image to be used - url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.8.0" + #url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.8.0" + #url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.10.0" + #url: "docker://registry.redhat.io/rhel9/rhel-guest-image:9.4" + url: "docker://registry.redhat.io/rhel9/rhel-guest-image:9.6" + #url: "docker://registry.redhat.io/rhel10/rhel-guest-image:10.0" # Name for a PVC to be cloned # source_pvc_name: "rhel-8.8" namespace: openshift-virtualization-os-images size: "50Gi" + access_modes: # e.g. ReadWriteMany | ReadWriteOnce + - ReadWriteMany network_definition: - name: sapbridge type: bridge @@ -84,21 +63,22 @@ sap_vm_provision_kubevirt_vm_host_specifications_dictionary: mountpoint: /hana disk_count: 1 # default: 1 disk_size: 2048 # size in GB, integer - disk_type: nas # KubeVirt Storage Class + disk_type: sapstorage # KubeVirt Storage Class + access_modes: # e.g. ReadWriteMany | ReadWriteOnce + - ReadWriteMany cloudinit: userData: |- #cloud-config timezone: Europe/Berlin - hostname: "{{ scaleout_origin_host_spec }}" + hostname: host1 user: {{ sap_vm_provision_kubevirt_vm_os_user if sap_vm_provision_kubevirt_vm_os_user is defined }} password: {{ sap_vm_provision_kubevirt_vm_os_user_password if sap_vm_provision_kubevirt_vm_os_user_password is defined }} chpasswd: expire: false ssh_authorized_keys: - - "{{ lookup('ansible.builtin.file', sap_vm_provision_ssh_host_public_key_file_path ) }}" + - "{{ lookup('ansible.builtin.file', sap_vm_provision_ssh_host_public_key_file_path ) if sap_vm_provision_ssh_host_public_key_file_path is defined }}" networkData: |- - network: - version: 2 - ethernets: - eth0: - dhcp4: true + version: 2 + ethernets: + eth0: + dhcp4: true diff --git a/roles/sap_hypervisor_node_preconfigure/README.md b/roles/sap_hypervisor_node_preconfigure/README.md index 4091eb4d..c76131a9 100644 --- a/roles/sap_hypervisor_node_preconfigure/README.md +++ b/roles/sap_hypervisor_node_preconfigure/README.md @@ -1,274 +1,144 @@ `Beta` + # sap_hypervisor_node_preconfigure + +![Ansible Lint for sap_hypervisor_node_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_hypervisor_node_preconfigure.yml/badge.svg) -Ansible Role for configuration of Hypervisor Nodes and Control Plane for hosting Virtual Machines with SAP Systems. +## Description + +The Ansible Role `sap_hypervisor_node_preconfigure` configures hypervisor nodes and the control plane for hosting virtual machines with SAP systems. -This Ansible Role can configure the following hypervisors in order to run SAP workloads: -- Red Hat OpenShift Virtualization (OCPV). The corresponding upstream project KubeVirt is not tested with this role. While this might work, there is no guarantee. -- Red Hat Enterprise Virtualization (RHV). The corresponding upstream project OVirt KVM is not tested with this role. While this might work, there is no guarantee. +This Ansible role supports the following hypervisors: +- Red Hat OpenShift Virtualization (OCPV): 4.14 or higher +- Red Hat Enterprise Virtualization (RHV): 4.4 or higher + - _Formerly called Red Hat Enterprise Virtualization (RHEV) prior to version 4.4_ + - _Note: This role is not compatible with standalone RHEL KVM (RHEL-KVM) hypervisor nodes._ -## Functionality +The hypervisor nodes for virtual machines hosting SAP software are configured by this role according to SAP Notes and best practices defined jointly by the hypervisor vendor and SAP. -The hypervisor nodes for Virtual Machines hosting SAP Software are amended by the Ansible Role according to SAP Notes and best practices defined by jointly by the Hypervisor vendor and SAP. The majority of these alterations are to improve the performance of SAP Software with the Virtual Machine and the Hypervisor. +The majority of these alterations are intended to improve the performance of SAP software on the virtual machine and the hypervisor. + + + -## Scope - -All hosts for SAP Software running one of the following hypervisors. - -**Hypervisor Versions** -- Red Hat OpenShift Virtualization (OCPV) version 4.14+ -- Red Hat Virtualization (RHV) version 4.4+ (Extended Support until 1H-2026) - - Contains 'Red Hat Virtualization Manager (RHV-M)' and the 'Red Hat Virtualization Host (RHV-H)' hypervisor nodes that this Ansible Role preconfigures - - _Formerly called Red Hat Enterprise Virtualization (RHEV) prior to version 4.4_ - - _Not to be confused with standalone RHEL KVM (RHEL-KVM) hypervisor nodes, which this Ansible Role is not compatible with_ + + ## Execution + + -### Sample execution - -For further information, see the [sample Ansible Playbooks in `/playbooks`](../playbooks/). For example: - -```shell -ansible-playbook --connection=local -i "localhost," \ -./playbooks/sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml \ --e @./playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml -``` - -### Suggested execution sequence +### Execution Flow + +1. Assert that required inputs have been provided. +2. Load hypervisor-specific variables depending on the value of `sap_hypervisor_node_preconfigure_platform`. +3. Configure the selected hypervisor. +4. Adjust performance configuration, if applicable. + -Prior to execution of this Ansible Role, there are no Ansible Roles suggested to be executed first. +### Example + +The platform specific playbook examples are available in their dedicated sections below. + -### Summary of execution flow -- Execute with specified Hypervisor platform using variable `sap_hypervisor_node_preconfigure_platform` -- Import default variables from `/vars` for specified Hypervisor platform -- Re-configure specified Hypervisor platform -- Append performance configuration for specified Hypervisor platform - -### Tags to control execution - -There are no tags used to control the execution of this Ansible Role - -## Platform: Red Hat OpenShift Virtualization - -Configure a plain vanilla Red Hat OpenShift cluster so it can be used for SAP workloads. +## Platform: Red Hat OpenShift Virtualization (redhat_ocp_virt) +Configures the Red Hat OpenShift cluster for SAP workloads. ### Requirements -- Jumphost which can access the Red Hat OpenShift cluster -- Optional: Ansible Automation Platform Controller can be used to facilitate the orchestration -- Red Hat OpenShift cluster: - - Cluster without any previous customization - - Credentials such as kubeconfig, admin user and password - - Worker nodes with minimum 96GB of memory (DRAM) - - For SAP HANA: Worker nodes with Intel CPU Instruction Sets: `TSX` ([SAP Note 2737837](https://me.sap.com/notes/2737837/E)) - - Storage - - Netapp filer with NFS using Astra Trident Operator or - - Local storage using Host Path Provisioner (HPP). - - OpenShift Data Foundation or other storage orchestrators have to be manually configured. - - -### Execution/Controller host - -An Ansible Automation Platform Controller can be used to facilitate the orchestration. A jumphost with access to the Red Hat OpenShift cluster is required. - -**Dependencies** -- OS Packages - - Python 3.9.7+ (i.e. CPython distribution) -- Python Packages: - - `kubernetes` 29.0.0+ -- Ansible - - Ansible Core 2.12.0+ - - Ansible Collections: - - `kubernetes.core` 3.0.0+ - - `community.okd` 3.0.1 - -See also the `requirements.yml` if running standalone. The requirements can be installed with -``` -# ansible-galaxy install -r requirements.yml -``` - -**During execution** -- For Red Hat OpenShift Virtualization (OCPV), use environment variable `K8S_AUTH_KUBECONFIG` - - -### Role Variables -Use [sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml](../playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml) as a starting point and add your configuration. - -Let's have a look at the most important variables you need to set. - -``` -########################################################### -# Red Hat OpenShift cluster connection details -########################################################### - -# Admin username for Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_admin_username: - -# Admin password for Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_admin_password: - -# Path to kubeconfig file Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_kubeconfig_path: - -# If this is set to true, the API endpoint and the -# CA Certificate are extracted from the kubeconfig file. -# If set to false, sap_hypervisor_node_preconfigure_ocp_endpoint and -# sap_hypervisor_node_preconfigure_ocp_ca_cert have to be specified. -sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig: true - -``` -You need to provide username and password for the Red Hat OpenShift Cluster. The `kubeconfig` file can be specified in `sap_hypervisor_node_preconfigure_ocp_kubeconfig_path` or if omitted, the environment variable `K8S_AUTH_KUBECONFIG` has to point to it. Default is, to use the CA certificate and Red Hat OpenShift cluster API endpoint as specified in the `kubeconfig` file (controlled by variable `sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig`). Make sure to specify the username and password for the cluster: `sap_hypervisor_node_preconfigure_ocp_admin_username` and `sap_hypervisor_node_preconfigure_ocp_admin_password`. - -Next are variables that define what storage configuration should be configured, if the operators should be installed and the configuration of the workers should be done. - -``` -########################################################### -# Configuration of what should be preconfigured -########################################################### - -# Install and configure the host path provisioner (hpp) for a local storage disk -sap_hypervisor_node_preconfigure_install_hpp: false - -# Install the trident NFS storage provider -sap_hypervisor_node_preconfigure_install_trident: false - -# Should the operators be installed -sap_hypervisor_node_preconfigure_install_operators: true - -# Configure the workers? -sap_hypervisor_node_preconfigure_setup_worker_nodes: true -``` - -The next section you have to modify are the cluster configuration details. Every worker has to have an entry in the `workers` section and make sure, that the name attribute corresponds with the cluster node name (here: worker-0). Adjust the network interface name you want to use. There are two types of networking technologies available: bridging or SR-IOV. See the configuration example file for more options (`playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml`). +- A fresh OpenShift cluster without prior custom configurations is required. +- Cluster credentials available: `kubeconfig`. +- Worker nodes with a minimum of 96GB of RAM. Note that larger configurations may be necessary for production SAP workloads. +- Storage: + - NetApp NFS storage provisioned by the Astra Trident Operator. + - Local storage provisioned by the Host Path Provisioner (HPP). + - OpenShift Data Foundation (ODF) and other storage orchestrators must be configured separately. +- For SAP HANA: Worker nodes with Intel CPU Instruction Sets: `TSX` ([SAP Note 2737837](https://me.sap.com/notes/2737837/E)) + +### Control Node Requirements +For a list of all collection prerequisites, please see the [Ansible Collection Readme](../../README.md#requirements). + +Direct access to the Red Hat OpenShift cluster is required. +- An Ansible Automation Platform Controller can be used to facilitate the orchestration. + +- Operating System packages: + - Python 3.11 or higher +- Python libraries and modules: + - `ansible-core` 2.16 or higher + - `kubernetes` >= 29.0.0 +- Ansible Collections: + - `kubernetes.core` >= 3.0.0 + - `community.okd` >= 3.0.1 + +### Platform Specific Variables +All platform specific variables are available in [vars/platform_defaults_redhat_ocp_virt.yml](vars/platform_defaults_redhat_ocp_virt.yml). + +The `kubeconfig` configuration file has to be provided by either: +1. The Ansible variable `sap_hypervisor_node_kubeconfig`. +2. The environment variable `K8S_AUTH_KUBECONFIG`. +3. The environment variable `KUBECONFIG`. +**NOTE:** If using the trident storage operator, the `kubeconfig` has also to contain a valid API token. + +Every worker has to have an entry in the `workers` section of the variable `sap_hypervisor_node_preconfigure_cluster_config` and make sure, that the name attribute corresponds with the cluster node name (e.g. worker-0). Adjust the network interface name you want to use. There are two types of networking technologies available: bridging or SR-IOV. There is a section for the `trident` configuration, this is required when installing the NetApp Astra Trident Operator for NFS storage. When using the host path provisioner, `worker_localstorage_device` has to point to the block device which should be used. - -``` -########################################################### -# Red Hat OpenShift cluster configuration details -########################################################### - -# Example configuration for redhat_ocp_virt -sap_hypervisor_node_preconfigure_cluster_config: - - # namespace under which the VMs are created, note this has to be - # openshift-sriov-network-operator in case of using SR-IOV network - # devices - vm_namespace: sap - - # Optional, configuration for trident driver for Netapp NFS filer - trident: - management: management.domain.org - data: datalif.netapp.domain.org - svm: sap_svm - backend: nas_backend - aggregate: aggregate_Name - username: admin - password: xxxxx - storage_driver: ontap-nas - storage_prefix: ocpv_sap_ - - # CPU cores which will be reserved for kubernetes - worker_kubernetes_reserved_cpus: "0,1" - - # Storage device used for host path provisioner as local storage. - worker_localstorage_device: /dev/vdb - - # detailed configuration for every worker that should be configured - workers: - - - name: worker-0 # name must match the node name - networks: # Example network config - - - name: sapbridge # using a bridge - description: SAP bridge - state: up - type: linux-bridge - ipv4: - enabled: false - auto-gateway: false - auto-dns: false - bridge: - options: - stp: - enabled: false - port: - - name: ens1f0 # network IF name - -``` -### Example Playbook +### Example See [sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml](../playbooks/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml) for an example. -### Example Usage Make sure to set the `K8S_AUTH_KUBECONFIG` environment variable, e.g. ``` export K8S_AUTH_KUBECONFIG=/path/to/my_kubeconfig ``` -To invoke the example playbook with the example configuration using your localhost as ansible host use the following command line: +To invoke the example playbook with the example configuration using your localhost as ansible host use this command: +- It has to be executed from [`/playbooks`](../playbooks/) directory. ```shell ansible-playbook --connection=local -i localhost, \ -playbooks/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml \ --e @playbooks/vars/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml + sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml \ + -e @./vars/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml ``` ## Platform: Red Hat Virtualization (RHV) -This Ansible Role allows preconfigure of Red Hat Virtualization (RHV), formerly called Red Hat Enterprise Virtualization (RHEV) prior to version 4.4 release. Red Hat Virtualization (RHV) consists of 'Red Hat Virtualization Manager (RHV-M)' and the 'Red Hat Virtualization Host (RHV-H)' hypervisor nodes that this Ansible Role preconfigures. Please note, Red Hat Virtualization is discontinued and maintenance support will end mid-2024. Extended life support for RHV ends mid-2026. -This Ansible Role does not preconfigure RHEL KVM (RHEL-KVM) hypervisor nodes. Please note that RHEL KVM is standalone, and does not have Management tooling (previously provided by RHV-M). - -### Requirements +Configures the Red Hat Virtualization (RHV) hypervisor nodes, formerly known as Red Hat Enterprise Virtualization (RHEV) prior to version 4.4. -**Prerequisites:** -- Hypervisor Administrator credentials -- RHV hypervisor(s) - - -**Platform-specific - Red Hat Virtualization (RHV)** -- Jumphost - -### Role Variables -See [sample-variables-sap-hypervisor-redhat-rhel-kvm-preconfigure.yml](../playbooks/vars/sample-variables-sap-hypervisor-redhat-rhel-kvm-preconfigure.yml) for details. +Red Hat Virtualization consists of a `Red Hat Virtualization Manager (RHV-M)` and the `Red Hat Virtualization Host (RHV-H)` hypervisor nodes that this role pre-configures. + - **End of Life note:** Red Hat Virtualization is discontinued and maintenance support will end mid-2024. Extended life support for RHV ends mid-2026. -`sap_hypervisor_node_preconfigure_reserved_ram (default: 100)` Reserve memory [GB] for hypervisor host. Depending in the use case should be at least 50-100GB. +This Ansible Role does not preconfigure RHEL KVM (RHEL-KVM) hypervisor nodes. + - Please note that RHEL KVM is a standalone hypervisor and does not include the management tooling provided by RHV-M. -`sap_hypervisor_node_preconfigure_reserve_hugepages (default: static)` Hugepage allocation method: {static|runtime}. -static: done at kernel command line which is slow, but safe -runtime: done with hugeadm which is faster, but can in some cases not ensure all HPs are allocated. - -`sap_hypervisor_node_preconfigure_kvm_nx_huge_pages (default: "auto")` Setting for the huge page shattering kvm.nx_huge_pages: {"auto"|"on"|"off"}. Note the importance of the quotes, otherwise off will be mapped to false. See https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html for additional information: -``` - kvm.nx_huge_pages= - [KVM] Controls the software workaround for the - X86_BUG_ITLB_MULTIHIT bug. - force : Always deploy workaround. - off : Never deploy workaround. - auto : Deploy workaround based on the presence of - X86_BUG_ITLB_MULTIHIT. - - Default is 'auto'. - - If the software workaround is enabled for the host, - guests do need not to enable it for nested guests. -``` - -`sap_hypervisor_node_preconfigure_tsx (default: "off")` Intel Transactional Synchronization Extensions (TSX): {"on"|"off"}. Note the importance of the quotes, otherwise off will be mapped to false. +### Requirements +For a list of all collection prerequisites, please see the [Ansible Collection Readme](../../README.md#requirements). -`sap_hypervisor_node_preconfigure_assert (default: false)` In assert mode, the parameters on the system are checked if the confirm with what this role would set. +- Hypervisor Administrator credentials +- One or more available RHV hypervisors. -`sap_hypervisor_node_preconfigure_ignore_failed_assertion (default: no)` Fail if assertion is invalid. +### Platform Specific Variables +All platform specific variables are available in [vars/platform_defaults_redhat_rhel_kvm.yml](vars/platform_defaults_redhat_rhel_kvm.yml). -`sap_hypervisor_node_preconfigure_run_grub2_mkconfig (default: yes)` Update the grub2 config. +### Example +See [sample-sap-hypervisor-redhat-rhel-kvm-preconfigure.yml](../playbooks/sample-sap-hypervisor-redhat-rhel-kvm-preconfigure.yml) for an example. -### Example Playbook -See [sample-sap-hypervisor-redhat-rhel-kvm-preconfigure.yml](../playbooks/sample-sap-hypervisor-redhat-rhel-kvm-preconfigure.yml) for an example. + + -### License +## License + Apache 2.0 - -### Author Information -Nils Koenig (nkoenig@redhat.com) + + +## Maintainers + +- [Nils Koenig](https://github.com/newkit) + + +## Role Variables + +The list of all available variables: [/defaults parameters file](./defaults/main.yml). +The platform specific variables are defined in their respective files under `vars/` directory. + \ No newline at end of file diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/assert/main.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/assert/main.yml new file mode 100644 index 00000000..1d5f4d3d --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/assert/main.yml @@ -0,0 +1,5 @@ +--- +- name: Check if all trident config values are defined + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/assert/trident.yml" + when: sap_hypervisor_node_preconfigure_install_trident diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/assert/trident.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/assert/trident.yml new file mode 100644 index 00000000..a5398028 --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/assert/trident.yml @@ -0,0 +1,37 @@ +--- +- name: Define required trident keys + ansible.builtin.set_fact: + __sap_hypervisor_node_preconfigure_required_trident_keys: + - management + - data + - svm + - backend + - aggregate + - username + - password + - storage_driver + - storage_prefix + - nfs_mount_options + - export_policy + - storageclass_name + - default_storageclass + +- name: Initialize list of missing or empty trident keys + ansible.builtin.set_fact: + __sap_hypervisor_node_preconfigure_trident_missing_or_empty_keys: [] + +- name: Check each trident key for existence and non-empty value + ansible.builtin.set_fact: + __sap_hypervisor_node_preconfigure_trident_missing_or_empty_keys: "{{ __sap_hypervisor_node_preconfigure_trident_missing_or_empty_keys + [item] }}" + when: > + sap_hypervisor_node_preconfigure_cluster_config.trident[item] is not defined or + (sap_hypervisor_node_preconfigure_cluster_config.trident[item] | string | trim | length == 0) + loop: "{{ __sap_hypervisor_node_preconfigure_required_trident_keys }}" + +- name: Assert all trident keys are present and non-empty + ansible.builtin.assert: + that: + - __sap_hypervisor_node_preconfigure_trident_missing_or_empty_keys | length == 0 + fail_msg: >- + The following trident keys are missing or empty: + {{ __sap_hypervisor_node_preconfigure_trident_missing_or_empty_keys | join(', ') }} diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/enable-featuregate-downwardmetrics.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/enable-featuregate-downwardmetrics.yml new file mode 100644 index 00000000..c5fbc088 --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/enable-featuregate-downwardmetrics.yml @@ -0,0 +1,13 @@ +--- +- name: Enable DownwardMetrics Feature Gate on kubevirt-hyperconverged + kubernetes.core.k8s: + state: present + definition: + apiVersion: hco.kubevirt.io/v1beta1 + kind: HyperConverged + metadata: + name: kubevirt-hyperconverged + namespace: "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}" + spec: + featureGates: + downwardMetrics: true diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/wait-mcp-finished-updating.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/wait-mcp-finished-updating.yml new file mode 100644 index 00000000..55de648f --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/wait-mcp-finished-updating.yml @@ -0,0 +1,21 @@ +--- +- name: Wait for worker MachineConfigPool to start updating + kubernetes.core.k8s_info: + api_version: machineconfiguration.openshift.io/v1 + kind: MachineConfigPool + name: worker + ignore_errors: true + retries: 5 + delay: 60 + register: __sap_hypervisor_node_preconfigure_register_mcp_info + until: (__sap_hypervisor_node_preconfigure_register_mcp_info.resources[0].status.conditions | selectattr('type', 'equalto', 'Updating'))[0].status == 'True' + +- name: Wait for worker MachineConfigPool to be updated + kubernetes.core.k8s_info: + api_version: machineconfiguration.openshift.io/v1 + kind: MachineConfigPool + name: worker + retries: 60 + delay: 60 + register: __sap_hypervisor_node_preconfigure_register_mcp_info + until: (__sap_hypervisor_node_preconfigure_register_mcp_info.resources[0].status.conditions | selectattr('type', 'equalto', 'Updated'))[0].status == 'True' diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/create-sap-bridge.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/create-sap-bridge.yml deleted file mode 100644 index bbdbdfff..00000000 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/create-sap-bridge.yml +++ /dev/null @@ -1,49 +0,0 @@ ---- -- name: Create SAP bridge NodeNetworkConfigurationPolicy - kubernetes.core.k8s: - state: present - definition: - apiVersion: nmstate.io/v1 - kind: NodeNetworkConfigurationPolicy - metadata: - name: "sap-bridge-policy-{{ worker.name }}" - spec: - nodeSelector: - kubernetes.io/hostname: "{{ worker.name }}" - desiredState: - interfaces: - - name: sapbridge - description: "Linux bridge with {{ worker.sap_bridge_interface }} as physical port to access SAP network" - type: linux-bridge - state: up - ipv4: - enabled: false - bridge: - options: - stp: - enabled: false - port: - - name: "{{ worker.sap_bridge_interface }}" - - -- name: Create SAP bridge NetworkAttachmentDefinition - kubernetes.core.k8s: - state: present - definition: - apiVersion: "k8s.cni.cncf.io/v1" - kind: NetworkAttachmentDefinition - metadata: - kubernetes.io/hostname: "{{ worker.name }}" - machineconfiguration.openshift.io/role: "{{ worker.name }}" - namespace: "{{ vm_namespace }}" - name: sap-bridge-network-definition - annotations: - k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/sapbridge - spec: - config: '{ - "cniVersion": "0.3.1", - "name": "sap-bridge-network-definition", - "type": "cnv-bridge", - "bridge": "sapbridge", - "macspoofchk": true - }' diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/label-worker-invtsc.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/label-worker-invtsc.yml deleted file mode 100644 index 57a52da2..00000000 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/label-worker-invtsc.yml +++ /dev/null @@ -1,11 +0,0 @@ ---- -- name: Label worker with invtsc flag - kubernetes.core.k8s: - state: present - definition: - apiVersion: v1 - kind: Namespace - metadata: - name: default - labels: - 'feature.node.kubernetes.io/cpu-feature-invtsc': enabled diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/main.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/main.yml index cb982cd6..5c2501d1 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/main.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/main.yml @@ -1,4 +1,8 @@ --- +- name: Check configuration + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/assert/main.yml" + - name: Get a list of all nodes from any namespace kubernetes.core.k8s_info: kind: Node @@ -9,7 +13,7 @@ __sap_hypervisor_node_preconfigure_register_worker_node_name_list: "{{ __sap_hypervisor_node_preconfigure_register_worker_node_name_list | \ d([]) + [__sap_hypervisor_node_preconfigure_register_worker_node.name] }}" - with_items: "{{ sap_hypervisor_node_preconfigure_cluster_config.workers }}" + loop: "{{ sap_hypervisor_node_preconfigure_cluster_config.workers }}" loop_control: loop_var: __sap_hypervisor_node_preconfigure_register_worker_node @@ -18,7 +22,7 @@ __sap_hypervisor_node_preconfigure_register_nodes: "{{ __sap_hypervisor_node_preconfigure_register_nodes | \ d([]) + [__sap_hypervisor_node_preconfigure_register_host] }}" - with_items: "{{ __sap_hypervisor_node_preconfigure_register_node_list['resources'] }}" + loop: "{{ __sap_hypervisor_node_preconfigure_register_node_list['resources'] }}" loop_control: loop_var: __sap_hypervisor_node_preconfigure_register_host when: __sap_hypervisor_node_preconfigure_register_host.metadata.name in __sap_hypervisor_node_preconfigure_register_worker_node_name_list @@ -52,57 +56,52 @@ # if system < 512GiB memory use 32GiB as upper boundary, 64GB otherwise as upper boundary - name: Calculate amount of hugepages to reserve (host memory < 512 GiB) ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_worker_reserved_hugepages: "{{ __sap_hypervisor_node_preconfigure_register_worker_memory_gib | int \ + __sap_hypervisor_node_preconfigure_register_worker_reserved_hugepages: + "{{ __sap_hypervisor_node_preconfigure_register_worker_memory_gib | int \ - sap_hypervisor_node_preconfigure_hypervisor_reserved_ram_host_lt_512 }}" when: __sap_hypervisor_node_preconfigure_register_worker_memory_gib | int < 512 - name: Calculate amount of hugepages to reserve (host memory >= 512 GiB) ansible.builtin.set_fact: - __sap_hypervisor_node_preconfigure_register_worker_reserved_hugepages: "{{ __sap_hypervisor_node_preconfigure_register_worker_memory_gib | int \ + __sap_hypervisor_node_preconfigure_register_worker_reserved_hugepages: + "{{ __sap_hypervisor_node_preconfigure_register_worker_memory_gib | int \ - sap_hypervisor_node_preconfigure_hypervisor_reserved_ram_host_ge_512 }}" when: __sap_hypervisor_node_preconfigure_register_worker_memory_gib | int >= 512 - name: Include prepare - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/prepare.yml" + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/prepare.yml" - name: Include patch cluster masters unschedulable - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/patch-cluster-masters-unschedulable.yml" + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/master/patch-cluster-masters-unschedulable.yml" - name: Include tuned virtual host - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/tuned-virtual-host.yml" - -- name: Include install CNV operator - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/install-cnv-operator.yml" - when: sap_hypervisor_node_preconfigure_install_operators + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/worker/tuned-virtual-host.yml" -- name: Include install sriov operator - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/install-sriov-operator.yml" - when: sap_hypervisor_node_preconfigure_install_operators - -- name: Include install nmstate operator - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/install-nmstate-operator.yml" +- name: Include install operators + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/operators/main.yml" when: sap_hypervisor_node_preconfigure_install_operators - name: Include setup worker nodes - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/setup-worker-nodes.yml" + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/worker/setup-worker-nodes.yml" when: sap_hypervisor_node_preconfigure_setup_worker_nodes -- name: Wait for all k8s nodes to be ready - kubernetes.core.k8s_info: - kind: Node - wait: yes - wait_condition: - status: True - type: Ready - wait_sleep: 20 - wait_timeout: 600 +- name: Include task enable featuregate downwardmetrics + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/cluster/enable-featuregate-downwardmetrics.yml" - name: Include Trident installation - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/install-trident.yml" + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/storage/install-trident.yml" when: sap_hypervisor_node_preconfigure_install_trident - name: Include local storage creation (HPP) - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/install-hpp.yml" + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/storage/install-hpp.yml" when: sap_hypervisor_node_preconfigure_install_hpp - name: Remove temporary directory diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/patch-cluster-masters-unschedulable.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/master/patch-cluster-masters-unschedulable.yml similarity index 100% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/patch-cluster-masters-unschedulable.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/master/patch-cluster-masters-unschedulable.yml diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-cnv-operator.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-cnv-operator.yml similarity index 87% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-cnv-operator.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-cnv-operator.yml index 83c21603..9f3e9d05 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-cnv-operator.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-cnv-operator.yml @@ -46,7 +46,7 @@ namespace: "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}" register: __sap_hypervisor_node_preconfigure_register_cnv_subscription_install_plan_name until: __sap_hypervisor_node_preconfigure_register_cnv_subscription_install_plan_name.resources[0].status.installPlanRef.name is defined - changed_when: True + changed_when: true - name: Set Install Plan Name ansible.builtin.set_fact: @@ -84,6 +84,17 @@ delay: 60 until: hco_webhook_pod.resources | selectattr('status.phase', 'equalto', 'Running') | list | length == hco_webhook_pod.resources | length +- name: Wait and check for pod with name hco-webhook under "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}" namespace + kubernetes.core.k8s_info: + kind: Pod + namespace: "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}" + label_selectors: + - name=hco-webhook + register: __sap_hypervisor_node_preconfigure_register_hco_webhook_pod + retries: 5 + delay: 60 + until: __sap_hypervisor_node_preconfigure_register_hco_webhook_pod.resources | selectattr('status.phase', 'equalto', 'Running') | list | length == __sap_hypervisor_node_preconfigure_register_hco_webhook_pod.resources | length + - name: Wait and check if hco-webhook-service exists in "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}" namespace kubernetes.core.k8s_info: kind: Service @@ -113,7 +124,7 @@ namespace: "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}" wait_condition: type: Available - status: "True" + status: true wait_timeout: 600 register: hyperconverged_status diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-nmstate-operator.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-nmstate-operator.yml similarity index 82% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-nmstate-operator.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-nmstate-operator.yml index 12f493a7..db9c171f 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-nmstate-operator.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-nmstate-operator.yml @@ -99,3 +99,14 @@ retries: 30 delay: 10 until: nmstate_status.resources | length > 0 + +- name: Wait and check for pod with label name nmstate-webhook under "{{ sap_hypervisor_node_preconfigure_nmstate_namespace }}" namespace + kubernetes.core.k8s_info: + kind: Pod + namespace: "{{ sap_hypervisor_node_preconfigure_nmstate_namespace }}" + label_selectors: + - name=nmstate-webhook + register: sap_hypervisor_node_preconfigure_register_webhook_pod + retries: 5 + delay: 60 + until: sap_hypervisor_node_preconfigure_register_webhook_pod.resources | selectattr('status.phase', 'equalto', 'Running') | list | length == sap_hypervisor_node_preconfigure_register_webhook_pod.resources | length diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-sriov-operator.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-sriov-operator.yml similarity index 100% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-sriov-operator.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/install-sriov-operator.yml diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/main.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/main.yml new file mode 100644 index 00000000..eed42e0e --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/operators/main.yml @@ -0,0 +1,12 @@ +--- +- name: Include install CNV operator + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/operators/install-cnv-operator.yml" + +- name: Include install sriov operator + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/operators/install-sriov-operator.yml" + +- name: Include install nmstate operator + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/operators/install-nmstate-operator.yml" diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/sriov-enabled-unsupported-nics.sh b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/sriov-enabled-unsupported-nics.sh deleted file mode 100644 index 6cec1a67..00000000 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/sriov-enabled-unsupported-nics.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/bash -# in order to allow unsupported SRIOV nics such as Mellanox -oc patch sriovoperatorconfig default --type=merge -n openshift-sriov-network-operator --patch '{ "spec": { "enableOperatorWebhook": false } }' diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-hpp.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/install-hpp.yml similarity index 54% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-hpp.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/install-hpp.yml index d2f3e9bb..67b09697 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-hpp.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/install-hpp.yml @@ -1,4 +1,12 @@ --- +- name: Include worker node mkfs loop + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/storage/worker-mkfs-loop.yml" + loop: "{{ sap_hypervisor_node_preconfigure_cluster_config.workers }}" + loop_control: + loop_var: __sap_hypervisor_node_preconfigure_register_worker + index_var: __sap_hypervisor_node_preconfigure_register_worker_nr + - name: Create systemd files for local storage handling kubernetes.core.k8s: state: present @@ -17,42 +25,22 @@ systemd: units: - contents: | - [Unit] - Description=Create mountpoint /var/localstorage and initialize filesystem - Before=var-localstorage.mount - [Service] - Type=oneshot - ExecStart=/bin/bash -c "if [[ $(lsblk -o FSTYPE {{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }} --noheadings) != 'xfs' ]]; then mkfs.xfs -f {{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }}; fi" - ExecStart=/bin/mkdir -p /var/localstorage - enabled: true - name: create-mountpoint-var-localstorage.service - - contents: | - [Unit] - After=create-mountpoint-var-localstorage.service - Requires=create-mountpoint-var-localstorage.service - [Mount] - What={{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }} - Where=/var/localstorage - Type=xfs [Install] WantedBy=local-fs.target - enabled: true - name: var-localstorage.mount - - contents: | [Unit] - Description=Set SELinux chcon for hostpath provisioner + Description=Create mountpoint {{ sap_hypervisor_node_preconfigure_hpp_mountpoint }}, mount and set selinux flag Before=kubelet.service - After=var-localstorage.mount [Service] - ExecStart=/usr/bin/chcon -Rt container_file_t /var/localstorage - [Install] - WantedBy=multi-user.target + Type=oneshot + ExecStart=/bin/mkdir -p {{ sap_hypervisor_node_preconfigure_hpp_mountpoint }} + ExecStart=mount {{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }} {{ sap_hypervisor_node_preconfigure_hpp_mountpoint }} + ExecStart=/usr/bin/chcon -Rt container_file_t {{ sap_hypervisor_node_preconfigure_hpp_mountpoint }} enabled: true - name: hostpath-provisioner.service + name: create-mount-localstorage.service -- name: Wait for mountpoint to be ready - ansible.builtin.pause: - minutes: 3 +- name: Include wait mcp finished updating + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/cluster/wait-mcp-finished-updating.yml" - name: Create hostpath provisioner (HPP) kubernetes.core.k8s: @@ -66,7 +54,7 @@ imagePullPolicy: IfNotPresent storagePools: - name: localstorage - path: /var/localstorage + path: "{{ sap_hypervisor_node_preconfigure_hpp_mountpoint }}" workload: nodeSelector: kubernetes.io/os: linux @@ -79,7 +67,7 @@ wait: true wait_condition: type: Available - status: "True" + status: true reason: Complete wait_sleep: 30 wait_timeout: 300 @@ -91,9 +79,9 @@ apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: - name: local + name: "{{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_storageclass_name }}" annotations: - storageclass.kubernetes.io/is-default-class: "true" + storageclass.kubernetes.io/is-default-class: "{{ 'true' if sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_default_storageclass is true else 'false' }}" provisioner: kubevirt.io.hostpath-provisioner reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-trident.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/install-trident.yml similarity index 86% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-trident.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/install-trident.yml index 6cf235a6..416c8af4 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/install-trident.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/install-trident.yml @@ -1,5 +1,5 @@ --- -- name: Download trident +- name: Download and extract trident ansible.builtin.unarchive: remote_src: true src: "{{ sap_hypervisor_node_preconfigure_install_trident_url }}" @@ -35,9 +35,9 @@ apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: - name: nas + name: "{{ sap_hypervisor_node_preconfigure_cluster_config.trident.storageclass_name }}" annotations: - storageclass.kubernetes.io/is-default-class: "true" + storageclass.kubernetes.io/is-default-class: "{{ 'true' if sap_hypervisor_node_preconfigure_cluster_config.trident.default_storageclass is true else 'false' }}" provisioner: csi.trident.netapp.io parameters: backendType: "{{ sap_hypervisor_node_preconfigure_cluster_config.trident.storage_driver }}" diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/trident-backend.json.j2 b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/trident-backend.json.j2 similarity index 79% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/trident-backend.json.j2 rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/trident-backend.json.j2 index e422aab1..b92969bb 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/trident-backend.json.j2 +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/trident-backend.json.j2 @@ -1,8 +1,8 @@ { - "nfsMountOptions": "nfsvers=3", - "defaults": { - "exportPolicy": "default" - }, + "nfsMountOptions": "{{ sap_hypervisor_node_preconfigure_cluster_config.trident.nfs_mount_options }}", + "defaults": { + "exportPolicy": "{{ sap_hypervisor_node_preconfigure_cluster_config.trident.export_policy }}" + }, "debug":false, "managementLIF":"{{ sap_hypervisor_node_preconfigure_cluster_config.trident.management }}", "dataLIF":"{{ sap_hypervisor_node_preconfigure_cluster_config.trident.data }}", @@ -15,4 +15,3 @@ "storagePrefix":"{{ sap_hypervisor_node_preconfigure_cluster_config.trident.storage_prefix }}", "version":1 } - diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/worker-mkfs-loop.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/worker-mkfs-loop.yml new file mode 100644 index 00000000..a5a87102 --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/storage/worker-mkfs-loop.yml @@ -0,0 +1,32 @@ +--- +- name: Create pod that initializes filesystem + kubernetes.core.k8s: + state: present + definition: + apiVersion: v1 + kind: Pod + metadata: + name: "make-filesystem-{{ __sap_hypervisor_node_preconfigure_register_worker.name }}" + namespace: default + spec: + nodeName: "{{ __sap_hypervisor_node_preconfigure_register_worker.name }}" + containers: + - name: "make-filesystem-{{ __sap_hypervisor_node_preconfigure_register_worker.name }}" + image: quay.io/fedora/fedora:latest + command: + - /bin/bash + - -c + - | + dnf install -y xfsprogs && \ + mkfs.xfs -f "{{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }}" + securityContext: + privileged: true + volumeMounts: + - mountPath: "{{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }}" + name: rawdisk + volumes: + - name: rawdisk + hostPath: + path: "{{ sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device }}" + type: BlockDevice + restartPolicy: Never diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/99-kargs-worker.yml.j2 b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/99-kargs-worker.yml.j2 similarity index 100% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/99-kargs-worker.yml.j2 rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/99-kargs-worker.yml.j2 diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/setup-worker-nodes.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/enable-cpumanager.yml similarity index 73% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/setup-worker-nodes.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/enable-cpumanager.yml index 50acdc0e..8b310eb3 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/setup-worker-nodes.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/enable-cpumanager.yml @@ -1,11 +1,4 @@ --- -- name: Include configure worker - ansible.builtin.include_tasks: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/configure-worker-node.yml" - with_items: "{{ sap_hypervisor_node_preconfigure_cluster_config.workers }}" - loop_control: - loop_var: __sap_hypervisor_node_preconfigure_register_worker - index_var: __sap_hypervisor_node_preconfigure_register_worker_nr - - name: Enable CPU Manager by patching MCP worker kubernetes.core.k8s: state: patched @@ -69,13 +62,6 @@ cpuManagerPolicy: static cpuManagerReconcilePeriod: 5s -- name: Render template - ansible.builtin.template: - src: 99-kargs-worker.yml.j2 - dest: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_role.path }}/99-kargs-worker.yml" - mode: "0644" - -- name: Enable hugepages - kubernetes.core.k8s: - state: present - src: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_role.path }}/99-kargs-worker.yml" +- name: Include wait mcp finished updating + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/cluster/wait-mcp-finished-updating.yml" diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/enable-kargs.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/enable-kargs.yml new file mode 100644 index 00000000..6ee89292 --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/enable-kargs.yml @@ -0,0 +1,15 @@ +--- +- name: Render template + ansible.builtin.template: + src: 99-kargs-worker.yml.j2 + dest: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_role.path }}/99-kargs-worker.yml" + mode: "0644" + +- name: Enable hugepages + kubernetes.core.k8s: + state: present + src: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_role.path }}/99-kargs-worker.yml" + +- name: Include wait mcp finished updating + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/cluster/wait-mcp-finished-updating.yml" diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/node-network.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/node-network.yml similarity index 91% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/node-network.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/node-network.yml index b8fc1e41..9a1212bc 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/node-network.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/node-network.yml @@ -21,6 +21,7 @@ - "{{ __sap_hypervisor_node_preconfigure_register_worker_network }}" when: __sap_hypervisor_node_preconfigure_register_worker_network.type == 'linux-bridge' or __sap_hypervisor_node_preconfigure_register_worker_network.type == 'ethernet' + or __sap_hypervisor_node_preconfigure_register_worker_network.type == 'vlan' - name: "Create NetworkAttachmentDefinition {{ __sap_hypervisor_node_preconfigure_register_worker_network.name }}" kubernetes.core.k8s: @@ -34,13 +35,13 @@ annotations: k8s.v1.cni.cncf.io/resourceName: "bridge.network.kubevirt.io/{{ __sap_hypervisor_node_preconfigure_register_worker_network.name }}" spec: - config: '{ - "cniVersion": "0.3.1", - "name": "sapbridge-network-definition", - "type": "cnv-bridge", - "bridge": "sapbridge", - "macspoofchk": true - }' + config: "{{ { + 'cniVersion': '0.3.1', + 'name': __sap_hypervisor_node_preconfigure_register_worker_network.name, + 'type': 'cnv-bridge', + 'bridge': __sap_hypervisor_node_preconfigure_register_worker_network.name, + 'macspoofchk': true + } | to_json }}" when: __sap_hypervisor_node_preconfigure_register_worker_network.type == 'linux-bridge' - name: Label the node with feature.node.kubernetes.io/network-sriov.capable=true diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/setup-worker-nodes.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/setup-worker-nodes.yml new file mode 100644 index 00000000..fdb14244 --- /dev/null +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/setup-worker-nodes.yml @@ -0,0 +1,19 @@ +--- +- name: Include worker node network loop + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/worker/worker-node-network-loop.yml" + loop: "{{ sap_hypervisor_node_preconfigure_cluster_config.workers }}" + loop_control: + loop_var: __sap_hypervisor_node_preconfigure_register_worker + index_var: __sap_hypervisor_node_preconfigure_register_worker_nr + when: sap_hypervisor_node_preconfigure_configure_worker_node_network + +- name: Include enable cpumanager + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/worker/enable-cpumanager.yml" + when: sap_hypervisor_node_preconfigure_enable_cpumanager + +- name: Include enable kargs + ansible.builtin.include_tasks: + file: "platform/{{ sap_hypervisor_node_preconfigure_platform }}/worker/enable-kargs.yml" + when: sap_hypervisor_node_preconfigure_enable_kargs diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/tuned-virtual-host.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/tuned-virtual-host.yml similarity index 89% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/tuned-virtual-host.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/tuned-virtual-host.yml index e2dd4f48..642c05e7 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/tuned-virtual-host.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/tuned-virtual-host.yml @@ -11,8 +11,8 @@ spec: profile: - data: | - [main] - include=virtual-host + [main] + include=virtual-host name: virtual-host recommend: - match: diff --git a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/configure-worker-node.yml b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/worker-node-network-loop.yml similarity index 53% rename from roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/configure-worker-node.yml rename to roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/worker-node-network-loop.yml index 9680189e..3da13649 100644 --- a/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/configure-worker-node.yml +++ b/roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/worker/worker-node-network-loop.yml @@ -1,18 +1,8 @@ --- - name: Include node network ansible.builtin.include_tasks: node-network.yml - with_items: "{{ __sap_hypervisor_node_preconfigure_register_worker.networks }}" + loop: "{{ __sap_hypervisor_node_preconfigure_register_worker.networks }}" loop_control: loop_var: __sap_hypervisor_node_preconfigure_register_worker_network index_var: __sap_hypervisor_node_preconfigure_register_worker_network_nr when: __sap_hypervisor_node_preconfigure_register_worker.networks is defined - -- name: Wait for all k8s nodes to be ready - kubernetes.core.k8s_info: - kind: Node - wait: true - wait_condition: - status: true - type: Ready - wait_sleep: 20 - wait_timeout: 600 diff --git a/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_ocp_virt.yml b/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_ocp_virt.yml index cc85822f..e7f08b94 100644 --- a/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_ocp_virt.yml +++ b/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_ocp_virt.yml @@ -6,28 +6,11 @@ # Red Hat OpenShift cluster connection details ########################################################### -# Admin username for Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_admin_username: - -# Admin password for Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_admin_password: - -# Path to kubeconfig file Red Hat OpenShift cluster connection -sap_hypervisor_node_preconfigure_ocp_kubeconfig_path: - -# If this is set to true, the API endpoint and the -# CA Certificate are extracted from the kubeconfig file. -# If set to false, sap_hypervisor_node_preconfigure_ocp_endpoint and -# sap_hypervisor_node_preconfigure_ocp_ca_cert have to be specified. -sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig: true - -# URL to the API endpoint of Red Hat OpenShift cluster -#sap_hypervisor_node_preconfigure_ocp_endpoint: - -# CA Certificate for Red Hat OpenShift cluster connection -# To extract the CA Cert from the kubeconfig, you can use -# grep certificate-authority-data ${KUBECONFIG} | awk '{ print $2 }' | base64 --decode > client-cert.pem -#sap_hypervisor_node_preconfigure_ocp_ca_cert: +# kubeconfig for Red Hat OpenShift cluster connection. +# Needs to contain a valid API token for trident storage operator to work. +# If not provided, the kubeconfig will be read from the environment variables +# KUBECONFIG or K8S_AUTH_KUBECONFIG +# sap_hypervisor_node_preconfigure_kubeconfig: ########################################################### @@ -47,6 +30,21 @@ sap_hypervisor_node_preconfigure_install_operators: true sap_hypervisor_node_preconfigure_setup_worker_nodes: true +################################################################################### +# Individual worker nodes config details, typically no need to change anything here. +# Requires sap_hypervisor_node_preconfigure_setup_worker_nodes to be set to true. +################################################################################### + +# Should the network of the nodes be configured? +sap_hypervisor_node_preconfigure_configure_worker_node_network: true + +# Should the cpumanager be enabled? This has to be set to true. +sap_hypervisor_node_preconfigure_enable_cpumanager: true + +# Should the kargs be applied to the worker nodes? +sap_hypervisor_node_preconfigure_enable_kargs: true + + ########################################################### # Configuration details ########################################################### @@ -69,6 +67,12 @@ sap_hypervisor_node_preconfigure_ignore_minimal_memory_check: false # Namespace to be used for the Red Hat Openshift Virtualization Operator sap_hypervisor_node_preconfigure_ocpv_namespace: openshift-cnv +# Namespace to be used for the Red Hat Openshift Nmstate Operator +sap_hypervisor_node_preconfigure_nmstate_namespace: openshift-nmstate + +# Mountpoint for hostpath provisioner +sap_hypervisor_node_preconfigure_hpp_mountpoint: /var/localstorage + ########################################################### # Red Hat OpenShift cluster configuration details @@ -93,6 +97,10 @@ sap_hypervisor_node_preconfigure_cluster_config: password: xxxxx storage_driver: ontap-nas storage_prefix: ocpv_sap_ + nfs_mount_options: nfsvers=3 + export_policy: default + storageclass_name: sapstorage # Name of the storage class (careful, there can only be one SC with this name) + default_storageclass: true # Should this storage class be default? # CPU cores which will be reserved for kubernetes worker_kubernetes_reserved_cpus: "0,1" @@ -100,6 +108,12 @@ sap_hypervisor_node_preconfigure_cluster_config: # Storage device used for host path provisioner as local storage. worker_localstorage_device: /dev/vdb + # Name of the storage class for locastorage + worker_localstorage_storageclass_name: sapstorage + + # Should the local storage class be default? + worker_localstorage_default_storageclass: true + # detailed configuration for every worker that should be configured workers: diff --git a/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_rhel_kvm.yml b/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_rhel_kvm.yml index 92d1f2f9..88c9cd05 100644 --- a/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_rhel_kvm.yml +++ b/roles/sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_rhel_kvm.yml @@ -19,15 +19,17 @@ sap_hypervisor_node_preconfigure_fail_if_reboot_required: false sap_hypervisor_node_preconfigure_run_grub2_mkconfig: true # Reserve memory [GB] for hypervisor host +# Depending in the use case should be at least 50-100GB. sap_hypervisor_node_preconfigure_reserved_ram: 100 # allocate hugepages: {static|runtime} # static: done at kernel command line which is slow, but safe -# runtime: done through tuned-profile +# runtime: done with hugeadm which is faster, but can in some cases not ensure all HPs are allocated. sap_hypervisor_node_preconfigure_reserve_hugepages: static # kvm.nx_huge_pages: {"auto"|"on"|"off"} # Note the importance of the quotes, otherwise off will be mapped to false +# See https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html for additional information sap_hypervisor_node_preconfigure_kvm_nx_huge_pages: "auto" # Intel Transactional Synchronization Extensions (TSX): {"on"|"off"} @@ -37,3 +39,7 @@ sap_hypervisor_node_preconfigure_tsx: "on" # fail if assertion is invalid sap_hypervisor_node_preconfigure_ignore_failed_assertion: false + +# Set to true to enable assert mode. +# In assert mode, the parameters on the system are checked if the confirm with what this role would set. +# sap_hypervisor_node_preconfigure_assert: false diff --git a/roles/sap_vm_preconfigure/README.md b/roles/sap_vm_preconfigure/README.md index 03289ca4..738b1eda 100644 --- a/roles/sap_vm_preconfigure/README.md +++ b/roles/sap_vm_preconfigure/README.md @@ -1,6 +1,7 @@ `WIP` # sap_vm_preconfigure +![Ansible Lint for sap_vm_preconfigure](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_preconfigure.yml/badge.svg) Ansible Role for Vendor-specific configuration preparation tasks for Virtual Machines running SAP Systems. diff --git a/roles/sap_vm_provision/PLATFORM_GUIDANCE.md b/roles/sap_vm_provision/PLATFORM_GUIDANCE.md index 4bdd8c70..bde5b138 100644 --- a/roles/sap_vm_provision/PLATFORM_GUIDANCE.md +++ b/roles/sap_vm_provision/PLATFORM_GUIDANCE.md @@ -5,31 +5,31 @@ Table of Contents: - [Recommended Infrastructure Platform authorizations](#recommended-infrastructure-platform-authorizations) - [Recommended Infrastructure Platform configuration](#recommended-infrastructure-platform-configuration) -## Key note - Connectivity - -The Ansible Control Node AKA Controller (i.e. device where Ansible Playbook is executed), must be able to directly call the platform's API endpoints. For example: +## Key note - Cloud Connectivity +The Ansible Control Node AKA Execution Node (i.e. device where Ansible Playbook is executed), must be able to directly call the platform's API endpoints. For example: - AWS EC2 API endpoint `ec2.us-east-1.amazonaws.com` -- VMware vSphere REST API endpoint `.:443` - -By default, a Cloud account will use Public internet endpoints which should be accessible in most cases. The Cloud account may utilise Private endpoints for security, as would an On-Premise Hypervisor. Examples include: - -- running an Ansible Playbook from a personal laptop, then the personal laptop acts as the Ansible Control Node and can access the platform's APIs using a Client-to-Site VPN Client (such as OpenVPN Connect) to provision Virtual Machines for deploying SAP software -- running an Ansible Playbook from an existing host (e.g. VM) inside the platform's private network, then the existing host acts as the Ansible Control Node and can access the platform's APIs to provision Virtual Machines for deploying SAP software - -The subsequent provisioned Virtual Machine, must be accessible too - this can utilise a Bastion for SSH Proxy connection, which is common for Cloud IaaS. - -The Ansible Control Node AKA Controller (i.e. device where Ansible Playbook is executed), must be able to SSH to the Ansible Target Node (i.e. Virtual Machine) using: +- VMware vSphere REST API endpoint `.:443` -- DEFAULT: SSH Proxy connection from Ansible control node, via Bastion host, to target node (`sap_vm_provision_bastion_execution: true`); with SSH Private Keys for the host and the bastion (`sap_vm_provision_ssh_host_private_key_file_path: "/path"` and `sap_vm_provision_ssh_bastion_private_key_file_path: "/path"`) -- Direct SSH connection from Ansible control node to target node (`sap_vm_provision_bastion_execution: false`); with SSH Private Key for the host (`sap_vm_provision_ssh_host_private_key_file_path: "/path"`). +By default, a Cloud account will use Public internet endpoints which should be accessible in most cases. +The Cloud account may utilize Private endpoints for security, as would an On-Premise Hypervisor. Examples include: +- Connection from a public device (e.g. Personal laptop). + - It can access Private endpoint using direct Cloud VPN solution or Client-to-Site VPN Client (e.g. OpenVPN Connect) to connect to Company network, which has access to Private endpoint. +- Connection from an existing host in private network in on-premise. + - It can access Private endpoint directly if on-premise network is connected with Cloud (e.g. Site-to-Site VPN, AWS Direct Connect, Azure ExpressRoute, etc.). +- Connection from an existing host in private network in Cloud. + - It can access Private endpoint directly. +By default, this Ansible Role utilizes Bastion host as SSH Proxy for connection to provisioned hosts, which is recommended method for Security. +This behavior is controlled by variable `sap_vm_provision_bastion_execution`: +- `true`: SSH Proxy connection from Ansible control node, via Bastion host, to target node with SSH Private Keys for the host `sap_vm_provision_ssh_bastion_private_key_file_path` and the bastion `sap_vm_provision_ssh_host_private_key_file_path`. +- `false`: Direct SSH connection from Ansible control node to target node with SSH Private Key for the host `sap_vm_provision_ssh_host_private_key_file_path`. -## Required resources when Ansible provisioning VMs -The following does not apply if Ansible to Terraform is used. +## Infrastructure Prerequisites for Ansible provisioning method +**NOTE:** The following does not apply if `sap_vm_provision_iac_type: ansible_to_terraform` is used. -See below for the drop-down list of required environment resources on an Infrastructure Platform resources when Ansible is used to provision Virtual Machines. +See below for the drop-down list of required environment resources on an Infrastructure Platform.
Amazon Web Services (AWS): @@ -41,7 +41,7 @@ See below for the drop-down list of required environment resources on an Infrast - Route53 (Private DNS) - Internet Gateway (SNAT) - EFS (NFS) -- Bastion host (AWS EC2 VS) +- Bastion host (AWS EC2 VS) - This becomes optional, if `sap_vm_provision_bastion_execution` is set to `false`. - Key Pair for hosts
@@ -56,7 +56,7 @@ See below for the drop-down list of required environment resources on an Infrast - Cloud NAT (SNAT) - DNS Managed Zone (Private DNS) - Filestore (NFS) or NFS server -- Bastion host (GCP CE VM) +- Bastion host (GCP CE VM) - This becomes optional, if `sap_vm_provision_bastion_execution` is set to `false`. @@ -72,7 +72,7 @@ See below for the drop-down list of required environment resources on an Infrast - Storage Account - Azure Files (aka. File Storage Share, NFS) - Private Endpoint Connection -- Bastion host (MS Azure VM) +- Bastion host (MS Azure VM) - This becomes optional, if `sap_vm_provision_bastion_execution` is set to `false`. - Key Pair for hosts @@ -88,7 +88,7 @@ See below for the drop-down list of required environment resources on an Infrast - Private DNS - Public Gateway (SNAT) - File Share (NFS) -- Bastion host (IBM Cloud VS) +- Bastion host (IBM Cloud VS) - This becomes optional, if `sap_vm_provision_bastion_execution` is set to `false`. - Key Pair for hosts @@ -102,7 +102,7 @@ See below for the drop-down list of required environment resources on an Infrast - Cloud Connection (from secure enclave to IBM Cloud) - Private DNS Zone - Public Gateway (SNAT) -- Bastion host (IBM Cloud VS or IBM Power VS) +- Bastion host (IBM Cloud VS or IBM Power VS) - This becomes optional, if `sap_vm_provision_bastion_execution` is set to `false`. - Key Pair for hosts (in IBM Power Workspace) @@ -121,7 +121,7 @@ See below for the drop-down list of required environment resources on an Infrast
Red Hat OpenShift Virtualization (kubevirt_vm) -- IMPORTANT: The playbook has to run with the environment variable `ANSIBLE_JINJA2_NATIVE=true` otherwise you will see an unmarshalling error when the VM is created. On Ansible Automation Platform Controller (AAPC) you have to set this in Settings --> Job Settings --> Extra Environment Variables, e.g. +- IMPORTANT: The playbook has to run with the environment variable `ANSIBLE_JINJA2_NATIVE=true` otherwise you will see an `unmarshalling` error when the VM is created. On Ansible Automation Platform Controller (AAPC) you have to set this in Settings --> Job Settings --> Extra Environment Variables, e.g. ``` { "ANSIBLE_JINJA2_NATIVE": "true", @@ -129,13 +129,9 @@ See below for the drop-down list of required environment resources on an Infrast } ``` -- Kubeconfig file, kubeadmin user and password for the cluster you want to deploy. Default behavior is to extract CA certificate and API endpoint from kubeconfig (`sap_vm_provision_kubevirt_vm_extract_kubeconfig: true`). Kubeconfig location will be read from `sap_vm_provision_kubevirt_vm_kubeconfig_path` and if that variable is not defined from environment variable `K8S_AUTH_KUBECONFIG`. +- Kubeconfig file, kubeadmin user and password for the cluster you want to deploy. Default behavior is to extract CA certificate and API endpoint from kubeconfig (`sap_vm_provision_kubevirt_vm_extract_kubeconfig: true`). Kubeconfig location will be read from `sap_vm_provision_kubevirt_vm_kubeconfig` and if that variable is not defined from environment variable `K8S_AUTH_KUBECONFIG` or `KUBECONFIG`. -- SSH Key Pair for VMs or provide a password - - `sap_vm_provision_ocp_guest_ssh_auth_mechanism`: Authentication mechanism to be used to connect to the guest. Possible options are: - - `password`: Make sure to set password in `sap_vm_provision_ocp_os_user_password`. - - `private_key`: Use the private ssh key at the location defined by `sap_vm_provision_ssh_host_private_key_file_path`. - - `private_key_data`: use the private ssh key provided in `sap_vm_provision_ssh_host_private_key_data` and write it to the location defined in `sap_vm_provision_ssh_host_private_key_file_path`. +- SSH Key Pair for VMs - Optional: Ansible Control Node host with access to OpenShift cluster. @@ -167,16 +163,12 @@ See below for the drop-down list of required environment resources on an Infrast - Datastore - Content Library - VM Template -
- ## Recommended Infrastructure Platform authorizations - See below for the drop-down list of recommended authorizations for each Infrastructure Platform. -
Amazon Web Services (AWS): @@ -195,53 +187,52 @@ aws iam attach-group-policy --group-name 'ag-sap-automation' --policy-arn arn:aw It is recommended to create new AWS IAM Policy with detailed actions to improve security. ```json { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": [ - "ec2:DescribeImages", - "ec2:DescribeInstances", - "ec2:DescribeTags", - "ec2:DescribeInstanceAttribute", - "ec2:DescribeSubnets", - "ec2:DescribeSecurityGroups", - "ec2:RunInstances", - "ec2:CreateTags", - "ec2:DescribeInstanceStatus", - "ec2:ModifyInstanceAttribute", - "ec2:DescribeRouteTables", - "route53:ListHostedZones", - "route53:ListResourceRecordSets", - "route53:ChangeResourceRecordSets", - "route53:GetChange", - "ec2:DescribeVolumes", - "ec2:CreateVolume", - "ec2:DeleteVolume", - "ec2:AttachVolume", - "ec2:DetachVolume", - "ec2:TerminateInstances", - "ec2:CreateRoute", - "iam:GetRole", - "iam:CreateRole", - "iam:ListInstanceProfilesForRole", - "iam:CreateInstanceProfile", - "iam:AddRoleToInstanceProfile", - "iam:ListAttachedRolePolicies", - "iam:ListRoleTags", - "iam:PutRolePolicy", - "iam:GetInstanceProfile", - "iam:PassRole", - "ec2:AssociateIamInstanceProfile", - "ec2:ReplaceRoute" - ], - "Resource": "*" - } - ] + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "ec2:DescribeImages", + "ec2:DescribeInstances", + "ec2:DescribeTags", + "ec2:DescribeInstanceAttribute", + "ec2:DescribeSubnets", + "ec2:DescribeSecurityGroups", + "ec2:RunInstances", + "ec2:CreateTags", + "ec2:DescribeInstanceStatus", + "ec2:ModifyInstanceAttribute", + "ec2:DescribeRouteTables", + "route53:ListHostedZones", + "route53:ListResourceRecordSets", + "route53:ChangeResourceRecordSets", + "route53:GetChange", + "ec2:DescribeVolumes", + "ec2:CreateVolume", + "ec2:DeleteVolume", + "ec2:AttachVolume", + "ec2:DetachVolume", + "ec2:TerminateInstances", + "ec2:CreateRoute", + "iam:GetRole", + "iam:CreateRole", + "iam:ListInstanceProfilesForRole", + "iam:CreateInstanceProfile", + "iam:AddRoleToInstanceProfile", + "iam:ListAttachedRolePolicies", + "iam:ListRoleTags", + "iam:PutRolePolicy", + "iam:GetInstanceProfile", + "iam:PassRole", + "ec2:AssociateIamInstanceProfile", + "ec2:ReplaceRoute" + ], + "Resource": "*" + } + ] } ``` -
@@ -307,7 +298,6 @@ dns.resourceRecordSets.get dns.resourceRecordSets.list dns.resourceRecordSets.update ``` -
@@ -385,7 +375,6 @@ It is recommended to create new Azure custom role with detailed actions to impro ``` Note: MS Azure VMs provisioned will contain Hyper-V Hypervisor virtual interfaces using eth* on the OS, and when Accelerated Networking (AccelNet) is enabled for the MS Azure VM then the Mellanox SmartNIC/DPU SR-IOV Virtual Function (VF) may use enP* on the OS. For further information, see [MS Azure - How Accelerated Networking works](https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works). During High Availability executions, failures may occur and may require additional variable 'sap_ha_pacemaker_cluster_vip_client_interface' to be defined. -
@@ -420,14 +409,12 @@ Alternatively, use the IBM Cloud web console: - `[OPTIONAL]` IAM Services > All Identity and Access enabled services > click All resources as scope + Platform Access as Viewer + Resource group access as Administrator - `[OPTIONAL]` Account Management > Identity and Access Management > click Platform access as Editor - `[OPTIONAL]` Account Management > IAM Access Groups Service > click All resources as scope + Platform Access as Editor -
IBM PowerVC: The recommended [IBM PowerVC Security Role](https://www.ibm.com/docs/en/powervc/latest?topic=security-managing-roles) is 'Administrator assistant' (admin_assist), because the 'Virtual machine manager' (vm_manager) role is not able to create IBM PowerVM Compute Template (required for setting OpenStack extra_specs specific to the IBM PowerVM hypervisor infrastructure platform, such as Processing Units). Note that the 'Administrator assistant' does not have the privilege to delete Virtual Machines. -
@@ -446,8 +433,6 @@ Issues were resolved by following [Troubleshooting SLES pay-as-you-go registrati ``` Cloud NAT parameter "minimum ports per VM instance" has to be increased to higher than 160 (Recommended higher). ``` - -
@@ -499,7 +484,6 @@ When VMware vCenter and vSphere clusters with VMware NSX virtualized network ove - For outbound internet connectivity, use SNAT configuration (e.g. rule added on NSX Gateway) set for the Subnet which the VMware VM Template is attached to. Alternatively, use a Web Forward Proxy. N.B. When VMware vCenter and vSphere clusters with direct network subnet IP allocations to the VMXNet network adapter (no VMware NSX network overlays), the above actions may not be required. -
diff --git a/roles/sap_vm_provision/README.md b/roles/sap_vm_provision/README.md index 58ea5856..b1cc74db 100644 --- a/roles/sap_vm_provision/README.md +++ b/roles/sap_vm_provision/README.md @@ -1,140 +1,134 @@ + # sap_vm_provision Ansible Role + +![Ansible Lint for sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_provision.yml/badge.svg) -Ansible Role to provision Virtual Machines to host SAP Software. +## Description + +The Ansible Role `sap_vm_provision` is used to provision Virtual Machines to host SAP Software. +The provisioning methods are: +- `Ansible` - Used with existing minimal landing zone. +- `Terraform` - Used to provision minimal landing zone. Partially compatible with [Terraform Modules for SAP](https://github.com/sap-linuxlab/terraform.modules_for_sap). -This Ansible Role will provision Virtual Machines to different Infrastructure Platforms; with optional Ansible to Terraform to provision minimal landing zone (partial compatibility via [Terraform Modules for SAP](https://github.com/sap-linuxlab/terraform.modules_for_sap)). +This Ansible Role follows requirements and best practices of each Infrastructure Platform, while providing near-homogenous setup across all of them. + -Primarily, this Ansible Role was designed to be executed end-to-end (i.e. Provision host/s, configure OS for SAP Software, install SAP Software, instantiate the SAP System); such as the [Ansible Playbooks for SAP](https://github.com/sap-linuxlab/ansible.playbooks_for_sap). + + - -## Functionality - -The provisioned hosts by the Ansible Role provide a near-homogenous setup across different Infrastructure Platforms, while following requirements and best practices defined by each vendor. - -A series of choices is provided by the Ansible Role: -- Infrastructure-as-Code type (Ansible or Ansible to Terraform) -- Infrastructure Platform -- Host Specification Dictionary, containing 1..n Plans -- Host OS Image Dictionary - -Dependent on the choices made by the end user, host/s will be provisioend to the target Infrastructure Platform. - -## Scope - -The code modularity and commonality of provisioning enables a wide gamut of SAP Software Solution Scenarios to be deployed to many Infrastructure Platforms with differing configuration. - -### Available Infrastructure Platforms - -- AWS EC2 Virtual Server instance/s -- Google Cloud Compute Engine Virtual Machine/s -- IBM Cloud, Intel Virtual Server/s -- IBM Cloud, Power Virtual Server/s -- Microsoft Azure Virtual Machine/s -- IBM PowerVM Virtual Machine/s _(formerly LPAR/s)_ -- OVirt Virtual Machine/s (e.g. Red Hat Enterprise Linux KVM) -- KubeVirt Virtual Machine/s (e.g. SUSE Rancher with Harvester HCI) `[Experimental]` -- Red Hat OpenShift Virtualization `[Experimental]` -- VMware vSphere Virtual Machine/s `[Beta]` - -### Known issues - -- VMware REST API combined with cloud-init is unstable, `userdata` configuration may not execute and provisioning will fail - - -## Requirements - -### Target Infrastructure Platform + +## Prerequisites (Control Node) +The prerequisites are listed only for Control Node, because Managed Nodes are provisioned during runtime. For a list of requirements and recommended authorizations on each Infrastructure Platform, please see the separate [Infrastructure Platform Guidance](./PLATFORM_GUIDANCE.md) document and the drop-down for each different Infrastructure Platform. -### Target hosts - -**OS Versions:** -- Red Hat Enterprise Linux 8.0+ -- SUSE Linux Enterprise Server 15 SP0+ - -### Execution/Controller host - -**Dependencies:** -- OS Packages - - Python 3.9.7+ (i.e. CPython distribution) - - IBM Cloud CLI _(when High Availability on IBM Cloud)_ - - Terraform 1.0.0-1.5.5 _(when Ansible to Terraform, or legacy Ansible Collection for IBM Cloud)_ -- Python Packages - - `requests` 2.0+ - - `passlib` 1.7+ - - `jmespath` 1.0.1+ - - `boto3` for Amazon Web Services - - `google-auth` for Google Cloud - - `https://raw.githubusercontent.com/ansible-collections/azure/dev/requirements-azure.txt` for Microsoft Azure - - `openstacksdk` for IBM PowerVM - - `ovirt-engine-sdk-python` for OVirt - - `aiohttp` for VMware - - `kubernetes` for Kubernetes based platforms such as Red Hat OpenShift Virtualization -- Ansible - - Ansible Core 2.12.0+ - - Ansible Collections: - - `amazon.aws` - - `azure.azcollection` - - `cloud.common` - - `cloud.terraform` - - `community.aws` - - `google.cloud` - - `ibm.cloudcollection` - - _(legacy, to be replaced with `ibm.cloud` in future)_ - - `kubevirt.core` for kubevirt_vm or Red Hat OpenShift Virtualization - - `openstack.cloud` - - `ovirt.ovirt` - - `vmware.vmware_rest` _(requires `cloud.common`)_ - - `community.okd` for Red Hat OpenShift Virtualization - -TODO: Split up above dependencies per platform. - +### Base Prerequisites +For list of all collection prerequisites, please see [Ansible Collection Readme](https://github.com/sap-linuxlab/community.sap_infrastructure/blob/main/README.md#requirements) +- Operating System packages: + - Python 3.11 or higher + - Terraform 1.0.0 to 1.5.5 _(when Ansible to Terraform, or legacy Ansible Collection for IBM Cloud)_ +- Python libraries and modules: + - `ansible-core` 2.16 or higher + - `requests` 2.0 or higher + - `passlib` 1.7 or higher + - `jmespath` 1.0.1 or higher +- Ansible Collections: + - `cloud.common` + - `cloud.terraform` When `Ansible to Terraform` is used. + +### Amazon Web Services (AWS) Prerequisites +- Python libraries and modules: + - `boto3` +- Ansible Collections: + - `amazon.aws` + - `community.aws` - Optional, as AWS is moving Ansible Modules from `community.aws` to `amazon.aws`. + +### Google Cloud (GCP) Prerequisites +- Python libraries and modules: + - `google-auth` +- Ansible Collections: + - `google.cloud` + +### Microsoft Azure Prerequisites +- Python libraries and modules: + - The list is maintained at [Azure Collection github](https://github.com/ansible-collections/azure/blob/dev/requirements.txt) + - Installation steps: + - Download file [in raw format](https://raw.githubusercontent.com/ansible-collections/azure/refs/heads/dev/requirements.txt) + - Install using pip `pip3 install -r requirements.txt` + - **NOTE:** Some requirements can be in conflict with other Infrastructure Platforms. We recommend installing Microsoft Azure a separate Python Virtual Environment. +- Ansible Collections: + - `azure.azcollection` + +### IBM Cloud Prerequisites +- Operating System packages: + - IBM Cloud CLI +- Ansible Collections: + - `ibm.cloudcollection` _(legacy, to be replaced with `ibm.cloud` in future)_ + +### IBM PowerVC Prerequisites +- Python libraries and modules: + - `openstacksdk` + +### KubeVirt Prerequisites +- Python libraries and modules: + - `kubernetes` +- Ansible Collections: + - `kubevirt.core` + +### OVirt Prerequisites +- Python libraries and modules: + - `ovirt-engine-sdk-python` +- Ansible Collections: + - `ovirt.ovirt` + +### VMware Prerequisites +- Python libraries and modules: + - `aiohttp` +- Ansible Collections: + - `vmware.vmware_rest` + ## Execution - -### Sample execution - -For further information, see the [sample Ansible Playbooks in `/playbooks`](../playbooks/). - -### Suggested execution sequence - -Prior to execution of this Ansible Role, there are no Ansible Roles suggested to be executed first. - -### Summary of execution flow - -- Define target Host/s Specifications with a 'plan' name (e.g. `test1_256gb_memory` containing 1 host of 256GB Memory for SAP HANA and 1 host for SAP NetWeaver); append to the Host Specification Dictionary -- Define target Host OS Image Dictionary, or use defaults provided for each Cloud Hyperscaler. -- Execute with chosen: - - Infrastructure-as-Code method (Ansible or Ansible to Terraform) using variable `sap_vm_provision_iac_type` - - Infrastructure Platform target using variable `sap_vm_provision_iac_platform` - - Selected plan using variable `sap_vm_provision_host_specification_plan` referring to the definition in the Host Specification Dictionary - - Variables specific to each Infrastructure Platform (e.g. `sap_vm_provision_aws_access_key`) - - Include files from subdirectory based upon chosen method and target (e.g. `/tasks/platform_ansible_to_terraform/aws_ec2_vs/`) -- Provision host/s -- Add hosts to Ansible Inventory Groups defined by the Host Specification Dictionary `sap_host_type` variable _(e.g. hana_primary, hana_secondary, nwas_ascs, nwas_ers, nwas_pas, nwas_aas, anydb_primary, anydb_secondary)_
- **NOTE:** Group names can be customized using `sap_vm_provision_group_*` variables in `vars/default.yml` (e.g. `sap_vm_provision_group_hana_primary`, `sap_vm_provision_group_nwas_ascs`, etc.). -- Perform additional tasks for host/s (e.g. DNS Records, /etc/hosts, register OS for Packages, register Web Forward Proxy) -- Set variables if other Ansible Roles are to be executed (e.g. variables for Ansible Roles in the `sap_install` Ansible Collection) -- Perform any tasks for High Availability (execution dependent on hosts in Ansible Inventory Groups) -- **POST:** Re-execute Ansible Role with variable `sap_vm_provision_iac_post_deployment: true` to update High Availability configurations using Load Balancer (i.e. LB Health Check Port moved to Linux Pacemaker listener) - - -### Required structure in Ansible Playbook - -_**CRITICAL NOTE**_ - -To provide parallelisation of provisioning, the following structure must be used to dynamically create an Ansible Inventory Group for the requested hostnames. Without this necessary pre-task, the Ansible Role will not function. - -> Design decision note: This required structure avoids the Ansible Role using a sequential loop, where each host will execute all Ansible Tasks before the next host is provisioned; or using an async loop which hides all Ansible Task output from the end user. - -This required structure will: - -- In the first Ansible Play using `localhost`, dynamically create an Ansible Inventory with the hostnames listed parsed from the Ansible Dictionary (variable named `sap_vm_provision_XYZ_host_specifications_dictionary` dependent on the Infrastructure Platform) -- In the second Ansible Play use the dynamic Ansible Inventory `sap_vm_provision_target_inventory_group`, create an Ansible Play Batch containing each target host in the dynamic Ansible Inventory, which will then execute all proceeding Ansible Tasks in parallel for each target host. - -**Structure to execute sap_vm_provision:** - + +A series of choices are deciding Ansible Role behavior: +- Infrastructure-as-Code Type `sap_vm_provision_iac_type` - Defines the provisioning method. +- Infrastructure Platform `sap_vm_provision_iac_platform` - Defines the target Infrastructure Platform. +- Host Specification Dictionary - Defines the definition of provisioned SAP system hosts. + +### Supported Infrastructure Platforms +- AWS EC2 Virtual Server instance +- Google Cloud Compute Engine Virtual Machines +- IBM Cloud, Intel Virtual Servers +- IBM Cloud, Power Virtual Servers +- Microsoft Azure Virtual Machines +- IBM PowerVM Virtual Machines _(formerly LPAR)_ +- OVirt Virtual Machines `[Experimental]` +- KubeVirt Virtual Machines `[Experimental]` (e.g. Red Hat OpenShift Virtualization) +- VMware vSphere Virtual Machines `[Experimental]` + + +### Execution Flow + +1. Assert that required inputs were provided. +2. Load Infrastructure Platform specific variables. +3. Provision hosts on selected Infrastructure Platform. +4. Create Ansible Inventory during runtime, based on the variable `sap_host_type` defined in Host Specification Dictionary. +5. Configure hosts (e.g. DNS Records, `/etc/hosts`, register OS for Packages, register Web Forward Proxy). +6. Provision High Availability resources, when required. +7. Set variables if other Ansible Roles are to be executed (e.g. variables for Ansible Roles in the `sap_install` Ansible Collection). +8. Remove temporary High Availability configurations (i.e. LB Health Check Port moved to Linux Pacemaker listener) when executed with variable `sap_vm_provision_iac_post_deployment: true`. + + +### Example + +The playbooks using this Ansible Role are required to dynamically create Ansible Inventory group during runtime, which will allow parallel provisioning of resources. + +**Reasoning behind this concept:** This required structure avoids the Ansible Role using a sequential loop, where each host will execute all Ansible Tasks before the next host is provisioned; or using an async loop which hides all Ansible Task output from the end user. + +For more examples on how to use this role in different installation scenarios, refer to the [ansible.playbooks_for_sap](https://github.com/sap-linuxlab/ansible.playbooks_for_sap) playbooks. +- These playbooks include Parallelization concept explained above. + +Example for `aws_ec2_vs`: ```yaml - name: Ansible Play to create dynamic inventory group for provisioning hosts: localhost @@ -145,8 +139,7 @@ This required structure will: ansible.builtin.add_host: name: "{{ item }}" group: sap_vm_provision_target_inventory_group - # Adjust var name in loop (i.e. replace _XYZ_ to the correct Ansible Dictionary) - loop: "{{ sap_vm_provision_XYZ_host_specifications_dictionary[sap_vm_provision_host_specification_plan].keys() }}" + loop: "{{ sap_vm_provision_aws_ec2_vs_host_specifications_dictionary[sap_vm_provision_host_specification_plan].keys() }}" - name: Ansible Play to provision hosts for SAP hosts: sap_vm_provision_target_inventory_group # Ansible Play target hosts pattern, use dynamic Inventory Group @@ -157,7 +150,7 @@ This required structure will: ansible.builtin.include_role: name: community.sap_infrastructure.sap_vm_provision -- name: Ansible Play for verify provisioned hosts for SAP +- name: Ansible Play for remaining tasks on provisioned hosts hosts: all tasks: @@ -165,31 +158,94 @@ This required structure will: ansible.builtin.debug: var: groups ``` +Explanation of workflow: +1. First play: `Ansible Play to create dynamic inventory group for provisioning` + - Control Node will create new Ansible Inventory group `sap_vm_provision_target_inventory_group` with hosts defined in the variable `sap_vm_provision_aws_ec2_vs_host_specifications_dictionary` under chosen plan `sap_vm_provision_host_specification_plan`. +2. Second play: `Ansible Play to provision hosts for SAP` + - Provisioning tasks are virtually executed on non-existent hosts, but Ansible Role executes provisioning with `delegate_to` Control Node. + - Configuration tasks after provisioning are executed on newly provisioned hosts. +3. Third play: `Ansible Play for remaining tasks on provisioned hosts` + - Example of how newly provisioned hosts can be targeted with additional tasks (e.g. SAP Installation). + +For further information, see the [sample Ansible Playbooks in `/playbooks`](../playbooks/). + -### Design assumptions with execution impact + + + +## Further Information - For Hyperscaler Cloud Service Providers that use Resource Groups (IBM Cloud, Microsoft Azure): - Virtual Machine and associated resources (Disks, Network Interfaces, Load Balancer etc.) will be provisioned to the same Resource Group as the targeted network/subnet. - Optional: Private DNS may be allocated to another Resource Group, and an optional variable is provided for this. - Virtual Disk with defined IOPS is only possible on AWS, Google Cloud, IBM Cloud -### Tags to control execution - -There are no tags used to control the execution of this Ansible Role - +### Known issues +- VMware REST API combined with cloud-init is unstable, `userdata` configuration may not execute and provisioning will fail + ## License - + Apache 2.0 - - -## Authors - -Sean Freeman -Nils Koenig (nkoenig@redhat.com) kubevirt_vm / Red Hat OpenShift Virtualization - ---- - -## Ansible Role Input Variables - -Please first check the [/defaults parameters file](./defaults/main.yml). + + +## Maintainers + +- [Sean Freeman](https://github.com/sean-freeman) +- [Marcel Mamula](https://github.com/marcelmamula) +- [Nils Koenig](https://github.com/newkit) - kubevirt_vm / Red Hat OpenShift Virtualization + + +## Role Variables + +The list of all available variables: [/defaults parameters file](./defaults/main.yml). + +**Following key variables are required.** + +### sap_vm_provision_iac_type +- _Type:_ `string`
+- _Choices:_ `ansible , ansible_to_terraform`
+ +Defines the provisioning method.
+ +### sap_vm_provision_iac_platform +- _Type:_ `string`
+- _Choices:_ `aws_ec2_vs , gcp_ce_vm , ibmcloud_vs , ibmcloud_powervs , msazure_vm , ibmpowervm_vm , kubevirt_vm , ovirt_vm , vmware_vm`
+ +Defines the target Infrastructure Platform.
+ +### Host Specification Dictionary +- _Type:_ `dict`
+- _Default:_ Default value is defined, but it has to be customized to represent required SAP system.
+ +Defines the definition of provisioned SAP system hosts.
+This variable name is unique for each Infrastructure Platform. Example: `sap_vm_provision_aws_ec2_vs_host_specifications_dictionary` for `aws_ec2_vs`.
+Customization options:
+- Adjust existing plan or add new (Selected by variable `sap_vm_provision_host_specification_plan`). +- Adjust number of hosts and their sizing. +- Adjust the variable `sap_host_type` to customize Ansible Inventory groups. **NOTE:** Group names can be customized using `sap_vm_provision_group_*` variables in `vars/default.yml` (e.g. `sap_vm_provision_group_hana_primary`, `sap_vm_provision_group_nwas_ascs`, etc.). +- Adjust filesystems (size, type, source, etc.).yes + +### Host OS Image Dictionary +- _Type:_ `list`
+- _Default:_ Defined for each supported Cloud platform. + +Defines list of predefined OS Images for each supported Cloud Platform. +This variable name is unique for each Infrastructure Platform. Example: `sap_vm_provision_aws_ec2_vs_host_os_image_dictionary` for `aws_ec2_vs`.
+Chosen OS Image is selected by variable unique variable for each Infrastructure Platform. Example: `sap_vm_provision_aws_ec2_vs_host_os_image` for `aws_ec2_vs`.
+Customization options:
+- Adjust existing or add new OS images that are available. + +### Credentials + +Each Infrastructure Platform has list of required variables defined in [/defaults parameters file](./defaults/main.yml). +Example for `aws_ec2_vs`: +- `sap_vm_provision_aws_access_key` +- `sap_vm_provision_aws_secret_access_key` +- `sap_vm_provision_aws_region` +- `sap_vm_provision_aws_vpc_availability_zone` +- `sap_vm_provision_aws_vpc_subnet_id` +- `sap_vm_provision_aws_vpc_sg_names` +- `sap_vm_provision_aws_key_pair_name_ssh_host_public_key` + + diff --git a/roles/sap_vm_provision/defaults/main.yml b/roles/sap_vm_provision/defaults/main.yml index 92c24797..5e9e5571 100644 --- a/roles/sap_vm_provision/defaults/main.yml +++ b/roles/sap_vm_provision/defaults/main.yml @@ -266,13 +266,8 @@ sap_vm_provision_ibmpowervm_key_pair_name_ssh_host_public_key: "" sap_vm_provision_ibmpowervm_placement_resource_name: "sap-collocation-rule-spread" sap_vm_provision_ibmpowervm_placement_strategy_spread: false -# Kubevirt -sap_vm_provision_kubevirt_api_key: "" -sap_vm_provision_kubevirt_cluster_url: "" -sap_vm_provision_kubevirt_vm_host_os_image_url: "" # e.g. docker://registry.redhat.io/rhel8/rhel-guest-image:8.6.0 -sap_vm_provision_kubevirt_os_user: "" -sap_vm_provision_kubevirt_os_user_password: "" -sap_vm_provision_kubevirt_target_namespace: "" +# Kubevirt, see below +# sap_vm_provision_kubevirt_vm_kubeconfig: /path/to/clusterconfigs/kubeconfig # OVirt sap_vm_provision_ovirt_engine_cafile: "" @@ -782,6 +777,11 @@ sap_vm_provision_ibmpowervm_vm_host_specifications_dictionary: # kubevirt / Red Hat OpenShift Virtualization # ####################################################### +# kubeconfig for Red Hat OpenShift cluster connection. +# If not provided, the kubeconfig will be read from the environment variables +# KUBECONFIG or K8S_AUTH_KUBECONFIG +# sap_vm_provision_kubevirt_vm_kubeconfig: /path/to/clusterconfigs/kubeconfig + # Namespace where the VM should be created in sap_vm_provision_kubevirt_vm_target_namespace: sap @@ -791,50 +791,22 @@ sap_vm_provision_kubevirt_vm_os_user: cloud-user # Password for the above user sap_vm_provision_kubevirt_vm_os_user_password: "" -# how to authenticate to the guest vm [password|private_key|private_key_data] -# password: uses provided password in sap_vm_provision_kubevirt_vm_os_user_password, make sure your ssh config allows password authentication -# private_key: use the private ssh key at the location defined by sap_vm_provision_ssh_host_private_key_file_path -# private_key_data: use the private ssh key provided in sap_vm_provision_ssh_host_private_key_data and write it to the location defined in sap_vm_provision_ssh_host_private_key_file_path -sap_vm_provision_kubevirt_vm_guest_ssh_auth_mechanism: private-key - -# Private SSH key file, must be accessible on the ansible controller -# sap_vm_provision_ssh_host_private_key_file_path: - -# private ssh key, make sure the indentation is correct, here it's two spaces at the beginning of every line -# sap_vm_provision_ssh_host_private_key_data: | -# < your key data> - -# Should the CA cert and the API endpoint be extracted from the kubeconfig file? -sap_vm_provision_kubevirt_vm_extract_kubeconfig: true - -# Should an existing VM be overwritten? -sap_vm_provision_kubevirt_vm_overwrite_vm: false - -# Kubeconfig file for cluster where VMs should be created -sap_vm_provision_kubevirt_vm_kubeconfig_path: /path/to/clusterconfigs/kubeconfig - -# In order to use secured communication, provide the CA cert bundle for the cluster. -# This can be extracted from the kubeconfig file with the following command from the -# kubeconfig file: -# grep certificate-authority-data ${KUBECONFIG} | awk '{ print $2 }' | base64 --decode > cluster-ca-cert.pem -# This variable will not be used if sap_vm_provision_kubevirt_vm_extract_kubeconfig = true -# sap_vm_provision_kubevirt_vm_ca_cert: /path/to/clusterconfigs/cluster-ca-cert.pem - -# API endpoint of the cluster -# This variable will not be used if sap_vm_provision_kubevirt_vm_extract_kubeconfig = true -# sap_vm_provision_kubevirt_api_vm_endpoint: https://api.cluster.domain.tld:6443 - -# Admin username for the cluster communication -sap_vm_provision_kubevirt_vm_admin_username: kubeadmin - -# Password for the above admin user -sap_vm_provision_kubevirt_vm_admin_password: AAAAA-BBBBB-CCCCC-DDDDD - # RAM Overhead [GiB] for virt-launcher container, this can be small for VMs < 1 TB and without SRIOV but should be increased to 16 or more for VMs > 1TB sap_vm_provision_kubevirt_vm_container_memory_overhead: 1 -# hostname of the ansible controller -sap_vm_provision_kubevirt_vm_ansible_controller: localhost # on AAP, this is localhost +# CPU performance settings which are applied to VM +sap_vm_provision_kubevirt_vm_performance_cpu_settings: + dedicatedCpuPlacement: true + model: host-passthrough + numa: + guestMappingPassthrough: {} + features: + - name: x2apic + policy: require + - name: rdtscp + policy: require + - name: invtsc + policy: require sap_vm_provision_kubevirt_vm_host_specifications_dictionary: example_host_specification_plan: @@ -848,11 +820,17 @@ sap_vm_provision_kubevirt_vm_host_specifications_dictionary: # Provide either an existing PVC or a URL for an OS image os_image: # either url or source_pvc_name have to be provided # URL for an image to be used - url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.8.0" + #url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.8.0" + #url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.10.0" + #url: "docker://registry.redhat.io/rhel9/rhel-guest-image:9.4" + url: "docker://registry.redhat.io/rhel9/rhel-guest-image:9.6" + #url: "docker://registry.redhat.io/rhel10/rhel-guest-image:10.0" # Name for a PVC to be cloned # source_pvc_name: "rhel-8.8" namespace: openshift-virtualization-os-images size: "50Gi" + access_modes: # e.g. ReadWriteMany | ReadWriteOnce + - ReadWriteMany network_definition: - name: sapbridge type: bridge @@ -863,24 +841,25 @@ sap_vm_provision_kubevirt_vm_host_specifications_dictionary: mountpoint: /hana disk_count: 1 # default: 1 disk_size: 2048 # size in GB, integer - disk_type: nas # KubeVirt Storage Class + disk_type: sapstorage # KubeVirt Storage Class + access_modes: # e.g. ReadWriteMany | ReadWriteOnce + - ReadWriteMany cloudinit: userData: |- #cloud-config timezone: Europe/Berlin - hostname: "{{ scaleout_origin_host_spec }}" + hostname: host1 user: {{ sap_vm_provision_kubevirt_vm_os_user if sap_vm_provision_kubevirt_vm_os_user is defined }} password: {{ sap_vm_provision_kubevirt_vm_os_user_password if sap_vm_provision_kubevirt_vm_os_user_password is defined }} chpasswd: expire: false ssh_authorized_keys: - - "{{ lookup('ansible.builtin.file', sap_vm_provision_ssh_host_public_key_file_path ) }}" + - "{{ lookup('ansible.builtin.file', sap_vm_provision_ssh_host_public_key_file ) if sap_vm_provision_ssh_host_public_key_file is defined }}" networkData: |- - network: - version: 2 - ethernets: - eth0: - dhcp4: true + version: 2 + ethernets: + eth0: + dhcp4: true # OVirt sap_vm_provision_ovirt_vm_boot_menu: false diff --git a/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_main.yml b/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_main.yml index 19815f62..c32b777e 100644 --- a/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_main.yml +++ b/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_main.yml @@ -1,11 +1,4 @@ --- -- name: Fail if sap_vm_provision_kubevirt_vm_os_user_password is not set and sap_vm_provision_kubevirt_vm_guest_ssh_auth_mechanism is set to password - ansible.builtin.fail: - msg: Password is not allowed to be empty or undefined (sap_vm_provision_kubevirt_vm_os_user_password). - when: - - sap_vm_provision_kubevirt_vm_guest_ssh_auth_mechanism == "password" - - sap_vm_provision_kubevirt_vm_os_user_password == "" or sap_vm_provision_kubevirt_vm_os_user_password == null - - name: Ansible Task block for looped provisioning of KubeVirt Virtual Machines any_errors_fatal: true # Using environment, no_log is ineffective and log will show 'EXEC /bin/sh -c 'ENV_VAR=value python3 /AnsiballZ_ansible_module_name.py && sleep 0' @@ -68,24 +61,14 @@ - not lookup('ansible.builtin.vars', loop_item, default='') is skipped - lookup('ansible.builtin.vars', loop_item, default='') is failed -- name: Write private ssh key to ansible_controller - delegate_to: "{{ sap_vm_provision_kubevirt_vm_ansible_controller }}" - no_log: true - ansible.builtin.copy: - dest: "{{ sap_vm_provision_ssh_host_private_key_file_path }}" - content: "{{ sap_vm_provision_ssh_host_private_key_data }}" - mode: "0600" - when: sap_vm_provision_kubevirt_vm_guest_ssh_auth_mechanism == "private_key_data" - - name: Ansible Task block to execute on target inventory hosts remote_user: "{{ sap_vm_provision_kubevirt_vm_os_user }}" become: true become_user: root delegate_to: "{{ inventory_hostname }}" vars: - ansible_password: "{{ sap_vm_provision_kubevirt_vm_os_user_password }}" ansible_ssh_private_key_file: "{{ sap_vm_provision_ssh_host_private_key_file_path }}" - ansible_ssh_common_args: "-o ConnectTimeout=180 -o ControlMaster=auto -o ControlPersist=3600s -o UserKnownHostsFile=/dev/null -o ForwardX11=no -o ProxyJump={{ __sap_vm_provision_kubevirt_vm_register_execution_host_user }}@{{ sap_vm_provision_execution_host }}" + ansible_ssh_common_args: "-o StrictHostKeyChecking=no -o ConnectTimeout=180 -o ControlMaster=auto -o ControlPersist=3600s -o UserKnownHostsFile=/dev/null -o ForwardX11=no -o ProxyJump={{ __sap_vm_provision_kubevirt_vm_register_execution_host_user }}@{{ sap_vm_provision_execution_host }}" block: @@ -132,13 +115,3 @@ - name: Register Package Repositories ansible.builtin.include_tasks: file: common/register_os.yml - - always: - - - name: Delete private ssh key from ansible_controller - delegate_to: "{{ sap_vm_provision_register_ansible_controller }}" - become: false - ansible.builtin.file: - path: "{{ sap_vm_provision_ssh_host_private_key_file_path }}" - state: absent - when: sap_vm_provision_kubevirt_vm_guest_ssh_auth_mechanism == "private_key_data" diff --git a/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_provision.yml b/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_provision.yml index 6ee29206..1b17553b 100644 --- a/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_provision.yml +++ b/roles/sap_vm_provision/tasks/platform_ansible/kubevirt_vm/execute_provision.yml @@ -34,7 +34,7 @@ }, }, 'storage' : { - 'accessModes': ['ReadWriteMany'], + 'accessModes': __sap_vm_provision_register_vm_config.os_image.access_modes | d(['ReadWriteMany']), 'resources': { 'requests': { 'storage': __sap_vm_provision_register_vm_config.os_image.size @@ -61,7 +61,7 @@ }, }, 'storage' : { - 'accessModes': ['ReadWriteMany'], + 'accessModes': __sap_vm_provision_register_vm_config.os_image.access_modes | d(['ReadWriteMany']), 'resources': { 'requests': { 'storage': __sap_vm_provision_register_vm_config.os_image.size @@ -81,24 +81,24 @@ storage_disks_map: |- {% set disks_map = [] -%} {% for storage_item in __sap_vm_provision_register_vm_config.storage_definition -%} - {% set vol = disks_map.extend([ - { - 'metadata': { 'name': (__sap_vm_provision_register_vm_name + '-' + storage_item.name | replace('_', '-')) }, - 'spec' : { - 'source' : { - 'blank' : {} - }, - 'storage' : { - 'accessModes': ['ReadWriteMany'], - 'resources': { - 'requests': { - 'storage': ((storage_item.disk_size | default(0)) | string) + 'Gi' - } - }, - 'storageClassName': storage_item.disk_type | default('') + {% set vol = disks_map.extend([ + { + 'metadata': { 'name': (__sap_vm_provision_register_vm_name + '-' + storage_item.name | replace('_', '-')) }, + 'spec' : { + 'source' : { + 'blank' : {} + }, + 'storage' : { + 'accessModes': storage_item.access_modes | d(['ReadWriteMany']), + 'resources': { + 'requests': { + 'storage': ((storage_item.disk_size | default(0)) | string) + 'Gi', } - } - }]) %} + }, + 'storageClassName': storage_item.disk_type | default(''), + } + } + }]) %} {%- endfor %} {{ disks_map }} @@ -243,23 +243,17 @@ cpu: cores: "{{ __sap_vm_provision_register_vm_config.kubevirt_vm_cpu_cores }}" threads: "{{ __sap_vm_provision_register_vm_config.kubevirt_vm_cpu_smt }}" - dedicatedCpuPlacement: true - model: host-passthrough - numa: - guestMappingPassthrough: {} - features: - - name: x2apic - policy: require - - name: rdtscp - policy: require - - name: invtsc - policy: require memory: guest: "{{ __sap_vm_provision_register_vm_config.kubevirt_vm_memory_gib }}Gi" hugepages: pageSize: 1Gi +- name: Apply CPU performance settings + ansible.builtin.set_fact: + __sap_vm_provision_register_vm_deploy_config: >- + {{ __sap_vm_provision_register_vm_deploy_config | combine({'domain': { 'cpu': __sap_vm_provision_register_vm_deploy_config.domain.cpu | combine(sap_vm_provision_kubevirt_vm_performance_cpu_settings)}}, recursive=True) }} + - name: Provision KubeVirt Virtual Machine kubevirt.core.kubevirt_vm: api_version: "{{ api_version | default(omit) }}" diff --git a/roles/sap_vm_temp_vip/INPUT_PARAMETERS.md b/roles/sap_vm_temp_vip/INPUT_PARAMETERS.md deleted file mode 100644 index 6ef41929..00000000 --- a/roles/sap_vm_temp_vip/INPUT_PARAMETERS.md +++ /dev/null @@ -1,66 +0,0 @@ -## Input Parameters for sap_vm_temp_vip Ansible Role - -### sap_vm_temp_vip_default_ip - -- _Type:_ `string` -- _Default:_ `ansible_default_ipv4.address` - -IP Address of default network interface is obtained from Ansible Facts and it is used for calculation of missing input parameters. - -### sap_vm_temp_vip_default_netmask - -- _Type:_ `string` -- _Default:_ `ansible_default_ipv4.netmask` - -Netmask of default network interface is obtained from Ansible Facts and it is used for calculation of missing input parameters. - -### sap_vm_temp_vip_default_prefix - -- _Type:_ `string` -- _Default:_ `ansible_default_ipv4.prefix` - -Prefix of default network interface is obtained from Ansible Facts and it is used for calculation of missing input parameters. - -### sap_vm_temp_vip_default_broadcast - -- _Type:_ `string` -- _Default:_ `ansible_default_ipv4.broadcast` - -Broadcast of default network interface is obtained from Ansible Facts and it is used for calculation of missing input parameters.
-This parameter is empty on some cloud platforms and VIP is created without broadcast if attempt to calculate fails. - -### sap_vm_temp_vip_default_interface - -- _Type:_ `string` -- _Default:_ `ansible_default_ipv4.interface` or `eth0` - -Default Network Interface name is obtained from Ansible Facts and it is used for calculation of missing input parameters.
-Ensure to use correct Network Interface if default interface from Ansible Facts does not represent desired Network Interface. - -### sap_vm_temp_vip_hana_primary -- _Type:_ `string` -- _Default:_ `sap_ha_pacemaker_cluster_vip_hana_primary_ip_address` - -Mandatory for SAP HANA cluster setup.
-VIP address is by default assigned from `sap_ha_pacemaker_cluster_vip_hana_primary_ip_address` input parameter used by [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster) role. - -### sap_vm_temp_vip_nwas_abap_ascs -- _Type:_ `string` -- _Default:_ `sap_ha_pacemaker_cluster_vip_nwas_abap_ascs_ip_address` - -Mandatory for SAP ASCS/ERS cluster setup.
-VIP address is by default assigned from `sap_ha_pacemaker_cluster_vip_nwas_abap_ascs_ip_address` input parameter used by [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster) role. - -### sap_vm_temp_vip_nwas_abap_ers -- _Type:_ `string` -- _Default:_ `sap_ha_pacemaker_cluster_vip_nwas_abap_ers_ip_address` - -Mandatory for SAP ASCS/ERS cluster setup.
-VIP address is by default assigned from `sap_ha_pacemaker_cluster_vip_hana_primary_ip_address` input parameter used by [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster) role. - -### sap_vm_temp_vip_anydb_primary -- _Type:_ `string` - -Mandatory for SAP AnyDB cluster setup. - - \ No newline at end of file diff --git a/roles/sap_vm_temp_vip/README.md b/roles/sap_vm_temp_vip/README.md index e7df2c42..9ef9d3c2 100644 --- a/roles/sap_vm_temp_vip/README.md +++ b/roles/sap_vm_temp_vip/README.md @@ -1,10 +1,11 @@ # sap_vm_temp_vip Ansible Role +![Ansible Lint for sap_vm_temp_vip](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_temp_vip.yml/badge.svg) ## Description -Ansible role `sap_vm_temp_vip` is used to enable installation of SAP Application and Database on High Availability clusters provisioned by [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) role. +The Ansible role `sap_vm_temp_vip` is used to enable installation of SAP Application and Database on High Availability clusters provisioned by [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) role. Installation of cluster environment requires temporary assignment of Virtual IP (VIP) before executing installation roles [sap_hana_install](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_hana_install) and [sap_swpm](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_swpm). - This is temporary and it will be replaced by Cluster VIP resource once cluster is configured by [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster) role. @@ -12,21 +13,33 @@ Installation of cluster environment requires temporary assignment of Virtual IP This role does not update `/etc/hosts` or DNS records, as these steps are performed by the [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) role. + +## Dependencies +- `community.sap_infrastructure` + - Roles: + - `sap_vm_provision` + - Reason: This role is expected to run after provisioning of resources by Ansible Role [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision). + + + ## Prerequisites Environment: - Assign hosts to correct groups, which are also used in other roles in our project - Supported cluster groups: `hana_primary, hana_secondary, anydb_primary, anydb_secondary, nwas_ascs, nwas_ers` - -Role dependency: -- [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision), for creating required resources: DNS, Load Balancers and Health Checks. ## Execution -Role can be execute separately or as part of [ansible.playbooks_for_sap](https://github.com/sap-linuxlab/ansible.playbooks_for_sap) playbooks. + +### Recommended +It is recommended to execute this role together with other roles in this collection, in the following order:
+1. [sap_vm_provision](https://github.com/sap-linuxlab/community.sap_infrastructure/tree/main/roles/sap_vm_provision) +2. *`sap_vm_temp_vip`* + + ### Execution Flow 1. Assert that required inputs were provided. @@ -61,6 +74,8 @@ Role can be execute separately or as part of [ansible.playbooks_for_sap](https:/ +## Further Information +For more examples on how to use this role in different installation scenarios, refer to the [ansible.playbooks_for_sap](https://github.com/sap-linuxlab/ansible.playbooks_for_sap) playbooks. ## License @@ -74,5 +89,63 @@ Apache 2.0 - [Marcel Mamula](https://github.com/marcelmamula) -## Role Input Parameters -All input parameters used by role are described in [INPUT_PARAMETERS.md](https://github.com/sap-linuxlab/community.sap_infrastructure/blob/main/roles/sap_vm_temp_vip/INPUT_PARAMETERS.md) +## Role Variables + +### sap_vm_temp_vip_default_ip +- _Type:_ `string` +- _Default:_ `ansible_default_ipv4.address` + +Specifies the IP Address of the default network interface. + +### sap_vm_temp_vip_default_netmask +- _Type:_ `string` +- _Default:_ `ansible_default_ipv4.netmask` + +Specifies the Netmask of the default network interface. + +### sap_vm_temp_vip_default_prefix +- _Type:_ `string` +- _Default:_ `ansible_default_ipv4.prefix` + +Specifies the prefix of the default network interface. + +### sap_vm_temp_vip_default_broadcast +- _Type:_ `string` +- _Default:_ `ansible_default_ipv4.broadcast` + +Specifies the broadcast of the default network interface.
+This parameter is empty on some cloud platforms and VIP is created without broadcast if attempt to calculate fails. + +### sap_vm_temp_vip_default_interface +- _Type:_ `string` +- _Default:_ `ansible_default_ipv4.interface` or `eth0` + +Specifies the default network interface name.
+Ensure to use correct network interface if default interface from Ansible Facts does not represent desired network interface. + +### sap_vm_temp_vip_hana_primary +- _Type:_ `string` +- _Default:_ `sap_ha_pacemaker_cluster_vip_hana_primary_ip_address` + +This variable is mandatory for SAP HANA cluster setup.
+The VIP address is by default assigned from `sap_ha_pacemaker_cluster_vip_hana_primary_ip_address` input parameter used by Ansible Role [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster). + +### sap_vm_temp_vip_nwas_abap_ascs +- _Type:_ `string` +- _Default:_ `sap_ha_pacemaker_cluster_vip_nwas_abap_ascs_ip_address` + +This variable is mandatory for SAP ASCS/ERS cluster setup.
+The VIP address is by default assigned from `sap_ha_pacemaker_cluster_vip_nwas_abap_ascs_ip_address` input parameter used by Ansible Role [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster). + +### sap_vm_temp_vip_nwas_abap_ers +- _Type:_ `string` +- _Default:_ `sap_ha_pacemaker_cluster_vip_nwas_abap_ers_ip_address` + +This variable is mandatory for SAP ASCS/ERS cluster setup.
+The VIP address is by default assigned from `sap_ha_pacemaker_cluster_vip_hana_primary_ip_address` input parameter used by Ansible Role [sap_ha_pacemaker_cluster](https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster). + +### sap_vm_temp_vip_anydb_primary +- _Type:_ `string` + +This variable is mandatory for SAP AnyDB cluster setup. + diff --git a/roles/sap_vm_verify/README.md b/roles/sap_vm_verify/README.md index 34f98e71..39baee84 100644 --- a/roles/sap_vm_verify/README.md +++ b/roles/sap_vm_verify/README.md @@ -1,6 +1,7 @@ `WIP` # sap_vm_verify Ansible Role +![Ansible Lint for sap_vm_verify](https://github.com/sap-linuxlab/community.sap_infrastructure/actions/workflows/ansible-lint-sap_vm_verify.yml/badge.svg) Ansible Role for verification of Virtual Machine state and readiness to perform SAP Software installation.