From 87116ea86ebe3a0388479e5e4f46c07035f2695a Mon Sep 17 00:00:00 2001 From: Adam DeConinck Date: Thu, 9 Dec 2021 21:58:07 +0000 Subject: [PATCH 1/5] Add Molecule testing for Singularity, plus infra for more roles Summary ------- - Fixes the ability to install Singularity on EL8 by enabling the necessary repos - Move the pre-execution logic for the Singularity Galaxy role into a new DeepOps role, singularity_wrapper - Modify the playbook to use singularity_wrapper - Add Molecule tests to singularity_wrapper role - Add a Github Action configuration to execute this and other Molecule tests of DeepOps roles - Add documentation to `docs/deepops/testing.md` to describe these tests and provide instructions on adding more Test plan --------- Clean execution of the Github action for this role Future work ----------- This change adds a lot of infrastructure that should make it easier to add testing to additional roles. Future changes should enable testing of more of the roles in DeepOps, but I didn't want to get *too* carried away in this one. :wink: --- .github/workflows/molecule.yml | 33 ++++++++ docs/deepops/testing.md | 76 ++++++++++++++++++- playbooks/container/singularity.yml | 7 +- roles/requirements.yml | 6 +- roles/singularity_wrapper/.yamllint | 33 ++++++++ roles/singularity_wrapper/defaults/main.yml | 10 +++ roles/singularity_wrapper/meta/main.yml | 9 +++ .../molecule/default/converge.yml | 7 ++ .../molecule/default/molecule.yml | 26 +++++++ .../molecule/default/verify.yml | 10 +++ roles/singularity_wrapper/tasks/main.yml | 35 +++++++++ 11 files changed, 241 insertions(+), 11 deletions(-) create mode 100644 .github/workflows/molecule.yml create mode 100644 roles/singularity_wrapper/.yamllint create mode 100644 roles/singularity_wrapper/defaults/main.yml create mode 100644 roles/singularity_wrapper/meta/main.yml create mode 100644 roles/singularity_wrapper/molecule/default/converge.yml create mode 100644 roles/singularity_wrapper/molecule/default/molecule.yml create mode 100644 roles/singularity_wrapper/molecule/default/verify.yml create mode 100644 roles/singularity_wrapper/tasks/main.yml diff --git a/.github/workflows/molecule.yml b/.github/workflows/molecule.yml new file mode 100644 index 000000000..f4165f104 --- /dev/null +++ b/.github/workflows/molecule.yml @@ -0,0 +1,33 @@ +--- +name: test ansible roles with molecule +on: + - push + - pull_request +jobs: + build: + runs-on: ubuntu-20.04 + strategy: + max-parallel: 4 + matrix: + deepops-role: + - singularity_wrapper + steps: + - name: check out repo + uses: actions/checkout@v2 + with: + path: "${{ github.repository }}" + - name: set up python + uses: actions/setup-python@v2 + with: + python-version: "3.9" + - name: install dependencies + run: | + python3 -m pip install --upgrade pip + python3 -m pip install molecule[docker] docker ansible + - name: run molecule test + run: | + cd "${{ github.repository }}/roles" + ansible-galaxy role install --force -r ./requirements.yml + ansible-galaxy collection install --force -r ./requirements.yml + cd "${{ matrix.deepops-role }}" + molecule test diff --git a/docs/deepops/testing.md b/docs/deepops/testing.md index c09f26bd7..26c2486db 100644 --- a/docs/deepops/testing.md +++ b/docs/deepops/testing.md @@ -1,12 +1,13 @@ # DeepOps Testing, CI/CD, and Validation -## DeepOps Continuous Integration Testing + +## DeepOps end-to-end testing The DeepOps project leverages a private Jenkins server to run continuous integration tests. Testing is done using the [virtual](../../virtual) deployment mechanism. Several Vagrant VMs are created, the cluster is deployed, tests are executed, and then the VMs are destroyed. The goal of the DeepOps CI is to prevent bugs from being introduced into the code base and to identify when changes in 3rd party platforms have occurred or impacted the DeepOps deployment mechanisms. In general, K8s and Slurm deployment issues are detected and resolved with urgency. Many components of DeepOps are 3rd party open source tools that may silently fail or suddenly change without notice. The team will make a best-effort to resolve these issues and include regression tests, however there may be times where a fix is unavailable. Historically, this has been an issue with Rook-Ceph and Kubeflow, and those GitHub communities are best equipped to help with resolutions. -### Testing Methodi +### Testing Method DeepOps CI contains two types of automated tests: @@ -63,6 +64,77 @@ A short description of the nightly testing is outlined below. The full suit of t | MIG configuration | | | | No testing support +## DeepOps Ansible role testing + +A subset of the Ansible roles in DeepOps have tests defined using [Ansible Molecule](https://molecule.readthedocs.io/en/latest/). +This testing mechanism allows the roles to be tested individually, providing additional test signal to identify issues which do not appear in the end-to-end tests. +These tests are run automatically for each pull request using [Github Actions](https://github.com/NVIDIA/deepops/actions). + +Molecule testing runs the Ansible role in quesiton inside a Docker container. +As such, not all roles will be easy to test witth this mechanism. +Roles which mostly involve installing software, configuring services, or executing scripts should generally be possible to test. +Roles which rely on the presence of specific hardware (such as GPUs), which reboot the nodes they act on, or which make changes to kernel configuration are going to be harder to test with Molecule. + +### Defining Molecule tests for a new role + +To add Molecule tests to a new role, the following procedure can be used. + +1. Ensure you have Docker installed in your development environment + +2. Install Ansible Molecule in your development environment + +``` +$ python3 -m pip install "molecule[docker,lint]" +``` + +3. Initialize Molecule in your new role + +``` +$ cd deepops/roles/ +$ molecule init scenario -r --driver docker +``` + +4. In the file `molecule/default/molecule.yml`, define the list of platforms to be tested. +DeepOps currently supports operating systems based on Ubuntu 18.04, Ubuntu 20.04, EL7, and EL8. +To test these stacks, the following `platforms` stanza can be used. + +``` +platforms: + - name: ubuntu-1804 + image: geerlingguy/docker-ubuntu1804-ansible + pre_build_image: true + - name: ubuntu-2004 + image: geerlingguy/docker-ubuntu2004-ansible + pre_build_image: true + - name: centos-7 + image: geerlingguy/docker-centos7-ansible + pre_build_image: true + - name: centos-8 + image: geerlingguy/docker-centos8-ansible + pre_build_image: true +``` + +5. If you haven't already, define your role's metadata in the file `meta/main.yml`. +A sample `meta.yml` is shown here: + +``` +galaxy_info: + role_name: + namespace: deepops + author: DeepOps Team + company: NVIDIA + description: + license: 3-Clause BSD + min_ansible_version: 2.9 +``` + +6. Once this is done, verify that your role executes successfully in the Molecule environment by running `molecule test`. If you run into any issues, consult the [Molecule documentation](https://molecule.readthedocs.io/en/latest/index.html) for help resolving them. + +7. (optional) In addition to testing successful execution, you can add additional tests which will be run after your role completes in a file `molecule/default/verify.yml`. This is an Ansible playbook that will run in the same environment as your playbook ran. For a simple example of such a verify playbook, see the [Enroot role](https://github.com/NVIDIA/ansible-role-enroot/blob/master/molecule/default/verify.yml). + +8. Once you're confident that your new tests are all passing, add your role to the `deepops-role` section in the `.github/workflows/molecule.yml` file. + + ## DeepOps Deployment Validation The Slurm and Kubernetes deployment guides both document cluster verification steps. These should be run during the installation process to validate a GPU workload can be executed on the cluster. diff --git a/playbooks/container/singularity.yml b/playbooks/container/singularity.yml index 16ca5b9ab..74208898f 100644 --- a/playbooks/container/singularity.yml +++ b/playbooks/container/singularity.yml @@ -1,10 +1,5 @@ --- - hosts: all become: yes - pre_tasks: - - name: create a folder for go - file: - path: "{{ golang_install_dir }}" - recurse: yes roles: - - lecorguille.singularity + - singularity_wrapper diff --git a/roles/requirements.yml b/roles/requirements.yml index da9b0c789..44ba03d8b 100644 --- a/roles/requirements.yml +++ b/roles/requirements.yml @@ -61,8 +61,8 @@ roles: - src: https://github.com/OSC/ood-ansible.git version: 'v2.0.3' +- src: abims_sbr.singularity + version: 3.7.1-1 + - src: gantsign.golang version: 2.4.0 - -- src: lecorguille.singularity - version: 1.2.0 diff --git a/roles/singularity_wrapper/.yamllint b/roles/singularity_wrapper/.yamllint new file mode 100644 index 000000000..882767605 --- /dev/null +++ b/roles/singularity_wrapper/.yamllint @@ -0,0 +1,33 @@ +--- +# Based on ansible-lint config +extends: default + +rules: + braces: + max-spaces-inside: 1 + level: error + brackets: + max-spaces-inside: 1 + level: error + colons: + max-spaces-after: -1 + level: error + commas: + max-spaces-after: -1 + level: error + comments: disable + comments-indentation: disable + document-start: disable + empty-lines: + max: 3 + level: error + hyphens: + level: error + indentation: disable + key-duplicates: enable + line-length: disable + new-line-at-end-of-file: disable + new-lines: + type: unix + trailing-spaces: disable + truthy: disable diff --git a/roles/singularity_wrapper/defaults/main.yml b/roles/singularity_wrapper/defaults/main.yml new file mode 100644 index 000000000..5be75a6a4 --- /dev/null +++ b/roles/singularity_wrapper/defaults/main.yml @@ -0,0 +1,10 @@ +--- +# vars for lecorguille.singularity +singularity_version: "3.7.3" +singularity_conf_path: "/etc/singularity/singularity.conf" +bind_paths: [] + +# vars for gantsign.golang +golang_version: "1.14.4" +golang_install_dir: "/opt/go/{{ golang_version }}" +golang_gopath: "/opt/go/packages" diff --git a/roles/singularity_wrapper/meta/main.yml b/roles/singularity_wrapper/meta/main.yml new file mode 100644 index 000000000..9fbd94944 --- /dev/null +++ b/roles/singularity_wrapper/meta/main.yml @@ -0,0 +1,9 @@ +--- +galaxy_info: + role_name: singularity_wrapper + namespace: deepops + author: DeepOps Team + company: NVIDIA + description: Wrap lecourguille.singularity role + license: 3-Clause BSD + min_ansible_version: 2.9 diff --git a/roles/singularity_wrapper/molecule/default/converge.yml b/roles/singularity_wrapper/molecule/default/converge.yml new file mode 100644 index 000000000..c0295f669 --- /dev/null +++ b/roles/singularity_wrapper/molecule/default/converge.yml @@ -0,0 +1,7 @@ +--- +- name: Converge + hosts: all + tasks: + - name: "Include singularity_wrapper" + include_role: + name: "singularity_wrapper" diff --git a/roles/singularity_wrapper/molecule/default/molecule.yml b/roles/singularity_wrapper/molecule/default/molecule.yml new file mode 100644 index 000000000..2962ff2e7 --- /dev/null +++ b/roles/singularity_wrapper/molecule/default/molecule.yml @@ -0,0 +1,26 @@ +--- +dependency: + name: galaxy + options: + requirements-file: requirements.yml +driver: + name: docker +platforms: + - name: ubuntu-1804 + image: geerlingguy/docker-ubuntu1804-ansible + pre_build_image: true + - name: ubuntu-2004 + image: geerlingguy/docker-ubuntu2004-ansible + pre_build_image: true + - name: centos-7 + image: geerlingguy/docker-centos7-ansible + pre_build_image: true + - name: centos-8 + image: geerlingguy/docker-centos8-ansible + pre_build_image: true +provisioner: + name: ansible + ansible_args: + - -vv +verifier: + name: ansible diff --git a/roles/singularity_wrapper/molecule/default/verify.yml b/roles/singularity_wrapper/molecule/default/verify.yml new file mode 100644 index 000000000..79044cd06 --- /dev/null +++ b/roles/singularity_wrapper/molecule/default/verify.yml @@ -0,0 +1,10 @@ +--- +# This is an example playbook to execute Ansible tests. + +- name: Verify + hosts: all + gather_facts: false + tasks: + - name: Example assertion + assert: + that: true diff --git a/roles/singularity_wrapper/tasks/main.yml b/roles/singularity_wrapper/tasks/main.yml new file mode 100644 index 000000000..a0675c73e --- /dev/null +++ b/roles/singularity_wrapper/tasks/main.yml @@ -0,0 +1,35 @@ +--- +- name: centos 8 - ensure powertools installed + block: + - name: ensure prereq packages installed + yum: + name: "dnf-plugins-core" + state: "present" + - name: enable powertools + command: "yum config-manager --set-enabled powertools" + register: enable_powertools + changed_when: enable_powertools.rc != 0 + when: (ansible_distribution == "CentOS") and (ansible_distribution_major_version == "8") + +- name: rhel 8 - ensure CRB repository is enabled + rhsm_repository: + name: "codeready-builder-for-rhel-8-x86_64-rpms" + when: (ansible_distribution == "Red Hat Enterprise Linux") and (ansible_distribution_major_version == "8") + +- name: debian - ensure apt cache is up to date + apt: + update_cache: yes + when: ansible_os_family == "Debian" + +- name: create a folder for go + file: + path: "{{ golang_install_dir }}" + recurse: yes + +- name: install golang explicitly + include_role: + name: gantsign.golang + +- name: install singularity + include_role: + name: abims_sbr.singularity From 7d80b583ed241e003c03287fee3911d374d88b42 Mon Sep 17 00:00:00 2001 From: Adam DeConinck Date: Fri, 7 Jan 2022 15:34:49 +0000 Subject: [PATCH 2/5] Fix setup.sh to deal with includes in requirements Annoyingly, includes in requirements.yml seem to be relative to the cwd of ansible-galaxy, rather than relative to the requirements file. https://github.com/ansible/ansible/issues/46385 Made changes to setup.sh in order to ensure we can install requirements even if we use includes. --- scripts/setup.sh | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/scripts/setup.sh b/scripts/setup.sh index e711c4f74..b1f8fbcbf 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -7,10 +7,14 @@ # Can be run standalone with: curl -sL git.io/deepops | bash # or: curl -sL git.io/deepops | bash -s -- 19.07 +# Determine current directory and root directory +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +ROOT_DIR="${SCRIPT_DIR}/.." + # Configuration ANSIBLE_VERSION="${ANSIBLE_VERSION:-2.9.21}" # Ansible version to install ANSIBLE_TOO_NEW="${ANSIBLE_TOO_NEW:-2.10.0}" # Ansible version too new -CONFIG_DIR="${CONFIG_DIR:-./config}" # Default configuration directory location +CONFIG_DIR="${CONFIG_DIR:-${ROOT_DIR}/config}" # Default configuration directory location DEEPOPS_TAG="${1:-master}" # DeepOps branch to set up JINJA2_VERSION="${JINJA2_VERSION:-2.11.1}" # Jinja2 required version PIP="${PIP:-pip3}" # Pip binary to use @@ -21,10 +25,6 @@ VENV_DIR="${VENV_DIR:-/opt/deepops/env}" # Path to python virtual environ # Set distro-specific variables . /etc/os-release - -SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" -ROOT_DIR="${SCRIPT_DIR}/.." - DEPS_DEB=(git virtualenv python3-virtualenv sshpass wget) DEPS_EL7=(git libselinux-python3 python-virtualenv python3-virtualenv sshpass wget) DEPS_EL8=(git python3-libselinux python3-virtualenv sshpass wget) @@ -146,10 +146,25 @@ fi # Install Ansible Galaxy roles if command -v ansible-galaxy &> /dev/null ; then echo "Updating Ansible Galaxy roles..." - as_user ansible-galaxy collection install --force -r "${ROOT_DIR}/roles/requirements.yml" >/dev/null - as_user ansible-galaxy role install --force -r "${ROOT_DIR}/roles/requirements.yml" >/dev/null - as_user ansible-galaxy collection install --force -i -r "${ROOT_DIR}/config/requirements.yml" >/dev/null - as_user ansible-galaxy role install --force -i -r "${ROOT_DIR}/config/requirements.yml" >/dev/null + initial_dir="$(pwd)" + roles_path="${ROOT_DIR}/roles/galaxy" + collections_path="${ROOT_DIR}/collections" + + # First, install requirements from role requirements. + # Note: due to a known issue in ansible-galaxy, this works best when the + # cwd is the same as the directory where the file is located. + # https://github.com/ansible/ansible/issues/46385 + cd "${ROOT_DIR}/roles" + as_user ansible-galaxy collection install -p "${collections_path}" --force -r "requirements.yml" >/dev/null + as_user ansible-galaxy role install -p "${roles_path}" --force -r "requirements.yml" >/dev/null + + # Install any user-defined config requirements + if [ -d "${CONFIG_DIR}" ]; then + cd "${CONFIG_DIR}" + as_user ansible-galaxy collection install -p "${collections_path}" --force -i -r "requirements.yml" >/dev/null + as_user ansible-galaxy role install -p "${roles_path}" --force -i -r "requirements.yml" >/dev/null + fi + cd "${initial_dir}" else echo "ERROR: Unable to install Ansible Galaxy roles, 'ansible-galaxy' command not found" fi From 123ae2922468a4c1c9e353ecbeaa604b7cd50344 Mon Sep 17 00:00:00 2001 From: Adam DeConinck Date: Fri, 7 Jan 2022 18:04:30 +0000 Subject: [PATCH 3/5] Add basic verification test Just check that we actually install Singularity --- .../molecule/default/verify.yml | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/roles/singularity_wrapper/molecule/default/verify.yml b/roles/singularity_wrapper/molecule/default/verify.yml index 79044cd06..b5afb1c0d 100644 --- a/roles/singularity_wrapper/molecule/default/verify.yml +++ b/roles/singularity_wrapper/molecule/default/verify.yml @@ -1,10 +1,13 @@ --- -# This is an example playbook to execute Ansible tests. - -- name: Verify +- name: verify hosts: all - gather_facts: false tasks: - - name: Example assertion + - name: check for path to singularity + command: which singularity + register: which_singularity + changed_when: which_singularity.rc != 0 + + - name: verify path to singularity assert: - that: true + that: + - "'/usr/local/bin/singularity' in which_singularity.stdout" From 86725b0ef0ce495e0218e665b23e37ef57fa12a1 Mon Sep 17 00:00:00 2001 From: Adam DeConinck Date: Tue, 18 Jan 2022 20:17:29 +0000 Subject: [PATCH 4/5] we don't need to support includes anymore, so move back to repo root --- scripts/setup.sh | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/scripts/setup.sh b/scripts/setup.sh index b1f8fbcbf..ca7cd6b25 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -150,13 +150,9 @@ if command -v ansible-galaxy &> /dev/null ; then roles_path="${ROOT_DIR}/roles/galaxy" collections_path="${ROOT_DIR}/collections" - # First, install requirements from role requirements. - # Note: due to a known issue in ansible-galaxy, this works best when the - # cwd is the same as the directory where the file is located. - # https://github.com/ansible/ansible/issues/46385 - cd "${ROOT_DIR}/roles" - as_user ansible-galaxy collection install -p "${collections_path}" --force -r "requirements.yml" >/dev/null - as_user ansible-galaxy role install -p "${roles_path}" --force -r "requirements.yml" >/dev/null + cd "${ROOT_DIR}" + as_user ansible-galaxy collection install -p "${collections_path}" --force -r "roles/requirements.yml" >/dev/null + as_user ansible-galaxy role install -p "${roles_path}" --force -r "roles/requirements.yml" >/dev/null # Install any user-defined config requirements if [ -d "${CONFIG_DIR}" ]; then From e4991dda7c0885359608a2aff431c37ff84133e7 Mon Sep 17 00:00:00 2001 From: Adam DeConinck Date: Tue, 18 Jan 2022 20:17:49 +0000 Subject: [PATCH 5/5] only run against config requirements.yml if file exists --- scripts/setup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/setup.sh b/scripts/setup.sh index ca7cd6b25..d65e1792e 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -155,7 +155,7 @@ if command -v ansible-galaxy &> /dev/null ; then as_user ansible-galaxy role install -p "${roles_path}" --force -r "roles/requirements.yml" >/dev/null # Install any user-defined config requirements - if [ -d "${CONFIG_DIR}" ]; then + if [ -d "${CONFIG_DIR}" ] && [ -f "${CONFIG_DIR}/requirement.yml" ] ; then cd "${CONFIG_DIR}" as_user ansible-galaxy collection install -p "${collections_path}" --force -i -r "requirements.yml" >/dev/null as_user ansible-galaxy role install -p "${roles_path}" --force -i -r "requirements.yml" >/dev/null