Add ability to specify a minor version for pulpcore_version

.ci/scripts/release.py

@@ -107,8 +107,8 @@ def validate_redmine_data(redmine_query_url, redmine_issues):
 
 # Second commit: release
 with open("roles/pulp_common/vars/main.yml") as f:
-    __pulp_version = yaml.safe_load(f)["__pulp_version"]
-    previous_pulpcore_version = __pulp_version.split('"')[1]
+    __pulpcore_version = yaml.safe_load(f)["__pulpcore_version"]
+    previous_pulpcore_version = __pulpcore_version.split('"')[1]
 
 with open("roles/pulp_common/defaults/main.yml") as f:
     previous_selinux_version = yaml.safe_load(f)["__pulp_selinux_version"]
@@ -119,7 +119,7 @@ def validate_redmine_data(redmine_query_url, redmine_issues):
 sed(
     [
         "-i",
-        f"/^__pulp_version:/s/{previous_pulpcore_version}/{pulpcore_version}/",
+        f"/^__pulpcore_version:/s/{previous_pulpcore_version}/{pulpcore_version}/",
         "roles/pulp_common/vars/main.yml",
     ]
 )

CHANGES/8847.feature

@@ -0,0 +1,2 @@
+Added support for specifying a minor version for `pulpcore_version`. A single version of pulp_installer can be used to
+install all bug fix releases within a minor release of pulpcore.

CHANGES/8847.removal

@@ -0,0 +1 @@
+`pulp_version` variable has been replaced with `pulpcore_version`.

docs/contributing.md

@@ -120,6 +120,9 @@ There are other (intentional) differences between tests:
    only did, `dynamic` because they will become containers soon anyway to test cluster installs.)
 1. `dynamic` - Due to a limitation of Ansible 2.8 with collections, these are not tested with
    Ansible 2.8.
+1. The `release-upgrade` scenario uses its own `converge.yml` playbook instead of the default one
+   used by all other scenarios. This playbook upgrades a Pulp installation multiple times to ensure
+   that all upgrade scenarios work correctly. 
 
 To test both webserver solutions we testing `apache` as webserver with
 

docs/customizing.md

@@ -66,6 +66,7 @@ In the following example, a multitude of variables are set:
 
 ```
   vars:
+    pulpcore_update: true
     pulp_workers: 4
     pulp_install_plugins:
       pulp-container: {}
@@ -86,6 +87,8 @@ In the following example, a multitude of variables are set:
       default_file_storage: 'storages.backends.azure_storage.AzureStorage'
 ```
 
+`pulpcore_update` is set to `true` which means that if pulpcore 3.16 is already installed, it will be upgraded to the latest patch release 3.16.z.
+
 `pulp_workers`, is set to `4`, which means that 4 worker processes will be started for processing jobs on a multi-core system. It replaces the default value of 2.
 
 `pulp_install_plugins` does the following:
@@ -138,7 +141,7 @@ Recommended Workflows for Pulpcore & Plugin Versioning
 Initial installation:
 
 1. Make sure you are running the latest version of the installer, which installs pulpcore's
-   latest version.
+   latest 3.y release.
 1. Confirm that all the latest stable releases of your desired plugins are compatible with
    pulpcore's latest version , such as by reading the release announcement email thread for
    pulpcore's latest version, reading the plugins README, or as a last resort, reading their `setup.py`.
@@ -164,13 +167,16 @@ Upgrading your installation:
 1. If they are not all compatible yet, **wait** for the plugins to be updated for
    compatibility.
 1. Upgrade `pulp_installer` to the latest version.
+1. Set `pulpcore_update` to `true`. This will ensure that even if you are on the same minor release, the latest 
+   patch release will be installed.
 1. Set `pulp_install_plugins` with each plugin listed as a key, and with each plugin having a key under it called `upgrade` set to the value `true`.
 1. Re-run `pulp_installer`.
 1. Make sure to save your variables/playbook for later usage.
 
 Example `pulp_install_plugins`:
 ```
   vars:
+    pulpcore_update: true
     pulp_install_plugins:
       pulp-container:
          upgrade: true
@@ -180,6 +186,58 @@ Example `pulp_install_plugins`:
          upgrade: true
 ```
 
+### Latest patch releases for minor releases of pulpcore and plugins
+
+Initial installation:
+
+1. Make sure you are running the latest version of the installer, which installs pulpcore's
+   latest 3.y release.
+1. Confirm that all the latest stable releases of your desired plugins are compatible with
+   pulpcore's latest version , such as by reading the release announcement email thread for
+   pulpcore's latest version, reading the plugins README, or as a last resort, reading their `setup.py`.
+1. Set `pulp_install_plugins` with each plugin listed as a key. For each each plugin add a key under it
+   called `version` set to the disired minor version of a plugin.
+1. Run `pulp_installer`.
+1. Make sure to save your variables/playbook for later usage.
+
+Example `pulp_install_plugins`:
+```
+  vars:
+    pulp_install_plugins:
+      pulp-container:
+        version: "2.8"
+      pulp-file:
+        version: "1.9"
+      pulp-rpm:
+        version: "3.15"
+```
+
+Upgrading your installation:
+
+1. Use the same version of `pulp_installer` that was used for the initial installation.
+1. Set `pulpcore_update` to `true`. This will upgrade your `pulpcore` package to the latest patch release.
+1. Set `pulp_install_plugins` with each plugin listed as a key. For each each plugin add a key under it
+   called `version` set to the currently install `x.y` plugin version. Add a key called `upgrade` set to
+   the value `true`.
+1. Re-run `pulp_installer`.
+1. Make sure to save your variables/playbook for later usage.
+
+Example `pulp_install_plugins`:
+```
+  vars:
+    pulpcore_update: true
+    pulp_install_plugins:
+      pulp-container:
+        version: "2.8"
+        upgrade: true
+      pulp-file:
+        version: "1.9"
+        upgrade: true
+      pulp-rpm:
+        version: "3.15"
+        upgrade: true
+```
+
 ### Specifying Exact Versions with Reproducibility:
 
 Initial installation:

docs/index.md

@@ -5,9 +5,11 @@ The Pulp 3 Ansible installer is a collection of Ansible roles that you can use t
 
 Each Ansible role installs and configures a component of Pulp.
 
-This version of the installer, 3.15.2-2, installs Pulp 3.15.2 specifically.
+This version of the installer, 3.15, installs the latest release of pulpcore 3.15.z. 
 
-If run against an older version of Pulp 3, it will upgrade it to 3.15.2.
+If run against an older version of Pulp 3, it will upgrade it to 3.15.z.
+
+If run again, the installed version of pulpcore 3.15 is not updated unless `pulpcore_update` is set to `true`.
 
 ---
 **Didn't find what you need to get started?**

molecule/release-upgrade/converge.yml

@@ -0,0 +1,144 @@
+---
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.14.5"
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8.0"
+          pulp_rpm:
+            version: "3.13.0"
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8.0"
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_all_services
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.14.5, pulp-file=1.8.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.14.5", "=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.8.0", "=")
+    - name: Assert pulp-rpm==3.13.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.13.0", "=")
+      when: ansible_facts.os_family == 'RedHat'
+
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.14"
+        pulpcore_update: false
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: false
+          pulp_rpm:
+            version: "3.13"
+            upgrade: false
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: fasle
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_common
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.14.5, pulp-file=1.8.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.14.5", "=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.8.0", "=")
+    - name: Assert pulp-rpm==3.13.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.13.0", "=")
+      when: ansible_facts.os_family == 'RedHat'
+
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.14"
+        pulpcore_update: true
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: true
+          pulp_rpm:
+            version: "3.13"
+            upgrade: true
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: true
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_common
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.14.6, pulp-file==1.8.2
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.14.6", ">=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.8.2", ">=")
+    - name: Assert pulp-rpm==3.13.3
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.13.3", ">=")
+      when: ansible_facts.os_family == 'RedHat'
+
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.15"
+        pulpcore_update: false
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.9"
+            upgrade: false
+          pulp_rpm:
+            version: "3.15"
+            upgrade: false
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.9"
+            upgrade: fasle
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_common
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.15.2, pulp-file=1.9.1
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.15.2", ">=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.9.1", ">=")
+    - name: Assert pulp-rpm==3.15.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.15.0", ">=")
+      when: ansible_facts.os_family == 'RedHat'

molecule/release-upgrade/group_vars/all

@@ -1,12 +1,6 @@
 ---
 __pulp_database_config_run_once: false
 pulp_default_admin_password: password
-pulp_upgrade: true
-pulp_install_plugins:
-  pulp-file:
-    upgrade: true
-  pulp-rpm:
-    upgrade: true
 pulp_settings:
   secret_key: secret
   content_origin: "https://{{ ansible_fqdn }}"

molecule/release-upgrade/host_vars/debian-11

@@ -1,6 +1 @@
 ansible_python_interpreter: /usr/bin/python3
-pulp_install_plugins:
-  pulp-file:
-    upgrade: true
-  # pulp-rpm:
-  #   upgrade: true

molecule/release-upgrade/molecule.yml

@@ -58,8 +58,6 @@ scenario:
     - create
     - prepare
     - converge
-    - idempotence
-    - check
     - side_effect
     - verify
 verifier:

roles/pulp_common/README.md

@@ -121,6 +121,9 @@ Role Variables
 * `pulp_certs_dir`: Path where to generate or drop the TLS certificates & keys for authentication
   tokens. Not used directly by pulp_common, but by roles that depend on it. Defaults to
   '{{ pulp_config_dir }}/certs' .
+* `pulpcore_update`: Boolean that specifies whether the pulpcore package should be updated to the
+  latest bug fix release within the minor release specified by `pulpcore_version`. Defaults
+  to `false`.
 
 Role Variables if installing from RPMs
 --------------------------------------
@@ -172,9 +175,9 @@ If it is set to "packages", the following variables are used, or behave *differe
 Role Variables for advanced usage
 ---------------------------------
 
-* `pulpcore_version`: Specify a specific version of pulpcore one would like to install or upgrade to.
-   By default the installer will do the right thing by using the version of pulpcore it is designed
-   for and tested with. It is strongly advised against setting.
+* `pulpcore_version`: Specify a minor version of pulpcore (e.g.: `3.15`) one would like to install or upgrade to.
+   By default the installer will do the right thing by using the minor version of pulpcore it is designed
+   for and tested with. This can also be a specific patch release (e.g.: `3.15.2`).
 * `pulp_service_timeout`: Set timeout value for pulp services. Defaults to 90.
 
 Shared Variables

roles/pulp_common/defaults/main.yml

@@ -21,6 +21,7 @@ pulp_user_home: '/var/lib/pulp'
 pulp_media_root: '/var/lib/pulp/media'
 pulp_pip_editable: yes
 pulp_use_system_wide_pkgs: false
+pulpcore_update: false
 prereq_pip_packages:
   - Jinja2
   - pygments

roles/pulp_common/tasks/install_packages.yml

@@ -140,8 +140,8 @@
   package_facts:
     manager: "auto"
 
-- name: Set __pulp_version to the RPM that was installed
+- name: Set __pulpcore_version to the RPM that was installed
   set_fact:
-    __pulp_version: "{{ ansible_facts.packages[pulp_pkg_pulpcore_name].0.version }}"
+    __pulpcore_version: "{{ ansible_facts.packages[pulp_pkg_pulpcore_name].0.version }}"
   when: pulpcore_version | default('') | length == 0
 ...