Add ability to specify a minor version for pulpcore_version

`pulpcore_version` now defaults to a specific minor release of pulpcore (e.g.: 3.15)
which the installer is going to install.

When the installer is run for the first time, the latest patch release of pulpcore
(e.g.: 3.15.2) is installed. If the installer is run again with the same value of
`pulpcore_version`, the pulpcore package is not updated. In order to update to the
latest patch release within a minor stream, the user must set `pulpcore_update` to
true.

If the user sets the `pulpcore_version` to a different minor release of pulpcore
(e.g.: 3.16), the installer will attempt to upgrade pulpcore to the latest patch
release (e.g.: 3.16.1) for that minor release of pulpcore.

If the user sets the `pulpcore_version` to a specific patch release, the installer
will attempt to install that specific release of pulpcore.

If the installed plugins are not compatible with the requested release of pulpcore,
the installer will not make any changes.

The `version` of a plugin can now also be specified as a minor release (e.g.: 1.8) or
a specific patch release (e.g.: 1.8.2). When the a minor release is specified and it
is the same as the installed version of the plugin, the `upgrade` parameter for that
plugin must be set to `true` in order for the plugin to be updated to the latest
patch release.

The internal variable `__pulp_version` was renamed to `__pulpcore_version`.

The release-static test scenario for molecule was also updated to test this feature.

fixes: #8847
https://pulp.plan.io/issues/8847

.ci/scripts/release.py

@@ -107,8 +107,8 @@ def validate_redmine_data(redmine_query_url, redmine_issues):
 
 # Second commit: release
 with open("roles/pulp_common/vars/main.yml") as f:
-    __pulp_version = yaml.safe_load(f)["__pulp_version"]
-    previous_pulpcore_version = __pulp_version.split('"')[1]
+    __pulpcore_version = yaml.safe_load(f)["__pulpcore_version"]
+    previous_pulpcore_version = __pulpcore_version.split('"')[1]
 
 with open("roles/pulp_common/defaults/main.yml") as f:
     previous_selinux_version = yaml.safe_load(f)["__pulp_selinux_version"]
@@ -119,7 +119,7 @@ def validate_redmine_data(redmine_query_url, redmine_issues):
 sed(
     [
         "-i",
-        f"/^__pulp_version:/s/{previous_pulpcore_version}/{pulpcore_version}/",
+        f"/^__pulpcore_version:/s/{previous_pulpcore_version}/{pulpcore_version}/",
         "roles/pulp_common/vars/main.yml",
     ]
 )

CHANGES/8847.feature

@@ -0,0 +1,2 @@
+Added support for specifying a minor version for `pulpcore_version`. A single version of pulp_installer can be used to
+install all bug fix releases within a minor release of pulpcore.

CHANGES/8847.removal

@@ -0,0 +1 @@
+`pulp_version` variable has been replaced with `pulpcore_version`.

docs/contributing.md

@@ -120,6 +120,9 @@ There are other (intentional) differences between tests:
    only did, `dynamic` because they will become containers soon anyway to test cluster installs.)
 1. `dynamic` - Due to a limitation of Ansible 2.8 with collections, these are not tested with
    Ansible 2.8.
+1. The `release-upgrade` scenario uses its own `converge.yml` playbook instead of the default one
+   used by all other scenarios. This playbook upgrades a Pulp installation multiple times to ensure
+   that all upgrade scenarios work correctly. 
 
 To test both webserver solutions we testing `apache` as webserver with
 

docs/customizing.md

@@ -66,6 +66,7 @@ In the following example, a multitude of variables are set:
 
 ```
   vars:
+    pulpcore_update: true
     pulp_workers: 4
     pulp_install_plugins:
       pulp-container: {}
@@ -86,6 +87,8 @@ In the following example, a multitude of variables are set:
       default_file_storage: 'storages.backends.azure_storage.AzureStorage'
 ```
 
+`pulpcore_update` is set to `true` which means that if pulpcore 3.16 is already installed, it will be upgraded to the latest patch release 3.16.z.
+
 `pulp_workers`, is set to `4`, which means that 4 worker processes will be started for processing jobs on a multi-core system. It replaces the default value of 2.
 
 `pulp_install_plugins` does the following:
@@ -138,7 +141,7 @@ Recommended Workflows for Pulpcore & Plugin Versioning
 Initial installation:
 
 1. Make sure you are running the latest version of the installer, which installs pulpcore's
-   latest version.
+   latest 3.y release.
 1. Confirm that all the latest stable releases of your desired plugins are compatible with
    pulpcore's latest version , such as by reading the release announcement email thread for
    pulpcore's latest version, reading the plugins README, or as a last resort, reading their `setup.py`.
@@ -164,13 +167,16 @@ Upgrading your installation:
 1. If they are not all compatible yet, **wait** for the plugins to be updated for
    compatibility.
 1. Upgrade `pulp_installer` to the latest version.
+1. Set `pulpcore_update` to `true`. This will ensure that even if you are on the same minor release, the latest 
+   patch release will be installed.
 1. Set `pulp_install_plugins` with each plugin listed as a key, and with each plugin having a key under it called `upgrade` set to the value `true`.
 1. Re-run `pulp_installer`.
 1. Make sure to save your variables/playbook for later usage.
 
 Example `pulp_install_plugins`:
 ```
   vars:
+    pulpcore_update: true
     pulp_install_plugins:
       pulp-container:
          upgrade: true
@@ -180,6 +186,58 @@ Example `pulp_install_plugins`:
          upgrade: true
 ```
 
+### Latest patch releases for minor releases of pulpcore and plugins
+
+Initial installation:
+
+1. Make sure you are running the latest version of the installer, which installs pulpcore's
+   latest 3.y release.
+1. Confirm that all the latest stable releases of your desired plugins are compatible with
+   pulpcore's latest version , such as by reading the release announcement email thread for
+   pulpcore's latest version, reading the plugins README, or as a last resort, reading their `setup.py`.
+1. Set `pulp_install_plugins` with each plugin listed as a key. For each each plugin add a key under it
+   called `version` set to the disired minor version of a plugin.
+1. Run `pulp_installer`.
+1. Make sure to save your variables/playbook for later usage.
+
+Example `pulp_install_plugins`:
+```
+  vars:
+    pulp_install_plugins:
+      pulp-container:
+        version: "2.8"
+      pulp-file:
+        version: "1.9"
+      pulp-rpm:
+        version: "3.15"
+```
+
+Upgrading your installation:
+
+1. Use the same version of `pulp_installer` that was used for the initial installation.
+1. Set `pulpcore_update` to `true`. This will upgrade your `pulpcore` package to the latest patch release.
+1. Set `pulp_install_plugins` with each plugin listed as a key. For each each plugin add a key under it
+   called `version` set to the currently install `x.y` plugin version. Add a key called `upgrade` set to
+   the value `true`.
+1. Re-run `pulp_installer`.
+1. Make sure to save your variables/playbook for later usage.
+
+Example `pulp_install_plugins`:
+```
+  vars:
+    pulpcore_update: true
+    pulp_install_plugins:
+      pulp-container:
+        version: "2.8"
+        upgrade: true
+      pulp-file:
+        version: "1.9"
+        upgrade: true
+      pulp-rpm:
+        version: "3.15"
+        upgrade: true
+```
+
 ### Specifying Exact Versions with Reproducibility:
 
 Initial installation:

docs/index.md

@@ -5,9 +5,11 @@ The Pulp 3 Ansible installer is a collection of Ansible roles that you can use t
 
 Each Ansible role installs and configures a component of Pulp.
 
-This version of the installer, 3.15.2-2, installs Pulp 3.15.2 specifically.
+This version of the installer, 3.15, installs the latest release of pulpcore 3.15.z. 
 
-If run against an older version of Pulp 3, it will upgrade it to 3.15.2.
+If run against an older version of Pulp 3, it will upgrade it to 3.15.z.
+
+If run again, the installed version of pulpcore 3.15 is not updated unless `pulpcore_update` is set to `true`.
 
 ---
 **Didn't find what you need to get started?**

molecule/release-upgrade/converge.yml

@@ -0,0 +1,144 @@
+---
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.14.5"
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8.0"
+          pulp_rpm:
+            version: "3.13.0"
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8.0"
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_all_services
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.14.5, pulp-file=1.8.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.14.5", "=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.8.0", "=")
+    - name: Assert pulp-rpm==3.13.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.13.0", "=")
+      when: ansible_facts.os_family == 'RedHat'
+
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.14"
+        pulpcore_update: false
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: false
+          pulp_rpm:
+            version: "3.13"
+            upgrade: false
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: fasle
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_common
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.14.5, pulp-file=1.8.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.14.5", "=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.8.0", "=")
+    - name: Assert pulp-rpm==3.13.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.13.0", "=")
+      when: ansible_facts.os_family == 'RedHat'
+
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.14"
+        pulpcore_update: true
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: true
+          pulp_rpm:
+            version: "3.13"
+            upgrade: true
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.8"
+            upgrade: true
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_common
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.14.6, pulp-file==1.8.2
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.14.6", ">=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.8.2", ">=")
+    - name: Assert pulp-rpm==3.13.3
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.13.3", ">=")
+      when: ansible_facts.os_family == 'RedHat'
+
+- hosts: all
+  tasks:
+    - set_fact:
+        pulpcore_version: "3.15"
+        pulpcore_update: false
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.9"
+            upgrade: false
+          pulp_rpm:
+            version: "3.15"
+            upgrade: false
+      when: ansible_facts.os_family == 'RedHat'
+    - set_fact:
+        pulp_install_plugins:
+          pulp_file:
+            version: "1.9"
+            upgrade: fasle
+      when: ansible_facts.os_family == 'Debian'
+    - include_role:
+        name: pulp_common
+    - name: Obtain list of packages & versions in the venv after install
+      pip_package_info:
+        clients: "/usr/local/lib/pulp/bin/pip"
+      register: pip_pkgs
+    - name: Assert pulpcore==3.15.2, pulp-file=1.9.1
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulpcore'][0].version is version("3.15.2", ">=")
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-file'][0].version is version("1.9.1", ">=")
+    - name: Assert pulp-rpm==3.15.0
+      assert:
+        that:
+          - pip_pkgs.packages['/usr/local/lib/pulp/bin/pip']['pulp-rpm'][0].version is version("3.15.0", ">=")
+      when: ansible_facts.os_family == 'RedHat'

molecule/release-upgrade/group_vars/all

@@ -1,12 +1,6 @@
 ---
 __pulp_database_config_run_once: false
 pulp_default_admin_password: password
-pulp_upgrade: true
-pulp_install_plugins:
-  pulp-file:
-    upgrade: true
-  pulp-rpm:
-    upgrade: true
 pulp_settings:
   secret_key: secret
   content_origin: "https://{{ ansible_fqdn }}"

molecule/release-upgrade/host_vars/debian-11

@@ -1,6 +1 @@
 ansible_python_interpreter: /usr/bin/python3
-pulp_install_plugins:
-  pulp-file:
-    upgrade: true
-  # pulp-rpm:
-  #   upgrade: true

molecule/release-upgrade/molecule.yml

@@ -58,8 +58,6 @@ scenario:
     - create
     - prepare
     - converge
-    - idempotence
-    - check
     - side_effect
     - verify
 verifier:

roles/pulp_common/README.md

@@ -121,6 +121,9 @@ Role Variables
 * `pulp_certs_dir`: Path where to generate or drop the TLS certificates & keys for authentication
   tokens. Not used directly by pulp_common, but by roles that depend on it. Defaults to
   '{{ pulp_config_dir }}/certs' .
+* `pulpcore_update`: Boolean that specifies whether the pulpcore package should be updated to the
+  latest bug fix release within the minor release specified by `pulpcore_version`. Defaults
+  to `false`.
 
 Role Variables if installing from RPMs
 --------------------------------------
@@ -172,9 +175,9 @@ If it is set to "packages", the following variables are used, or behave *differe
 Role Variables for advanced usage
 ---------------------------------
 
-* `pulpcore_version`: Specify a specific version of pulpcore one would like to install or upgrade to.
-   By default the installer will do the right thing by using the version of pulpcore it is designed
-   for and tested with. It is strongly advised against setting.
+* `pulpcore_version`: Specify a minor version of pulpcore (e.g.: `3.15`) one would like to install or upgrade to.
+   By default the installer will do the right thing by using the minor version of pulpcore it is designed
+   for and tested with. This can also be a specific patch release (e.g.: `3.15.2`).
 * `pulp_service_timeout`: Set timeout value for pulp services. Defaults to 90.
 
 Shared Variables

roles/pulp_common/defaults/main.yml

@@ -21,6 +21,7 @@ pulp_user_home: '/var/lib/pulp'
 pulp_media_root: '/var/lib/pulp/media'
 pulp_pip_editable: yes
 pulp_use_system_wide_pkgs: false
+pulpcore_update: false
 prereq_pip_packages:
   - Jinja2
   - pygments

roles/pulp_common/tasks/install_packages.yml

@@ -140,8 +140,8 @@
   package_facts:
     manager: "auto"
 
-- name: Set __pulp_version to the RPM that was installed
+- name: Set __pulpcore_version to the RPM that was installed
   set_fact:
-    __pulp_version: "{{ ansible_facts.packages[pulp_pkg_pulpcore_name].0.version }}"
+    __pulpcore_version: "{{ ansible_facts.packages[pulp_pkg_pulpcore_name].0.version }}"
   when: pulpcore_version | default('') | length == 0
 ...