[Design RFC] The ansible-pulp installer needs to handle multiple versions of Pulp #203

mikedep333 · 2020-01-10T04:27:39Z

I am requesting feedback from the Pulp user community on this design (variables, docs, and behavior stated/implied by them) of implementing:
[Epic] The ansible-pulp installer needs to handle multiple versions of Pulp
(Extended details are in the subtasks.)

The following are the design decisions:

The installer is focused around the pulpcore version, not the plugin versions, and locks it when the plugins get installed via pip "constraints." (Think of how you install your desired version of an app like LibreOffice, and then obtain plugins (LibreOffice extensions) for that version. This is the opposite of pulp_juicy depending on pulpcore, and therefore upgrading pulpcore as needed per the pip model.)
One plugin's need for a newer version of pulpcore should not break a currently working system with a set of plugins installed (because the rest require an older version of pulpcore.) This applies to both adding a new plugin, and upgrading an existing plugin. It is better for ansible-pulp to error out than to cause breakage.
ansible-pulp should be idempotent by default and not modify/upgrade currently working systems, unless there was configuration drift somehow (e.g., file permissions modified by 3rd party utility.)
Most users care about z-stream branches: Like pulpcore 3.0.z and pulp_file 0.1.z, not individual versions/releases. When a plugin adds support for a new pulpcore z-stream branch, it will have its own new z-stream branch. When a plugin adds major new features that users may be reluctant to upgrade to, they will have their own new y-stream or z-stream branch.
As always, ansible-pulp is neutral towards plugins and does not have any bundled specific logic for any of them. We love pulp_cardboardbox just as much as pulp_rpm.

The following limitations are being worked around:

Ansible galaxy does not let you download desired branches for a role, or a collection of roles (ansible-pulp will become a collection). It only supports a specific version, or the latest version. So the latest version of ansible-pulp should support multiple recent branches of Pulp. This is similar to FreeIPA's ansible collection..
pip does not have a complete dependency resolver like RPM/DEB/etc do.
We cannot tell pip "install the latest version of pulp_juicy that is compatible with pulpcore 3.0's branch."
We cannot tell pip "install whatever version of pulpcore satisfies all the plugins' required versions" because pip does "first found wins".
Upgrading pulpcore can always break a plugin; it's up to the user to research instead when to upgrade pulpcore.

The following alternatives are being strongly considered:

wrt limitation 1, just do versioned releases. (Obtaining the latest ansible-pulp 3.0.z once pulpcore 3.1.z comes out will be more complex.)
wrt limitations 2-5, use a layer like pip-compile to overcome pip's limitations. Perhaps as a preflight script.

Proposed design for #5890 https://pulp.plan.io/issues/5890 [noissue]

This enables the installer to apply logic specific to the Pulp branch, before it goes to install it.

mikedep333 · 2020-01-13T20:11:55Z

roles/pulp/README.md

@@ -26,9 +29,12 @@ Role Variables:
    See `prereq_pip_packages` also.
 * `pulp_install_api_service`: Whether to create systemd service files for
  pulpcore-api. Defaults to "true".
+* `pulp_update`: Whether to update/upgrade pulpcore to the latest stable release from PyPI within `pulp_branch`. Only affects systems where Pulp is already installed. To limit this to micro (z-stream) updates, make sure to set `pulp_branch`. If `pulpcore_version` is set, or `pulp_source_dir` is set, this has no effect and is effectively always `true`. Defaults to "false".


I'm leaning towards renaming both this and the plugin's field to "upgrade", because it controls whether or not "--upgrade" is passed to pip. (The ansible "pip" module's "state: latest" vs "state: present" maps to pip's "--upgrade" option.)

mdellweg · 2020-01-16T09:21:24Z

roles/pulp/README.md

+
+Updating your install:
+
+1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and pulp_juicy 1.0.3 -> 1.0.4)


Suggested change

1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and pulp_juicy 1.0.3 -> 1.0.4)

1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is still the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and pulp_juicy 1.0.3 -> 1.0.4)

mdellweg · 2020-01-16T09:25:20Z

roles/pulp/README.md

+
+Updating your install:
+
+1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and plugin pulp_juicy 1.0.3 -> 1.0.4)


Suggested change

1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and plugin pulp_juicy 1.0.3 -> 1.0.4)

1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is still the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and plugin pulp_juicy 1.0.3 -> 1.0.4)

mdellweg · 2020-01-16T09:25:50Z

roles/pulp/README.md

+
+1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and pulp_juicy 1.0.3 -> 1.0.4)
+2. Update ansible-pulp
+3. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.


Suggested change

3. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

3. Re-run the ansible-pulp role with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

mdellweg · 2020-01-16T09:26:28Z

roles/pulp/README.md

+2. Check if your plugins are compatible yet with pulpcore 3.1 yet. **Wait** for the plugins to be updated for compatibility if they are not updated yet before you attempt to update.
+3. If they are updated for compatibility:
+    1. Update ansible-pulp
+    2. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.


Suggested change

2. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

2. Re-run the ansible-pulp role with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

mdellweg · 2020-01-16T09:27:32Z

roles/pulp/README.md

+
+1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and plugin pulp_juicy 1.0.3 -> 1.0.4)
+2. Update ansible-pulp
+3. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.


Suggested change

3. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

3. Re-run the ansible-pulp role with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

mdellweg · 2020-01-16T09:27:54Z

roles/pulp/README.md

+2. Repeat step 2 from the initial install. Let's assume that , but pulp_juicy has gone to a new branch (pulp_juicy 1.0.3 -> 2.0.0) with major new features, while still being compatible with your pulpcore version (3.0).
+3.`Change pulp_install_plugins.pulp_juicy.branch` to `2.0`. This is its new current & perpetual value.
+4. set `pulp_install_plugins.pulp_juicy.update` to `true`; it will upgrade it. Set `pulp_update` to true as a precaution for the latest bugfixes  that the plugins may depend on.
+5. Update ansible-pulp


Suggested change

5. Update ansible-pulp

5. Update ansible-pulp.

mdellweg · 2020-01-16T09:28:02Z

roles/pulp/README.md

+3.`Change pulp_install_plugins.pulp_juicy.branch` to `2.0`. This is its new current & perpetual value.
+4. set `pulp_install_plugins.pulp_juicy.update` to `true`; it will upgrade it. Set `pulp_update` to true as a precaution for the latest bugfixes  that the plugins may depend on.
+5. Update ansible-pulp
+6. Run ansible-pulp


Suggested change

6. Run ansible-pulp

6. Run ansible-pulp.

mdellweg · 2020-01-16T09:32:15Z

roles/pulp/README.md

@@ -26,9 +29,12 @@ Role Variables:
    See `prereq_pip_packages` also.
 * `pulp_install_api_service`: Whether to create systemd service files for
  pulpcore-api. Defaults to "true".
+* `pulp_update`: Whether to update/upgrade pulpcore to the latest stable release from PyPI within `pulp_branch`. Only affects systems where Pulp is already installed. To limit this to micro (z-stream) updates, make sure to set `pulp_branch`. If `pulpcore_version` is set, or `pulp_source_dir` is set, this has no effect and is effectively always `true`. Defaults to "false".
+* `pulp_branch`: The branch of of pulpcore (or update/upgrade to the latest release from, see `pulp_update`). Defaults to the latest stable branch release of pulpcore at the time of ansible-pulp being released, which is currently `3.0`. It is recommended to set this if you ever plan to re-run the ansible installer; when a new branch is released and if you update ansible-pulp, some of your content plugins may not be compatible yet. An example value is `3.1`. Ignored if `pulp_source_dir` or `pulp_version` is set. Do not set this to a version higher than the current default (unless you are installing a development branch of pulpcore, and have updated ansible-pulp 1st.)


Suggested change

* `pulp_branch`: The branch of of pulpcore (or update/upgrade to the latest release from, see `pulp_update`). Defaults to the latest stable branch release of pulpcore at the time of ansible-pulp being released, which is currently `3.0`. It is recommended to set this if you ever plan to re-run the ansible installer; when a new branch is released and if you update ansible-pulp, some of your content plugins may not be compatible yet. An example value is `3.1`. Ignored if `pulp_source_dir` or `pulp_version` is set. Do not set this to a version higher than the current default (unless you are installing a development branch of pulpcore, and have updated ansible-pulp 1st.)

* `pulp_branch`: The branch of pulpcore (or update/upgrade to the latest release from, see `pulp_update`). Defaults to the latest stable branch release of pulpcore at the time of ansible-pulp being released, which is currently `3.0`. It is recommended to set this if you ever plan to re-run the ansible installer; when a new branch is released and if you update ansible-pulp, some of your content plugins may not be compatible yet. An example value is `3.1`. Ignored if `pulp_source_dir` or `pulp_version` is set. Do not set this to a version higher than the current default (unless you are installing a development branch of pulpcore, and have updated ansible-pulp 1st.)

mdellweg · 2020-01-16T09:35:16Z

roles/pulp/README.md

@@ -26,9 +29,12 @@ Role Variables:
    See `prereq_pip_packages` also.
 * `pulp_install_api_service`: Whether to create systemd service files for
  pulpcore-api. Defaults to "true".
+* `pulp_update`: Whether to update/upgrade pulpcore to the latest stable release from PyPI within `pulp_branch`. Only affects systems where Pulp is already installed. To limit this to micro (z-stream) updates, make sure to set `pulp_branch`. If `pulpcore_version` is set, or `pulp_source_dir` is set, this has no effect and is effectively always `true`. Defaults to "false".
+* `pulp_branch`: The branch of of pulpcore (or update/upgrade to the latest release from, see `pulp_update`). Defaults to the latest stable branch release of pulpcore at the time of ansible-pulp being released, which is currently `3.0`. It is recommended to set this if you ever plan to re-run the ansible installer; when a new branch is released and if you update ansible-pulp, some of your content plugins may not be compatible yet. An example value is `3.1`. Ignored if `pulp_source_dir` or `pulp_version` is set. Do not set this to a version higher than the current default (unless you are installing a development branch of pulpcore, and have updated ansible-pulp 1st.)
+* `pulp_version`: Optional. A specific version of pulp to install from PyPI. Ignored if `pulp_source_dir` is set. Overrides `pulp_update` to `true`.


Not sure if overriding update here is desired. If you specified '3.0' for example update can still guard, whether to install z-stream updates, or keep the current version.
Or do you think, one should specify ~=3.0.0 or >=3.0.0,<3.1 to have the latest bugfix release?

Thank you for the review!

I didn't plan to write any code to override it. Rather, I am mostly certain that the override is a limitation of ansible's pip module, which is a wrapper around the pip command. And I'd rather not write & maintain a ton of code to workaround it ("embrace the constraint.")

I assume that calling the ansible pip module with version (which is only documented as accepting values like 3.0.1, not >=3.0.0) maps to it calling pip install pulpcore==3.0.1. With version undefined, I assume it maps it to calling pip install pulpcore.

I know (because I looked at the code that calling the ansible pip module with state=latest maps to pip install --upgrade, whereas state=present maps to pip install.

However the pip command, --upgrade is ignored whenever the exact version is specified. It will upgrade or downgrade regardless. I just tested this to be certain:

(upgrade-test) [mdepaulo@mdepaulo upgrade-test]$ pip install --upgrade pytest==5.3.1 ... Successfully installed pytest-5.3.1 (upgrade-test) [mdepaulo@mdepaulo upgrade-test]$ pip install --upgrade pytest==5.3.2 ... Successfully installed pytest-5.3.2 (upgrade-test) [mdepaulo@mdepaulo upgrade-test]$ pip install --upgrade pytest==5.3.1 ... Successfully installed pytest-5.3.1 (upgrade-test) [mdepaulo@mdepaulo upgrade-test]$ pip install pytest==5.3.2 ... Successfully installed pytest-5.3.2 (upgrade-test) [mdepaulo@mdepaulo upgrade-test]$ pip install pytest==5.3.1 ... Successfully installed pytest-5.3.1

Maybe it is not documented, but i tried to pass a non exact version to it, and it didn't report an error. I guess we still need to verify this also works as expected, but '<3.1' with or without '--upgrade' should be two different things.

The examples suggest that it is possible to have complex version specifiers:
https://docs.ansible.com/ansible/latest/modules/pip_module.html#examples

mdellweg · 2020-01-16T11:26:57Z

FYI: I wanted to test an installation with 3.0.z now, so i started to add the versions in ansible-pulp.
#207

bmbouter · 2020-01-23T22:16:19Z

I was thinking about how having the latest installer keep up with the previous versions, and I think we should consider releasing the installer each time we release pulpcore. It seems unavoidable with this thinking. Assume there will be changingrequirements in pulpcore from a packaging perspective at some point, e.g. process names, extra files/services, etc. For an old installer to adequately handle the installer itself will need an update. Thus the safest way to use the installer is to first update it.

If we cut releases as pulpcore releases them, then we can avoid a lot of complexity by having the user on a specific version of pulpcore. The upgrade process is: first get the new installer. I think that makes for a simple upgrade experience.

What do you think about ^?

mikedep333 · 2020-01-28T20:52:16Z

I was thinking about how having the latest installer keep up with the previous versions, and I think we should consider releasing the installer each time we release pulpcore. It seems unavoidable with this thinking. Assume there will be changingrequirements in pulpcore from a packaging perspective at some point, e.g. process names, extra files/services, etc. For an old installer to adequately handle the installer itself will need an update. Thus the safest way to use the installer is to first update it.

I'm assuming naming conventions stay the same within a Y-stream branch of pulpcore, but improved compatibility (e.g., support for weirdly configured systems) would be on newer releases of ansible-pulp.

And we'd have logic per pulpcore Y-stream branch in the installer. But it would be safe for users to obtain the latest version of ansible-pulp.

If we cut releases as pulpcore releases them, then we can avoid a lot of complexity by having the user on a specific version of pulpcore. The upgrade process is: first get the new installer. I think that makes for a simple upgrade experience.

What do you think about ^?

I am now leaning towards your proposed approach.

One big advantage is that we could always switch models later, and it is less work to implement now. We would not be digging a hole for ourselves if we want to go to something like the original model.

The implementation / impacts would include:

We'd still lock the puplcore version via constraints (we already do this) so that plugins never upgrade it by accident.
Users would still be responsible for ensuring their plugins are compatible.
We'd still need to enable users doing the above by letting them specify branch constraints or specific versions for plugins. And I'm looking into "preflight" or integrated tooling like pip-compile to facilitate this (as well as the original plan.)
In the short-term at least, we still need the variable "pulp_install_plugins.upgrade", and probably the ability to specify the exact version of the plugin to install. Users could run the installer on one system (test environment) and then try to re-run it on another (production environment) a month later, but if the latest release from pip a month later is now incompatible, it will break production but not test.
We'd tell users in release notes to run ansible-galaxy collection install pulp.ansible-pulp:3.1.0 to install pulpcore 3.1.0. (After we implement publishing our roles as a collection on galaxy.)
They could still run ansible-galaxy collection install pulp.ansible-pulp for the latest pulpcore release.
For pre-releases (betas) and any unreleased commits on a branch, users would have to obtain from git (galaxy has no pre-releases), and/or there would be a variable to permit installing another version of pulpcore (pulp_source_dir for now.)
I estimate this would lead to an increase in users on older Z-releases, but not terribly so.
CI would need to be updated.
I'm worried that https://galaxy.ansible.com will always show the README for the latest version, and not the older versions (even when users select it), thus causing wrong instructions to be followed.

I'm researching this, testing things out, and thinking through it some more.

mikedep333 · 2020-05-04T19:08:06Z

FYI: We implemented the approach listed in the comment immediately above. Closing.

…Workflows Adapted from the prior design PR: pulp#203 fixes: #6874

mikedep333 added 5 commits January 6, 2020 15:21

The ansible-pulp installer needs to handle multiple versions of Pulp

ee340df

Proposed design for #5890 https://pulp.plan.io/issues/5890 [noissue]

Update README.md to specify the supported versions of Pulp

fa63e0a

Update README.md about versions

58162e4

Update README.md for pulp_branch being defaulted

0453c34

This enables the installer to apply logic specific to the Pulp branch, before it goes to install it.

Add Suggested Workflows for Pulpcore & Plugin Versioning

5755fa5

mikedep333 commented Jan 13, 2020

View reviewed changes

mdellweg reviewed Jan 16, 2020

View reviewed changes

mikedep333 closed this May 4, 2020

mikedep333 added a commit to mikedep333/pulp_installer that referenced this pull request Jun 1, 2020

Add a pulp_installer documentation section on Recommended Versioning …

c842a32

…Workflows Adapted from the prior design PR: pulp#203 fixes: #6874

mikedep333 mentioned this pull request Jun 1, 2020

Add a pulp_installer documentation section on Recommended Versioning … #313

Merged

mikedep333 added a commit to mikedep333/pulp_installer that referenced this pull request Jun 2, 2020

Add a pulp_installer documentation section on Recommended Versioning …

d178b21

…Workflows Adapted from the prior design PR: pulp#203 fixes: #6874

mikedep333 added a commit to mikedep333/pulp_installer that referenced this pull request Jun 2, 2020

Add a pulp_installer documentation section on Recommended Versioning …

45575f8

…Workflows Adapted from the prior design PR: pulp#203 fixes: #6874

mikedep333 mentioned this pull request Dec 14, 2020

Package: Honor specified versions #495

Merged

[Design RFC] The ansible-pulp installer needs to handle multiple versions of Pulp #203

[Design RFC] The ansible-pulp installer needs to handle multiple versions of Pulp #203

mikedep333 commented Jan 10, 2020 •

edited

mikedep333 Jan 13, 2020 •

edited

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020 •

edited

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020

mdellweg Jan 16, 2020 •

edited

mikedep333 Jan 16, 2020

mdellweg Jan 17, 2020

mdellweg Jan 17, 2020

mdellweg commented Jan 16, 2020

bmbouter commented Jan 23, 2020

mikedep333 commented Jan 28, 2020 •

edited

mikedep333 commented May 4, 2020 •

edited


		Updating your install:

		1. Check if `pulp_branch` has changed to a new stable branch in the latest version of the installer. This will happen whenever a new branch of `pulpcore` is released. Let's assume 3.0 is stil the latest, but there are new micro updates (like pulpcore 3.0.3 -> 3.0.4, and pulp_juicy 1.0.3 -> 1.0.4)

	3. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.
	3. Re-run the ansible-pulp role with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

	2. re-run the ansible-pulp with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.
	2. Re-run the ansible-pulp role with `update` set to `true` under `pulp_install_plugins`, and `pulp_update` set to `true`.

[Design RFC] The ansible-pulp installer needs to handle multiple versions of Pulp #203

[Design RFC] The ansible-pulp installer needs to handle multiple versions of Pulp #203

Conversation

mikedep333 commented Jan 10, 2020 • edited

mikedep333 Jan 13, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdellweg Jan 16, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdellweg Jan 16, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdellweg commented Jan 16, 2020

bmbouter commented Jan 23, 2020

mikedep333 commented Jan 28, 2020 • edited

mikedep333 commented May 4, 2020 • edited

mikedep333 commented Jan 10, 2020 •

edited

mikedep333 Jan 13, 2020 •

edited

mdellweg Jan 16, 2020 •

edited

mdellweg Jan 16, 2020 •

edited

mikedep333 commented Jan 28, 2020 •

edited

mikedep333 commented May 4, 2020 •

edited