Bump ansible-core controller minimum to Python 3.8

[nitzmahone]

SUMMARY

Testing and supporting older Pythons for the Ansible controller has an ever-increasing cost, especially as new Python releases are now coming yearly. With the sunset of Python 2.7, and in light of our current feature roadmap, we've set Python 3.8 as the minimum supported controller Python version for Ansible 2.11 and the release to follow it. To be clear: this does not affect the supported minimum Python version on remote targets (which will remain at Python 2.6 for the foreseeable future).

The Python 3.8 requirement for Ansible 2.11 will be "soft", in that breaking code changes that require Python 3.8 will not be made, but testing of older controller Python versions will stop during the 2.11 development cycle, and official Ansible packages will only be released for Python >= 3.8 (possibly including PyPI python_requires metadata, still TBD). The Ansible 2.12 controller will introduce code changes that will actively break compatibility with older Python versions.

The change will occur in a few phases:

• ansible-core 2.11 (currently devel branch) warning on Python < 3.8 (complete: Emit warning when running on the controller with a Python older than 3.8 #72467)
• Cessation of controller testing on Python < 3.8 (TBD, depends on CI changes to support split controller/remote testing)
• PyPI python_requires metadata update (TBD, if done, this would require passing --ignore-requires-python to pip to install on unsupported Python versions)
• ansible-core 2.11 beta/general release
• ansible-core 2.12 devel code changes that use Python 3.8 features

Additionally, there are related projects during 2.11 to maintain compatibility with controller-run modules like yum, dnf and selinux-aware modules that rely on OS-provided libraries like libselinux-python. The result will be that these modules will transparently run under the system Python version.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

[dmsimard]

o/

Would this also make 3.8 a requirement for plugins that run on the controller as well ? I'm thinking of things like action and callback plugins.

[sivel]

Would this also make 3.8 a requirement for plugins that run on the controller as well ? I'm thinking of things like action and callback plugins.

Yes.

[sivel]

bot_skip

[nirik]

So, from the angle of Fedora packages, this matters not at all to Fedora packaging. All active Fedora versions will be at least on 3.8 by the time this lands. However, EPEL is another story. ;( Right now epel7 and epel8 are using python-3.6. which is rhel7/8. Moving to 3.8 will break/cause both those to drop off.

Is there any thought to sticking with 3.6? What features are required to baseline at 3.8?

[gundalow]

Lots of feedback on this in Reddit https://www.reddit.com/r/ansible/comments/jxsc1x/rfc_python_38_for_ansible_controller_criteria_for/

[webknjaz]

@nirik RHEL8 has Python 3.8 and RHEL7 will probably only have Ansible Core 2.11 max. One of the deps will be libssh which is only packaged for RHEL8, for example.
Since a lot of stuff is now in Ansible Collections that are released separately, people will be able to keep receiving updates from those while using Ansible 2.8-2.11. And they'll get security patches for a while too. OTOH ancient CPythons will stop restricting us as much.

[dmsimard]

I am not familiar enough with python on windows and mac (if someone finds the info, I can add it to the table) but I went through the exercise of looking at the python versions that are shipped and available in a limited amount of relatively modern Linux distros:

	2.7	3.6	3.7	3.8	3.9	Supported until
rhel7/centos7	✔️	available	❌	❌	❌	June 30th, 2024
rhel8/centos8	available	✔️	❌	available	❌	May 31st, 2029 December 31, 2021
debian 10 buster	available	available	✔️	❌	❌	2024
ubuntu 18.04	available	✔️	available	available	❌	2023-04 / 2028-04
ubuntu 20.04	❌	❌	❌	✔️	available	2025-04 / 2030-04
fedora 32	available	available	available	✔️	available	2021-05-18
fedora 33	available	available	available	available	✔️	~2021-11

Sources for the table:

• https://wiki.centos.org/About/Product
• https://wiki.debian.org/Python
• https://en.wikipedia.org/wiki/Debian_version_history
• https://en.wikipedia.org/wiki/Ubuntu_version_history
• https://fedoraproject.org/wiki/SIGs/Python/features_history

In addition, data from pypistats.org shows that 2.7 is the most popular python version with 3.6 a distant second:

With that in mind, bumping the minimum version to 3.8 means that it would no longer be possible to install the latest version of ansible on RHEL7/Centos7 as well as any version of Debian -- Debian 11 "bullseye" will ship with 3.8 in 2021.

Although we can argue that alternatives exist (i.e, using an up-to-date distro as controller node, installing other python interpreters, running from a container, etc.), the result is that it will make Ansible harder and more complicated to run for a large amount of our users.

Would bumping the minimum to 3.6 be a good compromise for the time being ? It is still widely supported, available in all the distributions I've looked at and so would have less of an impact on the users.

[abadger]

@webknjaz A supported Python-3.8 can be installed on RHEL8 but that's not really the same thing as being shipped with RHEL8. RHEL8's python stack is built for Python-3.6. The other python interpreters which you can install don't have full stack support. Certain C-library bindings and python modules are only built for and supported on Python-3.6.

[sivel]

We are planning on dropping EL7, so that's not an issue. Ubuntu 18.04 does have py3.8.

Also, we have no plans of using 3.6. We might be swayed to use 3.7 as a minimum. But as of now, we have done the same research you have, and have still concluded we will set 3.8 as the minimum.

[dmsimard]

o/ @sivel

Ubuntu 18.04 does have py3.8.

Thanks, you're right -- I've updated the table.

We are planning on dropping EL7, so that's not an issue.

Do you know what that would look like in practice ? Would there be some sort of LTS release that would last until then ?

Also, we have no plans of using 3.6. We might be swayed to use 3.7 as a minimum. But as of now, we have done the same research you have, and have still concluded we will set 3.8 as the minimum.

This issue doesn't mention the research or the requirements that makes us settle on 3.8. Is that information available somewhere we can refer people to ?

[sivel]

Do you know what that would look like in practice ? Would there be some sort of LTS release that would last until then ?

At the moment Ansible 2.9 will be supported until 2023.

[webknjaz]

@dmsimard we don't care about windows because it's never been supported on the controller. But JFYI they usually install Python from the official website or "app store".
As for macos, the users can install any version from python.org but may also get it from homebrew or pyenv. So there's no problem there.

@abadger it sounds like 3.6 based bindings will be solved by calling the system python as a subprocess...

[nirik]

Not a major deal, but note for rhel/centos7: The chart has python 3.6 as "available (epel)" and it was, until rhel 7.6 was released. python 3.6 was shipped in rhel 7.6, so it should be supported until 7 is eol I would think.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.7_release_notes/index#enhancement_compiler-and-tools

@nirik RHEL8 has Python 3.8 and RHEL7 will probably only have Ansible Core 2.11 max. One of the deps will be libssh which is only packaged for RHEL8, for example.
Since a lot of stuff is now in Ansible Collections that are released separately, people will be able to keep receiving updates from those while using Ansible 2.8-2.11. And they'll get security patches for a while too. OTOH ancient CPythons will stop restricting us as much.

So, is there an expectation that collections should work with all ansible-base/engine/core version?
Or at least a requirement there that says what collections will work with what base/engine/core version?

Do you know what that would look like in practice ? Would there be some sort of LTS release that would last until then ?

At the moment Ansible 2.9 will be supported until 2023.

RHEL7 eol is June 30, 2024 it looks like.

[dmsimard]

Not a major deal, but note for rhel/centos7: The chart has python 3.6 as "available (epel)" and it was, until rhel 7.6 was released. python 3.6 was shipped in rhel 7.6, so it should be supported until 7 is eol I would think.

You're right, I forgot about that late python3 addition in the dot release.. fixed it.

So, is there an expectation that collections should work with all ansible-base/engine/core version?
Or at least a requirement there that says what collections will work with what base/engine/core version?

Collections have the ability to set supported versions of ansible in their meta/runtime.yml file like for example community.general: https://github.com/ansible-collections/community.general/blob/main/meta/runtime.yml#L2

[webknjaz]

So, is there an expectation that collections should work with all ansible-base/engine/core version?
Or at least a requirement there that says what collections will work with what base/engine/core version?

That may differ per collection depending on their maintainers but the mechanism in Core is in place for a while. My understanding is that collections may even have their own LTS and support matrix for a few Ansible versions back.

[felixfontein]

Collections have the ability to set supported versions of ansible in their meta/runtime.yml file like for example community.general: https://github.com/ansible-collections/community.general/blob/main/meta/runtime.yml#L2

Unfortunately this field is only read by ansible-base 2.10+, so if an Ansible 2.9 user is trying to use a collection that's not supported on 2.9, they will find out because something does not work, and potentially report a bug if they didn't read the documentation.

[webknjaz]

Yes, it may create minor inconveniences but I don't think that it's a major roadblock because the users will still be able to run things that support their Ansible, even if they may have to figure out which collection versions have that. Also, if that'll actually prove to be annoying we could raise a question about backporting this (although, it'll need a lot of negotiations/approvals if it'll ever come to actually doing this).

[bmillemathias]

for openSuse the version of python-3 is 3.6.10
https://software.opensuse.org/package/python3?search_term=%22python3%22

[zoredache]

Seems likely to make life more difficult for anyone using ansible-pull instead of the controller method of using ansible.

[webknjaz]

I think by the time people will actually want to use Ansible 2.12, those distros will bump their Python versions. Until that, they can keep using the supported Ansible version of 2.9 (which works with the content with collections just fine — actually, I think, Ansible 2.8 also works with collections).

[MarkusTeufelberger]

Major distros rarely if ever bump their supported versions of python3, they might just fall out of support instead or have newer releases (e.g. Debian 11 or RHEL 9). This still essentially kills ansible-pull as a viable deployment method on anything that's not the latest version of a (stable) distro. It also makes it really hard to use the latest version of Ansible on a developer machine that's updated a bit slower than "Oh, Ubuntu 20.04 is already out 3 days, time to run dist-upgrade!" (point aside that Ansible still doesn't provide CI or packages for 20.04 anyways).

It would be easier to understand the reasons behind this, if they are publicly communicated - especially since you did the analysis already outlined here and yet decided to go ahead anyways. Why?
The most prominent features compared to 3.6 that come to mind are better typing support and async stuff, but nothing that would make it completely impossible to support 3.6+ imho. So it seems a bit strange to already not test or release anything for 3.6 and 3.7 starting with the next minor version of Ansible (even with warnings that later 3.8+ will be required). Personally I'd rather see all currently supported versions of Python supported (and the current 'dev' branch of Python also tested to catch regressions early) on the controller.

Another point by the way: It is rare but not impossible to run Ansible through pypy3 for some performance gains or because pypy is also available as a standalone installation - the highest patch level available there is 3.7 (still in beta) or 3.6 (stable), 3.8 is not yet available at all and it seems like it'll still take a while until even 3.7 support is finished/stable there.

[webknjaz]

Major distros rarely if ever bump their supported versions of python3, they might just fall out of support instead or have newer releases (e.g. Debian 11 or RHEL 9).

This is why ansible-core will target envs like RHEL 8+.

This still essentially kills ansible-pull as a viable deployment method on anything that's not the latest version of a (stable) distro.

People don't seem to realize that you don't really need to use the latest ansible-core anymore. At least, it's not as vital now that the content is in the collections and the bottleneck will be what collections support, not the engine. And there's mechanisms to support collections in older versions of Ansible that support all of the cases you've described.

starting with the next minor version of Ansible

This sounds wrong: in Ansible release versions, a combination of the first two numeric version parts is considered a major release (2.8, 2.9, 2.10, 2.11 are all major releases).

Personally I'd rather see all currently supported versions of Python supported (and the current 'dev' branch of Python also tested to catch regressions early) on the controller.

They will still be supported, you'll just use Ansible 2.9 — it's not going EOL immediately.
Besides, by the time Ansible 2.12 is out the CPython usage in the wild will be different too.

Another point by the way: It is rare but not impossible to run Ansible through pypy3 for some performance gains

I doubt you'll get any visible performance gain: PyPy relies on JIT that is mostly useful for long-running processes. You'd still hit bottlenecks with the networking I/O and the remote code execution that spawns a new process for each task.

[MarkusTeufelberger]

This is why ansible-core will target envs like RHEL 8+.

Is there a list of officially supported Linux distributions (for controllers and target hosts) out there? I could not really find one.

People don't seem to realize that you don't really need to use the latest ansible-core anymore. At least, it's not as vital now that the content is in the collections and the bottleneck will be what collections support, not the engine. And there's mechanisms to support collections in older versions of Ansible that support all of the cases you've described.

As there is no separate collection for ansible.builtin modules, staying up to date with ansible-core is unfortunately still very relevant. Features in those modules are not backported to my knowledge and even rather trivial things (as a random example: #72763 - "just" a doc fix, I know...) are not ending up getting backported either. So far staying on an older version of ansible also just meant that you are accumulating a LOT of technical debt and making the upgrade more painful once support for your old version inevitably runs out. This also would mean that

This sounds wrong: in Ansible release versions, a combination of the first two numeric version parts is considered a major release (2.8, 2.9, 2.10, 2.11 are all major releases).

Ah well, seems like SemVer is only planned, not yet in effect (https://twitter.com/ansible/status/1329786737695092740). Also apparently only for ansible, not ansible-core, including version mismatches between releases and even independent releases on different time lines. Great... Not!

They will still be supported, you'll just use Ansible 2.9 — it's not going EOL immediately.
Besides, by the time Ansible 2.12 is out the CPython usage in the wild will be different too.

I'm not using 2.9 already, so downgrading is not really an attractive option. As 2.11 is already threatened to not get released on pypi for >=3.5 but only >=3.8 and that's going to happen in less than 6 months (hopefully), I don't see how CPython usage will shift significantly by then.

I'm just wondering why this (imho reasonable) policy has been suddenly changed without any public discussion or input:

https://docs.ansible.com/ansible/latest/dev_guide/developing_python_3.html#minimum-version-of-python-3-x-and-python-2-x

Python 3.5 was chosen as a minimum because it is the earliest Python 3 version adopted as the default Python by a Long Term Support (LTS) Linux distribution (in this case, Ubuntu-16.04). Previous LTS Linux distributions shipped with a Python 2 version which users can rely upon instead of the Python 3 version.

I also would understand the argument "ansible-core 2.11 will be released in Q2 2021, Ubuntu 16.04 is only in extended support by then, so the oldest LTS distro is ... instead, we use their python3 minor version as a minimum and prepare to increase/update this for ansible-core 2.12 accordingly, date TBD. Python2 support is dropped." - but jumping directly to a python version that's only the default on one single major LTS distro (which ironically is broken because there was no testing of it while it was not released and it still has no ansible or even ansible-base package for 2.10 available and is also not tested in CI #69203 and also no properly maintained downstream package exists (https://launchpad.net/ubuntu/+source/ansible/+bugs), so one kinda has to use the broken PPA or *shudder* pip) is really weird.

I doubt you'll get any visible performance gain

Have you actually benchmarked this? For a playbook that did 0 network calls (just assertions on host variables and inventory contents like "hosts named foo must be in group bar") there can be quite some differences for example. For network IO and the core design flaw of always spawning new processes finally mitogen has a new release out so Ansible 2.10 becomes semi-bearable again, but that's a different topic alltogether. It is also besides my point: Currently it is possible to run ansible-playbook using pypy3 for whatever reason, with these proposed changes it will no longer be supported. It might be a minor regression, but nevertheless something to keep in mind.

[felixfontein]

This sounds wrong: in Ansible release versions, a combination of the first two numeric version parts is considered a major release (2.8, 2.9, 2.10, 2.11 are all major releases).

Ah well, seems like SemVer is only planned, not yet in effect (https://twitter.com/ansible/status/1329786737695092740). Also apparently only for ansible, not ansible-core, including version mismatches between releases and even independent releases on different time lines. Great... Not!

For Ansible < 3 and ansible-base/ansible-core, 2.x is a major release.

Right now, only Ansible >= 3 will use semantic versioning. The 3.0.0 release will follow the Ansible 2.10.x line, which is ansible-base 2.10 + collections. Ansible 3.y.z will also be ansible-base 2.10.x + collections, and Ansible 4.y.z will be ansible-base 2.11.x + collections.

(More about this at https://hackmd.io/y7BBcweNR3aRVLuMbKkDxw for more details).

It might be helpful for this discussion to avoid using the name "Ansible" for ansible-base / ansible-core. That only makes it more confusing.

[webknjaz]

This is why ansible-core will target envs like RHEL 8+.

Is there a list of officially supported Linux distributions (for controllers and target hosts) out there? I could not really find one.

Mostly envs that run in CI can also be considered supported.

As there is no separate collection for ansible.builtin modules, staying up to date with ansible-core is unfortunately still very relevant.

I think this is something to be raised separately, maybe with the community team, to see if it's possible to extract this content.

I doubt you'll get any visible performance gain

Have you actually benchmarked this?

No. This is not something that anybody even brought up AFAIK.

For a playbook that did 0 network calls (just assertions on host variables and inventory contents like "hosts named foo must be in group bar") there can be quite some differences for example.

This is a corner case that may get a bit faster, yes. But I don't think that it's a primary use-case.

For network IO and the core design flaw of always spawning new processes finally mitogen has a new release out so Ansible 2.10 becomes semi-bearable again, but that's a different topic alltogether.

I believe we've absorbed all the improvements we could from mitogen. There are things that are not possible to migrate into ansible-core because they would break corner-cases other people rely on. If you didn't hit those cases it doesn't mean that others won't. So no, it's not a design flaw at all — it's rather a mitogen's downside that it breaks some things (even though it doesn't affect everyone).

It is also besides my point: Currently it is possible to run ansible-playbook using pypy3 for whatever reason, with these proposed changes it will no longer be supported. It might be a minor regression, but nevertheless something to keep in mind.

We've never tested against PyPy, nor declared compatibility. So calling it a breaking change is wrong. Only tested envs are truly supported. You cannot say something is a regression when we never said that it'd work in the first place.

It might be helpful for this discussion to avoid using the name "Ansible" for ansible-base / ansible-core. That only makes it more confusing.

True, I'll edit the topic to mention ansible-core.

[abadger]

As there is no separate collection for ansible.builtin modules, staying up to date with ansible-core is unfortunately still very relevant.

I think this is something to be raised separately, maybe with the community team, to see if it's possible to extract this content.

This would be a core team decision to make. It makes sense to me that you'd want to maintain it separately (but use dependencies to require both are installed... Not strictly version locked, though). If you also want to stop maintaining those plugins, you'd probably want to talk to the content team about them taking over

[felixfontein]

Since the name ansible.builtin is treated specially, it would probably make sense to use a new name, like ansible.core or ansible.base or ansible.basic, and use Ansible's https://github.com/ansible/ansible/blob/devel/lib/ansible/config/ansible_builtin_runtime.yml to redirect all ansible.builtin.<oldname>s to the new collection. If this is an option at all.

[dtantsur]

Late for the party, but want to add an important comment:

RHEL8 has Python 3.8

While this is true, it lacks non-pip-installable modules, such as python3-firewalld, which makes the firewalld module impossible to use. Same applies to python3-dnf, but I guess Ansible is working around it.

[felixfontein]

@dtantsur ansible-core 2.12 can happily run modules even on Python 2.6, it just needs 3.8+ on the target side. (Also some modules, such as dnf and apt, now have some code to try other Python interpreters automatically to find one which supports the required libraries.)

[sivel]

Didn't realize this issue was still open. This has already been implemented, and ansible-core 2.12 has already been released.