New Policy Regarding Python Compatibility for Collections

[Ompragash]

SUMMARY

• Added new policy regarding python compatibility for Collections.

ISSUE TYPE

• Docs Pull Request

[felixfontein]

Shouldn't we say something about which exceptions are ok (like not supporting Python 2.6 or Python 2 or Python 3.5, if explicitly documented)? And what we consider not to be a good idea (only supporting Python 3.8/3.9 for example)?

[Ompragash]

@felixfontein Yeah that'll be a good idea but again I think it'll conflict with what we say in CI Testing section below. Any thoughts?

Collections must run an equivalent of ansible-test sanity --docker. If they do not use --docker, they must make sure that all tests run, in particular the compile and import tests (which should run for all Python versions). Collections can choose to skip certain Python versions that they explicitly do not support; this needs to be documented in README.md and in every module and plugin (hint: use a docs fragment).

[pabelanger]

I think we should be more specific here.

python2.6 / python2.7 / python3.5 are all EOL. Expecting a new collection to support them, doesn't make too much sense. I think it is fair to say these are optional, if you want to support them.

https://www.python.org/downloads/ could be a good format

[gundalow]

See ansible/ansible#72668 (comment) which shows that some OS still support Python 2.7 till 2024

[jamescassell]

I think 2.6+ and 3.5+ is the right set.

[felixfontein]

I think the mention in README.md should mainly be required if most/all modules/plugins have this requirement, not if some random modules/plugins have different requirements. Alternatively, we should accept a blanket statement like Most modules and plugins support all Python versions supported by Ansible. Exceptions are noted in the module/plugin documentation.. This is really necessary for huge collections such as community.general and community.network, where keeping a list of modules/plugins with special requirements in README.md is really painful :)

[Andersson007]

I think the mention in README.md should mainly be required if most/all modules/plugins have this requirement, not if some random modules/plugins have different requirements. Alternatively, we should accept a blanket statement like Most modules and plugins support all Python versions supported by Ansible. Exceptions are noted in the module/plugin documentation.. This is really necessary for huge collections such as community.general and community.network, where keeping a list of modules/plugins with special requirements in README.md is really painful :)

+1 for the alternative:)

[felixfontein]

@Ompragash we were discussing about Python versions during today's community meeting. We agreed by vote on the following:

• Collections do not need to support Python 2.6 and Python 3.5 if they chose not to. This must be well documented though (for every plugin/module, and in the collection README).
• Add Collections SHOULD try to support 2.6 and 3.5 as well. Not supporting Python 2.6 means that you are dropping support for RHEL6, which has extended support until 2024. Not supporting Python 3.5 means that Python 2.7 has to be installed on Ubuntu Xenial (16.04), and that you have to support Python 2.7. Also note that dropping support for a Python version for an existing module/plugin is a breaking change, and thus requires a major release.

Can you integrate / add this to your proposal?

Also, before we switched topic, I proposed for the remaining versions (2.7, 3.6, 3.7, 3.8, ...): "we could require that these are supported (and 3.8 and 3.9), with the possible exception that a library used does not support them". We wanted to continue discussing this the next meeting. Do you have suggestions, or other / alternative suggestions?

It would probably be great if we could have a version of this PR that reflects something like all the above, so that we can vote on the PR (and merge if it is approved). Or at least have a new version so it can be approved the week thereafter.

[pabelanger]

Also, before we switched topic, I proposed for the remaining versions (2.7, 3.6, 3.7, 3.8, ...): "we could require that these are supported (and 3.8 and 3.9), with the possible exception that a library used does not support them". We wanted to continue discussing this the next meeting. Do you have suggestions, or other / alternative suggestions?

It would probably be great if we could have a version of this PR that reflects something like all the above, so that we can vote on the PR (and merge if it is approved). Or at least have a new version so it can be approved the week thereafter.

I couldn't attend the meeting, but something both @gundalow and @abadger referenced hits at the heart of it.

Controller vs remote-node is a good point

In the case of network, a lot of the content is run on the controller side. Given that ansible-core has stricter python requirements moving forward, we'd like the ability for that too. ansible-core 2.11 / 2.12 would never support python2.7 / python3.6 / python3.7 on the controller side, having the collection support it seems inconsistent. That really the gist of our point.

[felixfontein]

@pabelanger while ansible-core 2.11 is planning to announce that it requires Python 3.8+, it still works fine with Python 2.7 and Python 3.5+. So the restricting to Python 3.8+ on the controller side effectively won't happen before ansible-core 2.12, and thus not before Ansible 5.0.0.

[pabelanger]

@pabelanger while ansible-core 2.11 is planning to announce that it requires Python 3.8+, it still works fine with Python 2.7 and Python 3.5+. So the restricting to Python 3.8+ on the controller side effectively won't happen before ansible-core 2.12, and thus not before Ansible 5.0.0.

That is true, but there is a rather large warning in 2.11 that if you are running less then python3.8 you get a big deprecation warning:

ansible/ansible#72467

Which I read it, it works but should really be using python3.8. I think it is a fair ask for a collection to decide if they want to even support deprecated python versions or make the jump to the latest sooner.

[Ompragash]

• Add Collections SHOULD try to support 2.6 and 3.5 as well. Not supporting Python 2.6 means that you are dropping support for RHEL6, which has extended support until 2024. Not supporting Python 3.5 means that Python 2.7 has to be installed on Ubuntu Xenial (16.04) and that you have to support Python 2.7. Also, note that dropping support for a Python version for an existing module/plugin is a breaking change, and thus requires a major release.

@felixfontein adding the details around Python and OS supportability seems reasonable and I'll add the above changes to this PR that reflects like below:

Collection should be developed and tested against both Python 2.6 (or greater) and Python 3.5 (or greater) as Ansible supports a wide variety of machines and also it should adhere to the tips mentioned in the official Ansible Development Guide [here](https://docs.ansible.com/ansible/latest/dev_guide/developing_python_3.html#ansible-and-python-3).

Also note, if the Collections does not support Python 2.6 and Python 3.5 explicitly then kindly take the below points into consideration:

- Not supporting Python 2.6 means that you are dropping support for RHEL6, which has extended support until 2024. 

- Not supporting Python 3.5 means that Python 2.7 has to be installed on Ubuntu Xenial (16.04) and that you have to support Python 2.7. 

- Also, note that dropping support for a Python version for an existing module/plugin is a breaking change, and thus requires a major release.

[felixfontein]

@Ompragash looks good. Can you uppercase the SHOULD in the first sentence? Makes it more clear (and more similar to RFCs ;) ).

[Ompragash]

@felixfontein consider it done :)

[abadger]

Hi @Ompragash ! Despite being superficially simple, this is turning out to be a controversial policy :-) We have about five points of contention to resolve regarding this and at yesterday's meeting we made decisions on two of them which are related:

One point we think is important is to have different Python requirements for the controller-side and remote-side. On the controller-side, the python versions required may be higher than what is required on the remote-side. So we would like two sets of python requirements rather than the current draft's single set. (Deciding what these versions are is one of the remaining points of contention).

Figuring that out then lead us to the need to define what controller-side and remote-side means. We decided on the following definitions:

• controller-side: It's impossible for the plugin/module to run remotely
• remote-side: It is possible for the plugin/module to run remotely, even if it is uncommon in practice.

One example where the even if clause comes into play is with cloud modules. These are commonly run on the controller but in some environments, the controller might run on one system inside a corporate network which can't directly access the cloud machines. The user has to have the modules run on a bastion host which has access to the cloud.

We also note that the requirements for module_utils needs to be hashed out. Networking says they have some things which are modules (and so they can only pull their requirements from module_utils) but it's only possible to use the modules controller-side. Since we can't control what third parties do with those module_utils, we may want to make them obey the remote-side requirements. OTOH, that means the module_utils will have to support a larger set of python versions than the networking modules which form the bulk of their usage.

The below proposals are the verbatim decisions we approved yesterday:

PROPOSAL: Ask that the draft define controller-side and remote-side. If it's impossible for the plugin/module to rub remotely, then it can use the controller-side python requirements. If it is possible, just uncommon in practice, that it can run remotely, then it has to follow the remote side python requirements.

PROPOSAL: Ask that the draft define controller-side and remote-side like this: If it's impossible for the plugin/module to run remotely, then it can use the controller-side python requirements. If it is possible, just uncommon in practice, that it can run remotely, then it has to follow the remote side python requirements. Note that how module_utils falls into this needs to be hashed out

[felixfontein]

At today's community meeting, we had some more decisions for Python version requirements:

• We keep our previous voted on proposal: require 2.7 and 3.6+ support both for remote-side and controller-side, and suggest to also support 2.6 on remote-side and 3.5 on both sides (always assuming required libraries support this)
• The minimal bounds are adjusted when the ansible-core we depend on stops running on specific Python versions on remote or controller side
• convention for module_utils: a) no docstring = everything we recommend for remote-side is supported, b) docstring 'Python versions supported: as for controller-side' = everything we recommend for controller-side is supported, c) docstring with specific versions otherwise: 'Python versions supported: '
• module_utils can be marked for only internal use in the collection, but they MUST document this and MUST use a leading underscore for filenames. Making an existing module_utils private is a breaking change and requires a major version bump.
• collections MUST announce dropping support for Python versions in their changelog, if possible in advance (i.e. announce in previous versions when support will be dropped); if collections are aware of collection users (which import their code), they SHOULD announce it directly to their known users, and/or to Changes impacting Collection Contributors and maintainers #45, and/or The Bullhorn

[Ompragash]

@abadger @felixfontein I modified the proposal based on the discussions/suggestions and also do you think we can change the term controller-side/remote-side to control node/managed node to incline more towards common Ansible terminologies. Regarding module_utils can we have a separate section under Python Compatibility that says Standards and conventions while developing module utilities for collections and put in all the details?

#Python Compatibility
=================

Collection SHOULD be developed and tested using the below Python recommendations as Ansible supports wide variety of machines and also it should adhere to the tips mentioned in the official [Ansible Development Guide](https://docs.ansible.com/ansible/latest/dev_guide/developing_python_3.html#ansible-and-python-3).

**Python Requirements**

Python requirements for a collection varies between controller-side and remote-side. On the controller-side, the python versions required may be higher than what is required on the remote-side. While developing a collection, you need to understand the definitions of both  controller-side and remote-side that will help choose Python versions accordingly: 

- controller-side: It is impossible for the plugins/modules to run remotely
- remote-side: It is possible for the plugins/modules to run remotely, `even if` it is uncommon in practice.

One ideal scenario where the `even if` clause comes into play is when using Cloud modules. These modules mostly run on the controller node but in some environments, the controller might run on one machine inside a demilitarized zone which can't directly access the cloud machines. The user has to have the cloud modules run on a bastion host/jump server which has access to the cloud machines.

**Controller-side**

On the controller node, collection should support Python 2 (version 2.7) and Python 3 (Version 3.6 and higher) . We also suggest to support Python v3.5 if in case the required libraries support this version. 

**Remote-side**

On the managed node, collection needs to support Python 2 (version 2.7) and Python 3 (Version 3.6 and higher) but we also suggest to support Python version 2.6 and 3.5 based on the following points.

**Note**

If the Collections does not support Python 2.6 and/or Python 3.5 explicitly then kindly take the below points into consideration:

- Not supporting Python 2.6 on the managed node means that you are dropping support for RHEL6, which has extended support until 2024. 

- Not supporting Python 3.5 means that Python 2.7 has to be installed on Ubuntu Xenial (16.04) and that you have to support Python 2.7.

- Also, note that dropping support for a Python version for an existing module/plugin is a breaking change, and thus requires a major release. Hence, a collection MUST announce dropping support for Python versions in their changelog, if possible to announce in advance (i.e. in previous versions when support will be dropped).

[Andersson007]

we can change the term controller-side/remote-side to control node/managed node

"target" host/node is also an option

[abadger]

@Ompragash Thanks for pointing that out. We talked about it in the meeting today and decided that "remote" and "side" were both confusing. So we came up with some different terms which are less linked to the ansible controller node/managed node concept. How's this for the new terms and definitions:

• controller-environment: The plugins/modules always run in the same environment (Python interpreter, venv, host, etc) as ansible-core itself
• other-environment: It is possible, even if uncommon in practice, for the plugins/modules to run in a different environment than ansible-core itself.

[abadger]

And yes, I think a separate section for module_utils is fine.... they're the part with the most special cases so it makes sense that they have their own subsection

[Ompragash]

Suggested changes applied and below is the updated proposal:

Python Compatibility
=================

Collection SHOULD be developed and tested using the below Python recommendations as Ansible supports wide variety of machines and also it should adhere to the tips mentioned in the official [Ansible Development Guide](https://docs.ansible.com/ansible/latest/dev_guide/developing_python_3.html#ansible-and-python-3).

**Python Requirements**

Python requirements for a collection varies between controller-environment and other-environment. On the controller-environment, the python versions required may be higher than what is required on the other-environment's. While developing a collection, you need to understand the definitions of both  controller-environment and other-environment that will help choose Python versions accordingly: 

- `controller-environment`: The plugins/modules always run in the same environment (Python interpreter, venv, host, etc) as ansible-core itself.
- `other-environment`: It is possible, even if uncommon in practice, for the plugins/modules to run in a different environment than ansible-core itself.

One ideal scenario where the `even if` clause comes into play is when using Cloud modules. These modules mostly run on the controller node but in some environments, the controller might run on one machine inside a demilitarized zone which can't directly access the cloud machines. The user has to have the cloud modules run on a bastion host/jump server which has access to the cloud machines.

**Controller-environment**

On the controller node, collection should support Python 2 (version 2.7) and Python 3 (Version 3.6 and higher) . We also suggest to support Python v3.5 if in case the required libraries support this version. 

**Other-environment**

On the managed node, collection needs to support Python 2 (version 2.7) and Python 3 (Version 3.6 and higher) but we also suggest to support Python version 2.6 and 3.5 based on the following points.

**Note**

If the Collections does not support Python 2.6 and/or Python 3.5 explicitly then kindly take the below points into consideration:

- Not supporting Python 2.6 on the managed node means that you are dropping support for RHEL6, which has extended support until 2024. 

- Not supporting Python 3.5 means that Python 2.7 has to be installed on Ubuntu Xenial (16.04) and that you have to support Python 2.7.

- Also, note that dropping support for a Python version for an existing module/plugin is a breaking change, and thus requires a major release. Hence, a collection MUST announce dropping support for Python versions in their changelog, if possible to announce in advance (i.e. in previous versions when support will be dropped).


**Standards and conventions while developing module utilities for collection**

- `module_utils` can be marked for only internal use in the collection, but they MUST document this and MUST use a leading underscore for filenames.

- It is a breaking change when you make an existing `module_utils` private and in that case the collection requires a major version bump.

- Below are some recommendations for `module_utils` documentation: 

1. no docstring = everything we recommend for `other-environment` is supported

2. docstring 'Python versions supported: as for controller-environment' = everything we recommend for `controller-environment` is supported

3. docstring with specific versions otherwise: 'Python versions supported: '

[felixfontein]

We decided during today's meeting that the descriptions in **Controller-environment** and **Other-environment** should be as follows:

"In the controller environment, collections MUST support Python 2 (version 2.7) and Python 3 (Version 3.6 and higher), unless required libraries do not support these versions. Collections SHOULD also support Python v3.5 if all required libraries support this version."

"In the other environment, collections MUST support Python 2 (version 2.7) and Python 3 (Version 3.6 and higher), unless required libraries do not support these versions. Collections SHOULD also support Python v2.6 and v3.5 if all required libraries support this version.

It is important that we don't use controller node / managed node here, as this is not about nodes, but about environments.

[Ompragash]

@felixfontein @abadger @gundalow updated the proposal and committed the changes as well! Kindly review.

[samccann]

nits

[felixfontein]

As discussed and voted on in today's community meeting, we merge this with the above suggestions added.

[felixfontein]

@Ompragash thanks a lot for working on this for so long! I'm really happy we finally got this endless topic merged ;)