Python 3.12 support #1033
Comments
upekkha
added
affects-0.3
|
At the moment, python 3.12 is only supported on ansible-core 2.16. |
|
Agreed, but despite the lack of official support of Python 3.12, I had no problems running various playbooks with Python3.12 and ansible-core 2.14, 2.15 or 2.16 - but unfortunately only without mitogen. |
Yeah, this won't be trivial. |
|
Good news. I had a bit of time today to spend on this, and after 8 hours, got it working. I managed to make mitogen work with Python 3.12. And with Ansible 2.16.2 as a bonus. Ansible 2.16.x requires Python 3.8 (due to use of Verified configurations: Host:
Targets:
And as expected, it works smoothly. On a high latency link (225ms), with medium size playbooks (55 tasks), it finishes in 84 seconds vs 169 seconds without mitogen. I will do some more testing with Host being ansible 2.14.x, 2.15.x, and host using Python 3.6, 3.9, 3.11. I do not have any older machines anymore, but I can probably spin-up some VM for a test. The required modifications are a bit of ad-hoc, but are not too big but I should clean it up, and wrap in various conditions and try blocks, so it also works with Python 3.3 and older. (30% of this is extra debugging, that was added to help with fixing the issue; and another 10% are some patches from unmerged MRs, that I incorporated just in case) Attaching a draft patch version for now, in case, I forgot about this. |
|
Ok, I found few failing roles with my patch, so still requires some work.
This is with a target being Python 3.6.8, Centos 8. It is a bit strange, because I have docker python packages for docker on the target host, and it works (tested in the python interpreter there). There is no Tracing |
|
More details about the It looks like This causes a minor problem here. Indeed, that does not work on Python 3.6: This is numpy 1.24.4 (from fedora repos: python3-numpy-1.24.4-2.fc39.x86_64, not from pypi) I will contact For now workaround is to switch to All that said, Another option is to change this ( to also catch But I will also check my patches, why we send our version of Fortunately for now, there is an easy workaround, and it should affect very few roles / packages. |
|
Updated and slightly improved patch (to not use |
|
@baryluk: That's looking a lot more polished. You planning to file a PR? |
|
@stefanor Yes, absolutely will make a PR. Need to test it a bit more. (I do not run ansible every day, but I do run few times a week). Tested on two more plays today, all working perfectly. I will try to send PR this week. |
|
Found issue with this patch for mitogen, tested with ansible 2.15.8 (python 3.9.3) and ansible 2.16.2 (python 2.11.6) and destination host and Task Error: |
|
If to install on target |
|
@kzinas-adv I see. Thanks for the info. It looks like it tries to load cacert.pem (to validate s3 SSL certificates) as a package resource. It is not very common usage of imports in Python, but definitively something worth adding support for. I will see if I can add a support for this. |
|
Tried with python3.12 (Fedora 38 made with virtual env) Did not get where I am wrong? Patch is applied. |
If you are using a virtual env you will need to install the zombie-imp package. On fedora it looks like that package is installed as part of the system so it works outside of the virtual env. The description of the v2 patch says that you do not need it, but there still seems to be many 'import imp' statements, so that conversion doesn't seem to be complete? Anyway it is working very well for me with that install. |
|
You should not need If that still does not work, yes, please install |
I see. There is still one in Will send |
|
More issues when target is centos8(python3.6.8), host fedora39(3.12.1): |
|
@kzinas-adv This was already mentioned in the very first comment I posted, second paragraph - #1033 (comment) I also posted there a workaround. There might be other way out, but it is rather complicated. |
|
Workaround is to have python 3.8 on the target? |
Either that, or manually patch |
With current macOS 11 runner images (20231216.1) the `python` on `$PATH` is
Python 3.12 and setuptools isn't installed by default. E.g.
```
python -mtox -e "py27-mode_localhost-ansible4"
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc
/Users/runner/work/_temp/93a29c4c-f606-45e4-8dbd-a4a5f51b8730.sh
GLOB sdist-make: /Users/runner/work/1/s/setup.py
ERROR: invocation failed (exit code 1), logfile:
/Users/runner/work/1/s/.tox/log/GLOB-0.log
================================== log start
===================================
Traceback (most recent call last):
File "/Users/runner/work/1/s/setup.py", line 32, in <module>
from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
```
Installing setuptools under Python 3.12 chooses package versions incompatible
with Python 2.7. Additionally Mitogen isn't yet compatible with Python 3.12
(mitogen-hq#1033), so tests that call a local context with `python` fail.
|
I didn't finishe the v3 version of the patch, with the migogen/master.py ported fully to importlib. But it is shaping up. |
|
@baryluk I've got some Fedora 39 systems now with python 3.12, so I'm eagerly awaiting your next version. Thank you very much for working on this! |
With current macOS 11 runner images (20231216.1) the `python` on `$PATH` is
Python 3.12 and setuptools isn't installed by default. E.g.
```
python -mtox -e "py27-mode_localhost-ansible4"
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc
/Users/runner/work/_temp/93a29c4c-f606-45e4-8dbd-a4a5f51b8730.sh
GLOB sdist-make: /Users/runner/work/1/s/setup.py
ERROR: invocation failed (exit code 1), logfile:
/Users/runner/work/1/s/.tox/log/GLOB-0.log
================================== log start
===================================
Traceback (most recent call last):
File "/Users/runner/work/1/s/setup.py", line 32, in <module>
from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
```
Installing setuptools under Python 3.12 chooses package versions incompatible
with Python 2.7. Additionally Mitogen isn't yet compatible with Python 3.12
(mitogen-hq#1033), so tests that call a local context with `python` fail.
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4 under PEP 451. They replace the find_module() API of PEP 302, which was deprecated from Python 3.4. They were removed in Python 3.12 along with the imp module. This change adds support for the PEP 451 APIs. Mitogen should no longer import imp on Python versions that support ModuleSpec. Tests have been added to cover the new APIs. CI jobs have been added to cover Python 3.x on macOS. Refs mitogen-hq#1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4 under PEP 451. They replace the find_module() API of PEP 302, which was deprecated from Python 3.4. They were removed in Python 3.12 along with the imp module. This change adds support for the PEP 451 APIs. Mitogen should no longer import imp on Python versions that support ModuleSpec. Tests have been added to cover the new APIs. CI jobs have been added to cover Python 3.x on macOS. Refs mitogen-hq#1033
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4 under PEP 451. They replace the find_module() API of PEP 302, which was deprecated from Python 3.4. They were removed in Python 3.12 along with the imp module. This change adds support for the PEP 451 APIs. Mitogen should no longer import imp on Python versions that support ModuleSpec. Tests have been added to cover the new APIs. CI jobs have been added to cover Python 3.x on macOS. Refs mitogen-hq#1033
Most of the necessary changes were made in recent PEP 451 commits. This bumps the CI jobs, and declares the support. Test dependendancies are bumped to latest supportted/available versions. refs mitogen-hq#1033
|
Sorry for the long wait, but #1032 is ready for wider scrutiny. I'm seeking code review comments, and reports of anyone trying the branch in the wild moreati:docs-download-url. Note that on Python 3.12 it will require Ansible 6 (ansible-core 2.13). On Python 2.7 & 3.6-3.11 the supported Ansible versions are unchanged. @baryluk thanks for your work on this, would you like to be added to https://github.com/mitogen-hq/mitogen/blob/master/docs/contributors.rst? |
|
@moreati Thanks for picking this up. I had few more minor updates (and removal of uneeded changes in my patch), but it was progressing slowly (because it kind of works, and I had other work to do, so didn't spend much on mitogen). Your looks way better (including tests, and supporting legacy versions). Yes, if you borrowed any code, please add me to the list, and we go with your changes. Thanks! |
importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4 under PEP 451. They replace the find_module() API of PEP 302, which was deprecated from Python 3.4. They were removed in Python 3.12 along with the imp module. This change adds support for the PEP 451 APIs. Mitogen should no longer import imp on Python versions that support ModuleSpec. Tests have been added to cover the new APIs. CI jobs have been added to cover Python 3.x on macOS. Refs mitogen-hq#1033 Co-authored-by: Witold Baryluk <witold.baryluk@gmail.com>
Most of the necessary changes were made in recent PEP 451 commits. This bumps the CI jobs, and declares the support. Test dependendancies are bumped to latest supportted/available versions. refs mitogen-hq#1033
|
Mitogen 0.3.5 released, with Python 3.12 support. |
Python 3.12 has removed the
impmodule. It was deprecated since Python 3.7, but the deprecation warning ignored by mitogen since 2018 (c2c7caa).The suggested path forward would be to switch to using
importlibinstead. Some correspondences are easy (egimp.get_magic()->importlib.util.MAGIC_NUMBER), but others look non-trivial and require extra care to keep supporting old Python versions.I also quickly tried the hack of copying the old
imp.pyfrom Python 3.11 intomitogen/compatand basically replacingimport impwithimport mitogen.compat.imp as imp. This actually allowed me to import mitogen locally without any error messages, but running an ansible playbook fails withLet's hope the mitogen devs find some time looking into this to keep the project alive with Python 3.12+.
The text was updated successfully, but these errors were encountered: