Support Python 3.11

[jiasli]

Description

Python 3.11 has been released on 2022-10-24: https://www.python.org/downloads/release/python-3110/

Azure CLI needs to support Python 3.11 like how we supported 3.9 (#16611), 3.10 (#19857).

TODOs

• Knack supports Python 3.11
  • Declare support for Python 3.11 and drop support for Python 3.7 microsoft/knack#275
• azdev supports Python 3.11
  • pylint (Bump Pylint to 2.17 and flake8 to 6.0 azure-cli-dev-tools#359)
  • vcrpy (same as above)
• Dependencies are updated to the Python 3.11 compatible version
  • psycopg2 ({Packaging} Update psycopg2 to support Python 3.11 #24671)
  • pywin32 ({Packaging} Bump PyWin32 to 305 #24982)
  • cffi psutil ({Packaging} Bump cffi to 1.15.1 and psutil to 5.9.5 #26842)
• CI passes with Python 3.11
  • {Core} Make parser compatible with Python 3.11's argparse #24497
  • {Misc.} Improve code style and prepare for pylint 2.17 #26685
  • {CI} Make .flake8 compatible with Flake8 6.0.0 #25370
  • [Packaging] Support Python 3.11 #26923
    • {Compute} Change KeyVaultClient to CertificateClient in vm module #26991
    • {Keyvault} Vendor azure-keyvault track1 SDK to make customization for supporting Python 3.11 #27148
    • {aaz} Rename aaz profile package name by adding profile_ prefix for azure stack profiles. #27073

[jiasli]

psycopg2 (introduced by #22459) fails to install on Python 3.11:

Collecting psycopg2==2.9.3
  Downloading psycopg2-2.9.3.tar.gz (380 kB)
     ------------------------------------- 380.6/380.6 kB 23.1 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error

  python setup.py egg_info did not run successfully.
  exit code: 1

  [23 lines of output]
  running egg_info
  creating C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-4nrehwws\psycopg2.egg-info
  writing C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-4nrehwws\psycopg2.egg-info\PKG-INFO
  writing dependency_links to C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-4nrehwws\psycopg2.egg-info\dependency_links.txt
  writing top-level names to C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-4nrehwws\psycopg2.egg-info\top_level.txt
  writing manifest file 'C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-4nrehwws\psycopg2.egg-info\SOURCES.txt'

  Error: pg_config executable not found.

  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:

      python setup.py build_ext --pg-config /path/to/pg_config build ...

  or with the pg_config option in 'setup.cfg'.

  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.

  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).

  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Encountered error while generating package metadata.

[bebound]

I guess we need to update psycopg2 to 2.9.5
psycopg/psycopg2#1514

[bebound]

azdev's dependency pylint==2.11.1 breaks vcrpy, as wrapt 1.13 does not support 3.11.
wrapt support 3.11 in 1.14 https://wrapt.readthedocs.io/en/latest/changes.html#version-1-14-0
pylint will support 3.11 in 2.16

Version 1.14.0
Bugs Fixed
Python 3.11 dropped inspect.formatargspec() which was used in creating signature changing decorators. Now bundling a version of this function which uses Parameter and Signature from inspect module when available. The replacement function is exposed as wrapt.formatargspec() if need it for your own code.

├── pylint==2.11.1                                                                                  python code static checker
│   ├── astroid<2.9,>=2.8.0                                                                         An abstract syntax tree for Python with inference support.
│   │   ├── lazy-object-proxy>=1.4.0                                                                A fast and thorough lazy object proxy.
│   │   ├── setuptools>=20.0                                                                        Easily download, build, install, upgrade, and uninstall Python packages
│   │   └── wrapt<1.14,>=1.11

ImportError while importing test module '/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/network/tests/latest/test_private_endpoint_commands.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/azure-cli/azure/cli/command_modules/network/tests/latest/test_private_endpoint_commands.py:13: in <module>
    from azure.cli.testsdk import (
src/azure-cli-testsdk/azure/cli/testsdk/__init__.py:6: in <module>
    from .scenario_tests import live_only, record_only, get_sha1_hash
src/azure-cli-testsdk/azure/cli/testsdk/scenario_tests/__init__.py:13: in <module>
    from .base import IntegrationTestBase, ReplayableTest, LiveTest
src/azure-cli-testsdk/azure/cli/testsdk/scenario_tests/base.py:14: in <module>
    import vcr
env/lib/python3.11/site-packages/vcr/__init__.py:2: in <module>
    from .config import VCR
env/lib/python3.11/site-packages/vcr/config.py:11: in <module>
    from .cassette import Cassette
env/lib/python3.11/site-packages/vcr/cassette.py:8: in <module>
    import wrapt
env/lib/python3.11/site-packages/wrapt/__init__.py:10: in <module>
    from .decorators import (adapter_factory, AdapterFactory, decorator,
env/lib/python3.11/site-packages/wrapt/decorators.py:34: in <module>
    from inspect import ismethod, isclass, formatargspec
E   ImportError: cannot import name 'formatargspec' from 'inspect' (/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/inspect.py)

[bebound]

vcrpy supports 3.11 in kevin1024/vcrpy#667, but it's dependency multidict does not provide wheels for 3.11. Here is the issue Wheels for CPython 3.11.

vcrpy                   Automatically mock your HTTP interactions to simplify and speed up testing
├── PyYAML              YAML parser and emitter for Python
├── six>=1.5            Python 2 and 3 compatibility utilities
├── wrapt               Module for decorators, wrappers and monkey patching.
└── yarl                Yet another URL library
    ├── idna>=2.0       Internationalized Domain Names in Applications (IDNA)
    └── multidict>=4.0  multidict implementation

PS: This dependency tree is generated with johnnydep

[mrmuskrat]

Is there a workaround for users on an operating system with python3.11 preinstalled (like Fedora 37)?

[mrmuskrat]

Is there a workaround for users on an operating system with python3.11 preinstalled (like Fedora 37)?

I did find a workaround for myself this afternoon.

Steps:

1. Remove the azure-cli RPM: sudo dnf remove azure-cli
2. Install Python 3.10: sudo dnf install python3.10
3. Create a virtual environment for Python: python3.10 -m venv ~/.venv
4. Activate the virtual environment: . ~/.venv/bin/activate
5. Install the Azure CLI using pip: pip install azure-cli

[jiasli]

A CVE has been reported with Program Files (x86)\Microsoft SDKs\Azure\CLI2\python310.dll: #25246, but I can't see its validity yet. Anyway, let's try to use Python 3.11 if all blockers are now gone.

[jiasli]

PS: This dependency tree is generated with johnnydep

A more popular tool is pipdeptree: https://pypi.org/project/pipdeptree/ 😉

[peterbax117]

https://nvd.nist.gov/vuln/detail/CVE-2023-24329

With the new CVE-2023-24329, Program Files (x86)\Microsoft SDKs\Azure\CLI2\python310.dll now shows in security scans and will start to be automatically removed from environments.

[jiasli]

@peterbax117, it seems the Python community still has some debate on CVE-2023-24329: python/cpython#102153

[peterbax117]

@jiasli , Thank you for the information and link. I will follow-up on this with the customer and point them to the Python community. Appreciate the quick response.

[jiasli]

I have contacted Microsoft's internal Python community and got their reply:

Any risk here would be from people relying on urlparse to implement security measures like site blocking.

So the Python 3.10 itself is safe to use.

According to Python's official website, Python 3.10 is a fully supported version, including python310.dll. Therefore, we don't think Azure CLI is affected by CVE-2023-24329.

[luckman212]

Today I noticed azure-cli is the last remaining tool I have installed via Homebrew that depends on Python3.10. I do hope one day 3.11 is supported.

[roman-vynar]

This happens every time! Very annoying!

When the main python was 3.10, it depended from 3.9.
Now 3.11 vs 3.10.
Most likely it was the same way a long time ago as well but I can't confirm exactly.

Can we please stop hardcoding python version into this tool?
awscli has never had such issues and it's not different much.

Thank you!

[luckman212]

Python 3.12 is now the current version. We never even got 3.11 support! I hope this hardcoded dependency on 2+ year old versions of Python will be solved at some point.

[bebound]

@provenvelocity The PR is merged but we haven't released a new CLI version. 3.11 will be supported in next release.