Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global Arguments stop working in Python 3.9.8 #20269

Closed
jiasli opened this issue Nov 10, 2021 · 15 comments
Closed

Global Arguments stop working in Python 3.9.8 #20269

jiasli opened this issue Nov 10, 2021 · 15 comments
Assignees
Labels

Comments

@jiasli
Copy link
Member

jiasli commented Nov 10, 2021

Symptom

Global Arguments now stop working in Python 3.9.8:

> az cloud show --query profile
{
  "endpoints": {
    "activeDirectory": "https://login.microsoftonline.com",
    ...

Notice --query doesn't take effact.

Root cause

This is due to the argparse breaking change python/cpython#28420 which aims to solve https://bugs.python.org/issue45235. Python 3.9.8 is the first released version with this change. It will spread to all Python versions later.

In Knack, global arguments are registered on all root/intermediate/leaf parsers, and they share the same dest. Previously, leaf-level arguments takes precedence. After this change, root-level arguments take precedence.

We are not sure if this is intended or not, as according to Semantic Versioning, a PATCH version update shouldn't cause breaking change.

Since Azure CLI does some trimming on the command table, using --query on the root parser won't work. We have to use Knack directly to demonstrate it:

command 3.9.7 3.9.8
python .\knack\examples\exapp2 sample-json --query id
python .\knack\examples\exapp2 --query id sample-json

Proposed solution

Knack should only register global arguments on leaf parsers to avoid this issue. We don't allow using global arguments in the middle of a command anyway:

  • az cloud show --query profile
  • az --query profile cloud show
  • az cloud --query profile show

References

This issue is also affecting other CLI tools utilizing argparse:

A related topic for --output was extensively discussed before:

@ghost ghost added the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Nov 10, 2021
@jiasli jiasli pinned this issue Nov 10, 2021
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Nov 10, 2021
@yonzhan yonzhan added this to the Nov 2021 (2021-12-07) milestone Nov 10, 2021
@gvanrossum
Copy link

@rhettinger @ambv Can you look at this? It looks like a serious regression in 3.9.8.

@jiasli
Copy link
Member Author

jiasli commented Nov 12, 2021

Thank you so much @gvanrossum for the prompt response and fix! Really appreciate your help!

@jiasli
Copy link
Member Author

jiasli commented Nov 12, 2021

@jiasli
Copy link
Member Author

jiasli commented Nov 15, 2021

@ambv, can we get an ETA for the Python 3.9.9 release with the fix? We are seeing Python 3.9.8 deployed to Azure Pipelines and MacOS Homebrew (#20322), which is affecting more and more users. Really appreciate your help.

@jiasli
Copy link
Member Author

jiasli commented Nov 15, 2021

For MacOS users:

As it is not easy to install old/specific versions of Python (like 3.9.7) on MacOS with brew (Homebrew/discussions#155), we are working with brew team to release Azure CLI on Python 3.10 as soon as possible which is not affected (Homebrew/homebrew-core#86758).

For now, as a workaround, you may manually install Python 3.10 and install azure-cli with

brew install python@3.10
pip install azure-cli

@jiasli
Copy link
Member Author

jiasli commented Nov 15, 2021

When using --subscription argument, you will see error

  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python39\lib\argparse.py", line 1932, in take_action
    action(self, namespace, argument_values, option_string)
  File "d:\cli\azure-cli\src\azure-cli-core\azure\cli\core\commands\arm.py", line 357, in __call__
    profile = Profile(cli_ctx=namespace._cmd.cli_ctx)  # pylint: disable=protected-access
AttributeError: 'Namespace' object has no attribute '_cmd'

This is due to the behavior change of default value discussed in python/cpython#28420 (comment) - default value is no longer set before the Action logic fires in self._parse_known_args.

@ambv
Copy link

ambv commented Nov 15, 2021 via email

@Crayeth
Copy link

Crayeth commented Nov 15, 2021

~ % az --version
azure-cli                         2.30.0
core                              2.30.0

telemetry                          1.0.6

Python location '/usr/local/Cellar/azure-cli/2.30.0_1/libexec/bin/python'
Extensions directory '/Users/xxxxxx/.azure/cliextensions'
Python (Darwin) 3.10.0 (default, Oct 13 2021, 06:45:00) [Clang 13.0.0 (clang-1300.0.29.3)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

however

~ % az aks list -o table
Table output unavailable. Use the --query option to specify an appropriate query. Use --debug for more info.

still looks broken to me. This used to work. What am I doing wrong? I get the same result if I include a query like in the example:

~ % az aks list --query "[].{resource:resourceGroup, name:name}" -o table
Table output unavailable. Use the --query option to specify an appropriate query. Use --debug for more info.

@tuzi3040
Copy link

Table output unavailable. Use the --query option to specify an appropriate query. Use --debug for more info.

Looks similar to #20125. Though no Python version presented in that issue.

I came across the same problem when running az vm list.
This probably relates to python 3.10, cause this function normally on debian 10 with az 2.30.0 and python 3.6.10.

May be better to open another issue.

@Karrade7
Copy link

I'm releasing it today.

How should we test it?
Will it show up as a new version of az cli that can be upgraded via az upgrade?
Or is it merged to master and we should pull latest dev version?

Thanks!

@ambv
Copy link

ambv commented Nov 15, 2021

Sorry, I'm releasing Python 3.9.9 today, not azure-cli. The fixed Python 3.9.9 release is now available:

https://www.python.org/downloads/release/python-399/

@Karrade7
Copy link

thanks @ambv .
It looks like azure cli released a fix to hombrew as version 2.30.0_1 which now depends on python@3.10.
doing a brew update, brew upgrade fixed the issue for me.

Thanks for fixing as all my scripts were breaking before!

@yonzhan
Copy link
Collaborator

yonzhan commented Nov 15, 2021

Thanks @ambv, good to know you have released Python 3.9.9

@jiasli
Copy link
Member Author

jiasli commented Nov 16, 2021

@Crayeth, @tuzi3040, for the Table output unavailable. Use the --query option to specify an appropriate query. Use --debug for more info. issue , let's work on #20348.

@jiasli
Copy link
Member Author

jiasli commented Nov 16, 2021

Solution

If you directly use Python

As Python 3.9.9 has been released, please download and install the latest Python 3.9.9 from https://www.python.org/downloads/.

⚠ You may also use Python 3.10 to install Azure CLI, but be aware that --output table won't function correctly (#20348). It will be fixed in Azure CLI 2.31.0.

If you use Homebrew on MacOS

Homebrew/homebrew-core#86758 has been merged. Please update Azure CLI installation with

brew update && brew install azure-cli

⚠ As now the formula uses Python 3.10, --output table won't function correctly (#20348). It will be fixed in Azure CLI 2.31.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants