curl -L https://aka.ms/InstallAzureCli | bash fails on macOS monterey 12.0.1 because of outdated virtualenv library

[breiter]

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug
curl -L https://aka.ms/InstallAzureCli | bash fails at the step of creating a virtual python environment in $HOME/lib/azure-cli

bin, lib, and include directories are created in $HOME/lib/azure-cli and various files are copied and symlinked. But then I get an error like the following that sys.prefix points to the original python3 binary location and not the new virtual environment directory in $HOME/lib/azure-cli.

ERROR: The executable /Users/breiter/lib/azure-cli/bin/python3 is not functioning
ERROR: It thinks sys.prefix is '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8' (should be '/Users/breiter/lib/azure-cli')

To Reproduce
Using built-in /usr/bin/python3 and XCode 13.1 comand-line tools:

$ curl -L https://aka.ms/InstallAzureCli | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100  1405  100  1405    0     0    174      0  0:00:08  0:00:08 --:--:--   295
Downloading Azure CLI install script from https://azurecliprod.blob.core.windows.net/install.py to /var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_ERgSJA.
######################################################################## 100.0%
/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_ERgSJA: OK
Running install script.
-- Verifying Python version.
-- Python version 3.8.9 okay.

===> In what directory would you like to place the install? (leave blank to use '/Users/breiter/lib/azure-cli'): 
-- '/Users/breiter/lib/azure-cli' is not empty and may contain a previous installation.

===> Remove this directory? (y/N): y
-- Deleted '/Users/breiter/lib/azure-cli'.
-- Creating directory '/Users/breiter/lib/azure-cli'.
-- We will install at '/Users/breiter/lib/azure-cli'.

===> In what directory would you like to place the 'az' executable? (leave blank to use '/Users/breiter/bin'): 
-- The executable will be in '/Users/breiter/bin'.
-- Downloading virtualenv package from https://pypi.python.org/packages/source/v/virtualenv/virtualenv-16.7.7.tar.gz.
-- Downloaded virtualenv package to /var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmp2ui5f98b/virtualenv-16.7.7.tar.gz.
-- Checksum of /var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmp2ui5f98b/virtualenv-16.7.7.tar.gz OK.
-- Extracting '/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmp2ui5f98b/virtualenv-16.7.7.tar.gz' to '/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmp2ui5f98b'.
-- Executing: ['/Applications/Xcode.app/Contents/Developer/usr/bin/python3', 'virtualenv.py', '--python', '/Applications/Xcode.app/Contents/Developer/usr/bin/python3', '/Users/breiter/lib/azure-cli']
Already using interpreter /Applications/Xcode.app/Contents/Developer/usr/bin/python3
Using base prefix '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8'
New python executable in /Users/breiter/lib/azure-cli/bin/python3
Also creating executable in /Users/breiter/lib/azure-cli/bin/python
ERROR: The executable /Users/breiter/lib/azure-cli/bin/python3 is not functioning
ERROR: It thinks sys.prefix is '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8' (should be '/Users/breiter/lib/azure-cli')
ERROR: virtualenv is not compatible with this system or executable
Traceback (most recent call last):
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_ERgSJA", line 415, in <module>
    main()
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_ERgSJA", line 399, in main
    create_virtualenv(tmp_dir, install_dir)
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_ERgSJA", line 144, in create_virtualenv
    exec_command(cmd, cwd=working_dir)
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_ERgSJA", line 105, in exec_command
    subprocess.check_call(command_list, cwd=cwd, env=env)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Applications/Xcode.app/Contents/Developer/usr/bin/python3', 'virtualenv.py', '--python', '/Applications/Xcode.app/Contents/Developer/usr/bin/python3', '/Users/breiter/lib/azure-cli']' returned non-zero exit status 100.

Using python39 installed with MacPorts.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  1405  100  1405    0     0    369      0  0:00:03  0:00:03 --:--:--   897
Downloading Azure CLI install script from https://azurecliprod.blob.core.windows.net/install.py to /var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_GhQktk.
######################################################################## 100.0%
/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_GhQktk: OK
Running install script.
-- Verifying Python version.
-- Python version 3.9.7 okay.

===> In what directory would you like to place the install? (leave blank to use '/Users/breiter/lib/azure-cli'): 
-- '/Users/breiter/lib/azure-cli' is not empty and may contain a previous installation.

===> Remove this directory? (y/N): y
-- Deleted '/Users/breiter/lib/azure-cli'.
-- Creating directory '/Users/breiter/lib/azure-cli'.
-- We will install at '/Users/breiter/lib/azure-cli'.

===> In what directory would you like to place the 'az' executable? (leave blank to use '/Users/breiter/bin'): 
-- The executable will be in '/Users/breiter/bin'.
-- Downloading virtualenv package from https://pypi.python.org/packages/source/v/virtualenv/virtualenv-16.7.7.tar.gz.
-- Downloaded virtualenv package to /var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmpszmiyjd7/virtualenv-16.7.7.tar.gz.
-- Checksum of /var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmpszmiyjd7/virtualenv-16.7.7.tar.gz OK.
-- Extracting '/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmpszmiyjd7/virtualenv-16.7.7.tar.gz' to '/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/tmpszmiyjd7'.
-- Executing: ['/opt/local/bin/python3', 'virtualenv.py', '--python', '/opt/local/bin/python3', '/Users/breiter/lib/azure-cli']
Already using interpreter /opt/local/bin/python3
Using base prefix '/opt/local/Library/Frameworks/Python.framework/Versions/3.9'
New python executable in /Users/breiter/lib/azure-cli/bin/python3
Also creating executable in /Users/breiter/lib/azure-cli/bin/python
ERROR: The executable /Users/breiter/lib/azure-cli/bin/python3 is not functioning
ERROR: It thinks sys.prefix is '/opt/local/Library/Frameworks/Python.framework/Versions/3.9' (should be '/Users/breiter/lib/azure-cli')
ERROR: virtualenv is not compatible with this system or executable
Traceback (most recent call last):
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_GhQktk", line 415, in <module>
    main()
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_GhQktk", line 399, in main
    create_virtualenv(tmp_dir, install_dir)
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_GhQktk", line 144, in create_virtualenv
    exec_command(cmd, cwd=working_dir)
  File "/var/folders/sl/11rbvmp17h18fm57j0n_c4_80000gq/T/azure_cli_install_tmp_GhQktk", line 105, in exec_command
    subprocess.check_call(command_list, cwd=cwd, env=env)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/local/bin/python3', 'virtualenv.py', '--python', '/opt/local/bin/python3', '/Users/breiter/lib/azure-cli']' returned non-zero exit status 100.

Expected behavior
The install script should run to completion and result in a functioning az binary.

Environment summary
macOS monterey 12.0.1 x86_64
GNU bash, version 5.1.8(1)-release (x86_64-apple-darwin21.1.0)
python 3.8.9 (apple) or python 3.9.7 (macports)

Additional context
I think the issue is that virtualenv 16.x is not compatible with macOS 12. I tried modifying install.py to use 16.7.12 and that also failed. I then installed py39-virtualenv (which is 20.9.0) from macports and modified install.py to use virtualenv from the path rather than downloading it from pypi.

$ diff -u install.py.ori install.py
--- install.py.ori	2021-10-26 15:30:52.000000000 +0200
+++ install.py	2021-10-26 15:32:29.000000000 +0200
@@ -125,23 +125,24 @@
 
 
 def create_virtualenv(tmp_dir, install_dir):
-    download_location = os.path.join(tmp_dir, VIRTUALENV_ARCHIVE)
-    print_status('Downloading virtualenv package from {}.'.format(VIRTUALENV_DOWNLOAD_URL))
-    response = urlopen(VIRTUALENV_DOWNLOAD_URL)
-    with open(download_location, 'wb') as f: f.write(response.read())
-    print_status("Downloaded virtualenv package to {}.".format(download_location))
-    if is_valid_sha256sum(download_location, VIRTUALENV_ARCHIVE_SHA256):
-        print_status("Checksum of {} OK.".format(download_location))
-    else:
-        raise CLIInstallError("The checksum of the downloaded virtualenv package does not match.")
-    print_status("Extracting '{}' to '{}'.".format(download_location, tmp_dir))
-    package_tar = tarfile.open(download_location)
-    package_tar.extractall(path=tmp_dir)
-    package_tar.close()
-    virtualenv_dir_name = 'virtualenv-'+VIRTUALENV_VERSION
-    working_dir = os.path.join(tmp_dir, virtualenv_dir_name)
-    cmd = [sys.executable, 'virtualenv.py', '--python', sys.executable, install_dir]
-    exec_command(cmd, cwd=working_dir)
+    #download_location = os.path.join(tmp_dir, VIRTUALENV_ARCHIVE)
+    #print_status('Downloading virtualenv package from {}.'.format(VIRTUALENV_DOWNLOAD_URL))
+    #response = urlopen(VIRTUALENV_DOWNLOAD_URL)
+    #with open(download_location, 'wb') as f: f.write(response.read())
+    #print_status("Downloaded virtualenv package to {}.".format(download_location))
+    #if is_valid_sha256sum(download_location, VIRTUALENV_ARCHIVE_SHA256):
+    #    print_status("Checksum of {} OK.".format(download_location))
+    #else:
+    #    raise CLIInstallError("The checksum of the downloaded virtualenv package does not match.")
+    #print_status("Extracting '{}' to '{}'.".format(download_location, tmp_dir))
+    #package_tar = tarfile.open(download_location)
+    #package_tar.extractall(path=tmp_dir)
+    #package_tar.close()
+    #virtualenv_dir_name = 'virtualenv-'+VIRTUALENV_VERSION
+    #working_dir = os.path.join(tmp_dir, virtualenv_dir_name)
+    cmd = ['virtualenv', '--python', sys.executable, install_dir]
+    #exec_command(cmd, cwd=working_dir)
+    exec_command(cmd)
 
 
 def install_cli(install_dir, tmp_dir):

This worked and ran to completion.

$ az --version
azure-cli                         2.29.1

core                              2.29.1
telemetry                          1.0.6

Python location '/Users/breiter/lib/azure-cli/bin/python'
Extensions directory '/Users/breiter/.azure/cliextensions'

Python (Darwin) 3.9.7 (default, Oct 26 2021, 00:27:12) 
[Clang 12.0.0 (clang-1200.0.32.28)]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

Please let us know how we are doing: https://aka.ms/azureclihats
and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy

[yonzhan]

@jiasli for awareness

[AdamSLevy]

This still persists. Would be nice to not have to fight the install script.

[bebound]

The recommended way to install CLI on MacOS is using Homebrew.