Setting install_scripts in distutils.cfg results in a broken virtualenv install.

[arlaneenalra]

• [ /] Minimal reproducible example or detailed descriptions:

Due to a need to have multiple, non-interfering python installs running locally on a machine. We have a custom distutils.cfg (as per https://docs.python.org/3.6/install/index.html?highlight=distutils%20cfg#distutils-configuration-files ) that has an install_scripts flag set in the mentioned system path.

Example distutils.cfg for macOS based system:

[install]
prefix=/usr/local
install_scripts=/usr/local/opt/python@3.6.8/bin

Creating a virtualenv using the above configuration and a python install containing that config at prefix/lib/pythonver/distutils/distutils.cfg results in a virutalenv directory with the structure:

├── lib
│   └── python3.6
│       └── site-packages ....
└── usr
    └── local
        └── opt
            └── python@3.6.8
                └── bin

Note that binaries are in usr/local/opt/python@3.6.8/bin instead of bin/

• [/ ] the output of the virtual environment creation with the -vvv --with-traceback flags included

 ~ % virtualenv --python=python3 -vvv --with-traceback venv
219 setup logging to NOTSET [DEBUG report:43]
236 find interpreter for spec PythonSpec(implementation=CPython, major=3) [INFO builtin:43]
236 proposed PythonInfo(spec=CPython3.6.8.final.0-64, exe=/usr/local/opt/python@3.6.8/bin/python3.6, platform=darwin, version='3.6.8 (default, Feb 13 2020, 09:47:29) \n[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:49]
236 accepted PythonInfo(spec=CPython3.6.8.final.0-64, exe=/usr/local/opt/python@3.6.8/bin/python3.6, platform=darwin, version='3.6.8 (default, Feb 13 2020, 09:47:29) \n[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:51]
240 filesystem is not case-sensitive [DEBUG info:28]
279 create virtual environment via CPython3Posix(dest=/Users/csalch/venv, clear=False, global=False) [INFO session:51]
279 create folder /Users/csalch/venv/lib/python3.6/site-packages [DEBUG _sync:20]
280 create folder /Users/csalch/venv/usr/local/opt/python@3.6.8/bin [DEBUG _sync:20]
280 write /Users/csalch/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
280 	home = /usr/local/Cellar/python@3.6.8/3.6.8_2/Frameworks/Python.framework/Versions/3.6 [DEBUG pyenv_cfg:38]
280 	implementation = CPython [DEBUG pyenv_cfg:38]
280 	version_info = 3.6.8.final.0 [DEBUG pyenv_cfg:38]
281 	virtualenv = 20.0.4 [DEBUG pyenv_cfg:38]
281 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
281 	base-prefix = /usr/local/Cellar/python@3.6.8/3.6.8_2/Frameworks/Python.framework/Versions/3.6 [DEBUG pyenv_cfg:38]
281 	base-exec-prefix = /usr/local/Cellar/python@3.6.8/3.6.8_2/Frameworks/Python.framework/Versions/3.6 [DEBUG pyenv_cfg:38]
281 	base-executable = /usr/local/opt/python@3.6.8/bin/python3.6 [DEBUG pyenv_cfg:38]
281 symlink /usr/local/opt/python@3.6.8/bin/python3.6 to /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/python [DEBUG _sync:39]
282 ============================== target debug ============================== [DEBUG session:53]
283 debug via /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/python /usr/local/lib/python3.6/site-packages/virtualenv/create/debug.py [DEBUG creator:191]
282 {
  "sys": {
    "executable": "/usr/local/opt/python@3.6.8/bin/python3.6",
    "_base_executable": null,
    "prefix": "/usr/local/opt/python@3.6.8/bin/../Frameworks/Python.framework/Versions/3.6",
    "base_prefix": "/usr/local/opt/python@3.6.8/bin/../Frameworks/Python.framework/Versions/3.6",
    "real_prefix": null,
    "exec_prefix": "/usr/local/opt/python@3.6.8/bin/../Frameworks/Python.framework/Versions/3.6",
    "base_exec_prefix": "/usr/local/opt/python@3.6.8/bin/../Frameworks/Python.framework/Versions/3.6",
    "path": [
      "/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python36.zip",
      "/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python3.6",
      "/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload",
      "/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages"
    ],
    "meta_path": [
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "3.6.8 (default, Feb 13 2020, 09:47:29) \n[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.17)]",
  "os": "<module 'os' from '/usr/local/opt/python@3.6.8/bin/../Frameworks/Python.framework/Versions/3.6/lib/python3.6/os.py'>",
  "site": "<module 'site' from '/usr/local/opt/python@3.6.8/bin/../Frameworks/Python.framework/Versions/3.6/lib/python3.6/site.py'>",
  "datetime": "<module 'datetime' from '/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python3.6/datetime.py'>",
  "math": "<module 'math' from '/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload/math.cpython-36m-darwin.so'>",
  "json": "<module 'json' from '/usr/local/opt/python@3.6.8/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py'>"
} [DEBUG session:54]
340 add seed packages via FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/csalch/Library/Application Support/virtualenv/seed-v1 via=copy [INFO session:58]
341 Attempting to acquire lock 4353297544 on /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels.lock [DEBUG filelock:270]
342 Lock 4353297544 acquired on /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels.lock [INFO filelock:274]
345 install pip from wheel /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
345 install setuptools from wheel /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels/setuptools-45.2.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
345 install wheel from wheel /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
346 copy /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/setuptools-45.2.0-py3-none-any/easy_install.py to /Users/csalch/venv/lib/python3.6/site-packages/easy_install.py [DEBUG _sync:47]
346 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/csalch/venv/lib/python3.6/site-packages/wheel [DEBUG _sync:47]
347 copy /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info.virtualenv to /Users/csalch/venv/lib/python3.6/site-packages/pip-20.0.2.dist-info.virtualenv [DEBUG _sync:47]
348 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /Users/csalch/venv/lib/python3.6/site-packages/pip [DEBUG _sync:47]
348 copy /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/setuptools-45.2.0-py3-none-any/setuptools-45.2.0.dist-info.virtualenv to /Users/csalch/venv/lib/python3.6/site-packages/setuptools-45.2.0.dist-info.virtualenv [DEBUG _sync:47]
349 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/setuptools-45.2.0-py3-none-any/setuptools to /Users/csalch/venv/lib/python3.6/site-packages/setuptools [DEBUG _sync:47]
363 copy /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info.virtualenv to /Users/csalch/venv/lib/python3.6/site-packages/wheel-0.34.2.dist-info.virtualenv [DEBUG _sync:47]
364 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/csalch/venv/lib/python3.6/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:47]
468 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/wheel-3.6 to 755 [INFO util:566]
469 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/wheel3 to 755 [INFO util:566]
470 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/wheel to 755 [INFO util:566]
470 generated console scripts wheel wheel-3.6 wheel3 [DEBUG base:48]
515 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/setuptools-45.2.0-py3-none-any/pkg_resources to /Users/csalch/venv/lib/python3.6/site-packages/pkg_resources [DEBUG _sync:47]
538 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/setuptools-45.2.0-py3-none-any/setuptools-45.2.0.dist-info to /Users/csalch/venv/lib/python3.6/site-packages/setuptools-45.2.0.dist-info [DEBUG _sync:47]
548 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/easy_install to 755 [INFO util:566]
549 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/easy_install3 to 755 [INFO util:566]
549 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/easy_install-3.6 to 755 [INFO util:566]
550 generated console scripts easy_install-3.6 easy_install easy_install3 [DEBUG base:48]
736 copy directory /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /Users/csalch/venv/lib/python3.6/site-packages/pip-20.0.2.dist-info [DEBUG _sync:47]
741 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/pip-3.6 to 755 [INFO util:566]
741 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/pip to 755 [INFO util:566]
741 changing mode of /Users/csalch/venv/usr/local/opt/python@3.6.8/bin/pip3 to 755 [INFO util:566]
742 generated console scripts pip3 pip-3.6 pip [DEBUG base:48]
742 Attempting to release lock 4353297544 on /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels.lock [DEBUG filelock:315]
742 Lock 4353297544 released on /Users/csalch/Library/Application Support/virtualenv/seed-v1/3.6/wheels.lock [INFO filelock:318]
742 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]
746 write /Users/csalch/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
747 	home = /usr/local/Cellar/python@3.6.8/3.6.8_2/Frameworks/Python.framework/Versions/3.6 [DEBUG pyenv_cfg:38]
747 	implementation = CPython [DEBUG pyenv_cfg:38]
747 	version_info = 3.6.8.final.0 [DEBUG pyenv_cfg:38]
747 	virtualenv = 20.0.4 [DEBUG pyenv_cfg:38]
747 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
747 	base-prefix = /usr/local/Cellar/python@3.6.8/3.6.8_2/Frameworks/Python.framework/Versions/3.6 [DEBUG pyenv_cfg:38]
747 	base-exec-prefix = /usr/local/Cellar/python@3.6.8/3.6.8_2/Frameworks/Python.framework/Versions/3.6 [DEBUG pyenv_cfg:38]
747 	base-executable = /usr/local/opt/python@3.6.8/bin/python3.6 [DEBUG pyenv_cfg:38]
747 created virtual environment in 530ms CPython3Posix(dest=/Users/csalch/venv, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/csalch/Library/Application Support/virtualenv/seed-v1 via=copy [WARNING __main__:24]

• [/ ] pip list of the environment where virtualenv is installed into if not using the zipapp

 ~ % source venv/bin/activate
source: no such file or directory: venv/bin/activate
 ~ % source venv/usr/local/opt/python@3.6.8/bin/activate
(venv) ~ % pip list
Traceback (most recent call last):
  File "/Users/csalch/venv/usr/local/opt/python@3.6.8/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip._internal.cli.main'
(venv) ~ % which python
/Users/csalch/venv/usr/local/opt/python@3.6.8/bin/python
(venv) ~ % python -m pip list
Package             Version
------------------- -------
anisble             0.1
ansible             2.9.4
appdirs             1.4.3
cffi                1.13.2
cryptography        2.8
distlib             0.3.0
filelock            3.0.12
importlib-metadata  1.5.0
importlib-resources 1.0.2
Jinja2              2.10.3
MarkupSafe          1.1.1
pip                 19.3.1
pycparser           2.19
PyYAML              5.3
ruamel.yaml         0.16.5
ruamel.yaml.clib    0.2.0
setuptools          42.0.2
six                 1.13.0
virtualenv          20.0.4
wheel               0.33.6
zipp                2.2.0
(venv) ~ % cat $(which pip)
#!/Users/csalch/venv/usr/local/opt/python@3.6.8/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
(venv) ~ %

[arlaneenalra]

I will note that this works as expected in 16.7.9

[andrewegel]

As from #1632 i'll re-paste here:

Reporting the same bug here on MacOS, only I'm highlighting the old vs new-unexpected behavior in a (hopefully) easy reproduction case:

./.test-venv/bin/pip list | grep virtualenv
virtualenv         16.7.9
./.test-venv/bin/python -m virtualenv .mac-os-venv
Using base prefix '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7'
New python executable in /Users/aegelhofer/.mac-os-venv/bin/python
Installing setuptools, pip, wheel...
done.
ls ~/.mac-os-venv/bin/*
/Users/aegelhofer/.mac-os-venv/bin/activate		/Users/aegelhofer/.mac-os-venv/bin/pip
/Users/aegelhofer/.mac-os-venv/bin/activate.csh		/Users/aegelhofer/.mac-os-venv/bin/pip3
/Users/aegelhofer/.mac-os-venv/bin/activate.fish	/Users/aegelhofer/.mac-os-venv/bin/pip3.7
/Users/aegelhofer/.mac-os-venv/bin/activate.ps1		/Users/aegelhofer/.mac-os-venv/bin/python
/Users/aegelhofer/.mac-os-venv/bin/activate.xsh		/Users/aegelhofer/.mac-os-venv/bin/python-config
/Users/aegelhofer/.mac-os-venv/bin/activate_this.py	/Users/aegelhofer/.mac-os-venv/bin/python3
/Users/aegelhofer/.mac-os-venv/bin/easy_install		/Users/aegelhofer/.mac-os-venv/bin/python3.7
/Users/aegelhofer/.mac-os-venv/bin/easy_install-3.7	/Users/aegelhofer/.mac-os-venv/bin/wheel
rm -rf .mac-os-venv
./.test-venv/bin/pip install -U virtualenv
Looking in indexes:
 ...
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (1.14.0)
Requirement already satisfied, skipping upgrade: importlib-metadata<2,>=0.12; python_version < "3.8" in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (1.5.0)
Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied, skipping upgrade: appdirs<2,>=1.4.3 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (1.4.3)
Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.0 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (0.3.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in ./.test-venv/lib/python3.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (2.2.0)
Installing collected packages: virtualenv
  Found existing installation: virtualenv 16.7.9
    Uninstalling virtualenv-16.7.9:
      Successfully uninstalled virtualenv-16.7.9
Successfully installed virtualenv-20.0.4
./.test-venv/bin/python -m virtualenv .mac-os-venv
created virtual environment in 169ms CPython3Posix(dest=/Users/aegelhofer/.mac-os-venv, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/aegelhofer/Library/Application Support/virtualenv/seed-v1 via=copy

Now at this point I expect the python and pip "binaries" to be under ~/.mac-os-venv/bin/*, but they now seem to be under ~/.mac-os-venv/<sourced-prefix>/bin/*:

ls ~/.mac-os-venv/bin/*
zsh: no matches found: /Users/aegelhofer/.mac-os-venv/bin/*
ls ~/.mac-os-venv/usr/local/bin/*
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate		/Users/aegelhofer/.mac-os-venv/usr/local/bin/pip
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.csh	/Users/aegelhofer/.mac-os-venv/usr/local/bin/pip-3.7
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.fish	/Users/aegelhofer/.mac-os-venv/usr/local/bin/pip3
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.ps1	/Users/aegelhofer/.mac-os-venv/usr/local/bin/python
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.xsh	/Users/aegelhofer/.mac-os-venv/usr/local/bin/python3
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate_this.py	/Users/aegelhofer/.mac-os-venv/usr/local/bin/python3.7
/Users/aegelhofer/.mac-os-venv/usr/local/bin/easy_install	/Users/aegelhofer/.mac-os-venv/usr/local/bin/wheel
/Users/aegelhofer/.mac-os-venv/usr/local/bin/easy_install-3.7	/Users/aegelhofer/.mac-os-venv/usr/local/bin/wheel-3.7
/Users/aegelhofer/.mac-os-venv/usr/local/bin/easy_install3	/Users/aegelhofer/.mac-os-venv/usr/local/bin/wheel3

This behavior change messes with tools like tox that expect to have the pip binary under <ENV-ROOT>/bin - Reading the changelog I didn't see anything mentioning this behavior change

If this is un-related, I can file a separate bug, but it seems suspect that we're getting hung up on the same issues relating to "prefixes" here. Attached is my debug output of virtualenv-20.0.4 command creating a virtualenv:

virtualenv-debug-output-issue-1632.txt

[gaborbernat]

With virtualenv 20 we have changed the way we've been creating virtual environments. Instead of using our own brittle in-house mode we've switched to the official upstream mechanism as described in https://www.python.org/dev/peps/pep-0405/ (known as the venv module). The behaviour you're described here is present with that module too, so the bug per se is not here, but upstream within CPython, so I'd recommend opening an issue under http://bugs.python.org/.

Now, as far as a workaround we'can provide on our side is maybe generate a pth file that patches the setuptools and distutils to ignore these global configuration flags (as we already do for python 2 - see https://github.com/pypa/virtualenv/blob/master/src/virtualenv/create/via_global_ref/builtin/python2/site.py). @andrewegel @arlaneenalra would that work for you?

cc @brettcannon for your awareness as the venv module maintainer (AFAIK)

[brettcannon]

Technically @vsajip is the maintainer of venv, but I have been contributing lately to it. 😄