Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripts and bin in bad location #1663

Closed
gaborbernat opened this issue Feb 25, 2020 · 35 comments
Closed

Scripts and bin in bad location #1663

gaborbernat opened this issue Feb 25, 2020 · 35 comments

Comments

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 25, 2020

Reporting the same bug here on MacOS, only I'm highlighting the old vs new-unexpected behavior in a (hopefully) easy reproduction case:

./.test-venv/bin/pip list | grep virtualenv
virtualenv         16.7.9
./.test-venv/bin/python -m virtualenv .mac-os-venv
Using base prefix '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7'
New python executable in /Users/aegelhofer/.mac-os-venv/bin/python
Installing setuptools, pip, wheel...
done.
ls ~/.mac-os-venv/bin/*
/Users/aegelhofer/.mac-os-venv/bin/activate		/Users/aegelhofer/.mac-os-venv/bin/pip
/Users/aegelhofer/.mac-os-venv/bin/activate.csh		/Users/aegelhofer/.mac-os-venv/bin/pip3
/Users/aegelhofer/.mac-os-venv/bin/activate.fish	/Users/aegelhofer/.mac-os-venv/bin/pip3.7
/Users/aegelhofer/.mac-os-venv/bin/activate.ps1		/Users/aegelhofer/.mac-os-venv/bin/python
/Users/aegelhofer/.mac-os-venv/bin/activate.xsh		/Users/aegelhofer/.mac-os-venv/bin/python-config
/Users/aegelhofer/.mac-os-venv/bin/activate_this.py	/Users/aegelhofer/.mac-os-venv/bin/python3
/Users/aegelhofer/.mac-os-venv/bin/easy_install		/Users/aegelhofer/.mac-os-venv/bin/python3.7
/Users/aegelhofer/.mac-os-venv/bin/easy_install-3.7	/Users/aegelhofer/.mac-os-venv/bin/wheel
rm -rf .mac-os-venv
./.test-venv/bin/pip install -U virtualenv
Looking in indexes:
 ...
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (1.14.0)
Requirement already satisfied, skipping upgrade: importlib-metadata<2,>=0.12; python_version < "3.8" in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (1.5.0)
Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied, skipping upgrade: appdirs<2,>=1.4.3 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (1.4.3)
Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.0 in ./.test-venv/lib/python3.7/site-packages (from virtualenv) (0.3.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in ./.test-venv/lib/python3.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (2.2.0)
Installing collected packages: virtualenv
  Found existing installation: virtualenv 16.7.9
    Uninstalling virtualenv-16.7.9:
      Successfully uninstalled virtualenv-16.7.9
Successfully installed virtualenv-20.0.4
./.test-venv/bin/python -m virtualenv .mac-os-venv
created virtual environment in 169ms CPython3Posix(dest=/Users/aegelhofer/.mac-os-venv, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/aegelhofer/Library/Application Support/virtualenv/seed-v1 via=copy

Now at this point I expect the python and pip "binaries" to be under ~/.mac-os-venv/bin/*, but they now seem to be under ~/.mac-os-venv/<sourced-prefix>/bin/*:

ls ~/.mac-os-venv/bin/*
zsh: no matches found: /Users/aegelhofer/.mac-os-venv/bin/*
ls ~/.mac-os-venv/usr/local/bin/*
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate		/Users/aegelhofer/.mac-os-venv/usr/local/bin/pip
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.csh	/Users/aegelhofer/.mac-os-venv/usr/local/bin/pip-3.7
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.fish	/Users/aegelhofer/.mac-os-venv/usr/local/bin/pip3
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.ps1	/Users/aegelhofer/.mac-os-venv/usr/local/bin/python
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate.xsh	/Users/aegelhofer/.mac-os-venv/usr/local/bin/python3
/Users/aegelhofer/.mac-os-venv/usr/local/bin/activate_this.py	/Users/aegelhofer/.mac-os-venv/usr/local/bin/python3.7
/Users/aegelhofer/.mac-os-venv/usr/local/bin/easy_install	/Users/aegelhofer/.mac-os-venv/usr/local/bin/wheel
/Users/aegelhofer/.mac-os-venv/usr/local/bin/easy_install-3.7	/Users/aegelhofer/.mac-os-venv/usr/local/bin/wheel-3.7
/Users/aegelhofer/.mac-os-venv/usr/local/bin/easy_install3	/Users/aegelhofer/.mac-os-venv/usr/local/bin/wheel3

This behavior change messes with tools like tox that expect to have the pip binary under <ENV-ROOT>/bin - Reading the changelog I didn't see anything mentioning this behavior change

If this is un-related, I can file a separate bug, but it seems suspect that we're getting hung up on the same issues relating to "prefixes" here. Attached is my debug output of virtualenv-20.0.4 command creating a virtualenv:

virtualenv-debug-output-issue-1632.txt

Originally posted by @andrewegel in #1632 (comment)

PS. moved here as it's a new issue.

@gaborbernat gaborbernat changed the title Prefixes in bad location Scripts and bin in bad location Feb 25, 2020
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Feb 25, 2020

@andrewegel can you validate if this is still an issue with virtualenv 20.0.5?

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Feb 25, 2020

@arlaneenalra can you validate this no longer happens with master

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Feb 25, 2020

I don't think this should be still the case, but if so we'll reopen.

@arlaneenalra

This comment has been minimized.

Copy link

@arlaneenalra arlaneenalra commented Feb 25, 2020

This is definitely still happening:

% pip3 install -U virtualenv
Looking in indexes: https://<sanitized>.com/repository/pypi/simple
Collecting virtualenv
  Downloading https://<sanitized>.com/repository/pypi/files.pythonhosted.org/https/packages/33/4e/9f3e70ed2bf3de603bf4de46cb60af1309c8d97fd81cbde317507a57573e/virtualenv-20.0.5-py2.py3-none-any.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 2.0 MB/s
Requirement already satisfied, skipping upgrade: appdirs<2,>=1.4.3 in /usr/local/lib/python3.6/site-packages (from virtualenv) (1.4.3)
Requirement already satisfied, skipping upgrade: importlib-metadata<2,>=0.12; python_version < "3.8" in /usr/local/lib/python3.6/site-packages (from virtualenv) (1.5.0)
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in /usr/local/lib/python3.6/site-packages (from virtualenv) (1.13.0)
Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in /usr/local/lib/python3.6/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied, skipping upgrade: importlib-resources<2,>=1.0; python_version < "3.7" in /usr/local/lib/python3.6/site-packages (from virtualenv) (1.0.2)
Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.0 in /usr/local/lib/python3.6/site-packages (from virtualenv) (0.3.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (2.2.0)
Installing collected packages: virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 16.7.9
    Uninstalling virtualenv-16.7.9:
      Successfully uninstalled virtualenv-16.7.9
Successfully installed virtualenv-20.0.5
 % virtualenv --python python3 venv
created virtual environment CPython3.6.8.final.0-64 in 556ms
  creator CPython3Posix(dest=/Users/csalch/test/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/csalch/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
 % cd venv
% ls
lib  pyvenv.cfg  usr
 % ls -l
total 4
drwxr-xr-x 3 csalch admin  96 Feb 25 10:57 lib
-rw-r--r-- 1 csalch admin 451 Feb 25 10:57 pyvenv.cfg
drwxr-xr-x 3 csalch admin  96 Feb 25 10:57 usr
 % tree
.
├── lib
│   └── python3.6
│       └── site-packages
... clip ...
├── pyvenv.cfg
└── usr
    └── local
        └── opt
            └── python@3.6.8
                └── bin
                    ├── activate
                    ├── activate.csh
                    ├── activate.fish
                    ├── activate.ps1
                    ├── activate.xsh
                    ├── activate_this.py
                    ├── easy_install
                    ├── easy_install-3.6
                    ├── easy_install3
                    ├── pip
                    ├── pip-3.6
                    ├── pip3
                    ├── python -> /usr/local/opt/python@3.6.8/bin/python3.6
                    ├── python3 -> python
                    ├── python3.6 -> python
                    ├── wheel
                    ├── wheel-3.6
                    └── wheel3

67 directories, 507 files
% virtualenv --version
virtualenv 20.0.5 from /usr/local/lib/python3.6/site-packages/virtualenv/__init__.py
%
@andrewegel

This comment has been minimized.

Copy link

@andrewegel andrewegel commented Feb 25, 2020

@gaborbernat This is not fixed with 20.0.5:

./.test-venv/bin/pip list | grep vir
virtualenv         20.0.5
./.test-venv/bin/python -m virtualenv .mac-os-venv
created virtual environment CPython3.7.3.final.0-64 in 178ms
  creator CPython3Posix(dest=/Users/aegelhofer/.mac-os-venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/aegelhofer/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
ls ~/.mac-os-venv/
Library		lib		pyvenv.cfg	usr
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Feb 25, 2020

What about master?

@arlaneenalra

This comment has been minimized.

Copy link

@arlaneenalra arlaneenalra commented Feb 26, 2020

No joy in 20.0.7 either.

@gaborbernat gaborbernat reopened this Feb 26, 2020
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Feb 26, 2020

Can you test 20.0.6 we had to revart the fix due to pypa/pip#7778

@arlaneenalra

This comment has been minimized.

Copy link

@arlaneenalra arlaneenalra commented Feb 26, 2020

No joy with 20.0.6 as well. We have a distutils.cfg file present that looks like:

[install]
prefix=/usr/local
install_scripts=/usr/local/opt/python@3.6.8/bin

[build_ext]
include_dirs=/usr/local/include:/usr/local/opt/openssl@1.1/include:/usr/local/opt/sqlite/include
library_dirs=/usr/local/lib:/usr/local/opt/openssl@1.1/lib:/usr/local/opt/sqlite/lib

It lives in <python install root>/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/distutils.cfg

It look like that config is leaking into virtualenv somehow ...

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Feb 27, 2020

Will give a try with that config and report back next week 👍

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 4, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.8; see https://pypi.org/project/virtualenv/20.0.8/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-8-2020-03-04). Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.
thanks

@andrewegel

This comment has been minimized.

Copy link

@andrewegel andrewegel commented Mar 4, 2020

This is still not fixed in 20.0.8

% ./.test-venv/bin/pip list| grep virt
virtualenv         20.0.8
% ./.test-venv/bin/python -m virtualenv .mac-os-venv
created virtual environment CPython3.7.3.final.0-64 in 182ms
  creator CPython3Posix(dest=/Users/aegelhofer/.mac-os-venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
% ls .mac-os-venv
Library		lib		pyvenv.cfg	usr
@gaborbernat gaborbernat reopened this Mar 4, 2020
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 4, 2020

Can you provide detailed instructions to reproduce? Ideally a docker image, otherwise Python used, creation log, tree of the created virtual environment, etc 👍 also what configuration files you've used for distutils

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 9, 2020

Closing until someone can reproduce this. I fail to do so on the environments accessible to me, so it's unactionable at the moment. If someone can create a reproducible example we'll reopen.

@gaborbernat gaborbernat closed this Mar 9, 2020
@andrewegel

This comment has been minimized.

Copy link

@andrewegel andrewegel commented Mar 9, 2020

Please re-open.

Ideally a docker image

AFAIK MacOS doesn't have a "docker image"

Can you provide detailed instructions to reproduce?

So, MacOS Catalina 10.15.3 came with a stock /usr/bin/python3 (version 3.7.3) - It also came with what looks like a "light weight" virtual env:

aegelhofer@Andrew-Egelhofer's- docs % /usr/bin/python3 -m venv --help
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
            [--upgrade] [--without-pip] [--prompt PROMPT]
            ENV_DIR [ENV_DIR ...]

Creates virtual Python environments in one or more target directories.

positional arguments:
  ENV_DIR               A directory to create the environment in.

optional arguments:
  -h, --help            show this help message and exit
  --system-site-packages
                        Give the virtual environment access to the system
                        site-packages dir.
  --symlinks            Try to use symlinks rather than copies, when symlinks
                        are not the default for the platform.
  --copies              Try to use copies rather than symlinks, even when
                        symlinks are the default for the platform.
  --clear               Delete the contents of the environment directory if it
                        already exists, before environment creation.
  --upgrade             Upgrade the environment directory to use this version
                        of Python, assuming Python has been upgraded in-place.
  --without-pip         Skips installing or upgrading pip in the virtual
                        environment (pip is bootstrapped by default)
  --prompt PROMPT       Provides an alternative prompt prefix for this
                        environment.

Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.

This virtualenv doesn't have this issue. MacOS also will no longer allow edits under /usr/lib (or /Library/), so I have to generate a "user" pip install:

aegelhofer@Andrew-Egelhofer's- ~ % /usr/bin/pip3 install --user virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/c8/96/0d929b45d4c0cfbb81fecb41c4ea5b48ea8834b9184e1fef4a7871b01eb1/virtualenv-20.0.9-py2.py3-none-any.whl (4.6MB)
    100% |████████████████████████████████| 4.6MB 7.4MB/s
Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting distlib<1,>=0.3.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/7d/29/694a3a4d7c0e1aef76092e9167fbe372e0f7da055f5dcf4e1313ec21d96a/distlib-0.3.0.zip (571kB)
    100% |████████████████████████████████| 573kB 11.1MB/s
Collecting appdirs<2,>=1.4.3 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Requirement already satisfied: six<2,>=1.9.0 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages (from virtualenv) (1.12.0)
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/b2/34/bfcb43cc0ba81f527bc4f40ef41ba2ff4080e047acb0586b56b3d017ace4/zipp-3.1.0-py3-none-any.whl
Building wheels for collected packages: distlib
  Building wheel for distlib (setup.py) ... done
  Stored in directory: /Users/aegelhofer/Library/Caches/pip/wheels/6e/e8/db/c73dae4867666e89ba3cfbc4b5c092446f0e584eda6f409cbb
Successfully built distlib
Installing collected packages: zipp, importlib-metadata, filelock, distlib, appdirs, virtualenv
  The script virtualenv is installed in '/Users/aegelhofer/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 virtualenv-20.0.9 zipp-3.1.0
You are using pip version 19.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
aegelhofer@Andrew-Egelhofer's- ~ % /usr/bin/python3 -m virtualenv --version
virtualenv 20.0.9 from /Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/virtualenv/__init__.py

Now I create it:

aegelhofer@Andrew-Egelhofer's- ~ % /usr/bin/python3 -m virtualenv -vvvvvvvvv ~/.mac-os-venv

Debug Log is here: virtualenv-macos-20.0.9-debug-output.txt

find output of ~/.mac-os-venv is here:

virtualenv-macos-20.0.9-find-output.txt

also what configuration files you've used for distutils

This is stock configuration from Apple, so AFAIK, I'm not doing anything different, so I'm attaching what I think is what you're looking for:

virtualenv-macos-20.0.9-distlib-sysconfig.txt

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 9, 2020

@andrewegel where does virtualenv-macos-20.0.9-distlib-sysconfig.txt come from? Path?

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 9, 2020

I have Catalina and don't have that python install. How does one install on Catalina to be able to reproduce?

@andrewegel

This comment has been minimized.

Copy link

@andrewegel andrewegel commented Mar 9, 2020

@andrewegel where does virtualenv-macos-20.0.9-distlib-sysconfig.txt come from? Path?

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pip/_vendor/distlib/_backport/sysconfig.cfg

I have Catalina and don't have that python install. How does one install on Catalina to be able to reproduce?

This came pre-installed with the laptop that was provisioned to me by my employer, so I don't have a definitive answer, I think it was one of the Command Line Tools you can obtain from Apple:

https://developer.apple.com/download/more/

I've tried downloading one, installing and testing, but that didn't seem to put anything at /use/bin/python3 or anything - so I guess we have to chock this up to w/e environment I'm running, and is therefore now a wide-spread bug.

If I find time to dig deep into this issue, I'll send a PR with a fix, but no promises.

@arlaneenalra

This comment has been minimized.

Copy link

@arlaneenalra arlaneenalra commented Mar 9, 2020

Here's a small repo that reproduces this issue in a linux based docker container:

https://github.com/arlaneenalra/broken-virtualenv-example

With short README containing instructions on how to duplicate the issue.

This is definitely still a problem with 20.0.9

@gaborbernat gaborbernat reopened this Mar 10, 2020
gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves pypa#1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves pypa#1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 10, 2020

@arlaneenalra @andrewegel thanks for the information provided, I believe this will now be finally fixed by #1705; please check and report back if not.

gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves pypa#1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves pypa#1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves pypa#1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves pypa#1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit that referenced this issue Mar 10, 2020
Ignore configuration file parsing when getting distutils data.

Resolves #1663.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 10, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.10; see https://pypi.org/project/virtualenv/20.0.10/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-10-2020-03-10). Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.
thanks

@arlaneenalra

This comment has been minimized.

Copy link

@arlaneenalra arlaneenalra commented Mar 10, 2020

@gaborbernat This appears to pass in the Docker example and when I use the custom brew based python setup I mentioned in #1663 (comment). So at least my issue appears to be fixed.

It's possible there are/were other ways to trigger the same behavior, so I'd like to see @andrewegel comment prior to this being called definitely fixed.

@andrewegel

This comment has been minimized.

Copy link

@andrewegel andrewegel commented Mar 10, 2020

I just tried version 20.0.10, and I'm still seeing the same behavior - Without a valid reproducible case, I can't justify leaving this open.

@gaborbernat gaborbernat reopened this Mar 10, 2020
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 10, 2020

@andrewegel please post the -vvv with 20.0.10; and also the json file we lock/write while doing the environment build.

@andrewegel

This comment has been minimized.

Copy link

@andrewegel andrewegel commented Mar 10, 2020

% ./Library/Python/3.7/bin/virtualenv --version
virtualenv 20.0.10 from /Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/virtualenv/__init__.py
% ./Library/Python/3.7/bin/virtualenv -vvvvvvv ~/.macos-env
164 setup logging to NOTSET [DEBUG report:43]
169 find interpreter for spec PythonSpec(path=/Library/Developer/CommandLineTools/usr/bin/python3) [INFO builtin:44]
169 proposed PythonInfo(spec=CPython3.7.3.final.0-64, exe=/Library/Developer/CommandLineTools/usr/bin/python3, platform=darwin, version='3.7.3 (default, Feb 26 2020, 15:24:45) \n[Clang 11.0.3 (clang-1103.0.32.26)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
169 accepted PythonInfo(spec=CPython3.7.3.final.0-64, exe=/Library/Developer/CommandLineTools/usr/bin/python3, platform=darwin, version='3.7.3 (default, Feb 26 2020, 15:24:45) \n[Clang 11.0.3 (clang-1103.0.32.26)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:52]
174 filesystem is not case-sensitive [DEBUG info:28]
316 create virtual environment via CPython3Posix(dest=/Users/aegelhofer/.macos-env, clear=False, global=False) [INFO session:52]
316 create folder /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages [DEBUG _sync:21]
316 create folder /Users/aegelhofer/.macos-env/lib/python3.7 [DEBUG _sync:21]
317 create folder /Users/aegelhofer/.macos-env/usr/local/bin [DEBUG _sync:21]
317 write /Users/aegelhofer/.macos-env/pyvenv.cfg [DEBUG pyenv_cfg:34]
317 	home = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
317 	implementation = CPython [DEBUG pyenv_cfg:38]
317 	version_info = 3.7.3.final.0 [DEBUG pyenv_cfg:38]
317 	virtualenv = 20.0.10 [DEBUG pyenv_cfg:38]
317 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
317 	base-prefix = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
317 	base-exec-prefix = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
317 	base-executable = /Library/Developer/CommandLineTools/usr/bin/python3 [DEBUG pyenv_cfg:38]
317 symlink /Library/Developer/CommandLineTools/usr/bin/python3 to /Users/aegelhofer/.macos-env/usr/local/bin/python [DEBUG _sync:40]
318 create /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/_virtualenv.py [DEBUG api:60]
319 create virtualenv import hook file /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/_virtualenv.pth [DEBUG api:63]
319 ============================== target debug ============================== [DEBUG session:54]
319 debug via /Users/aegelhofer/.macos-env/usr/local/bin/python /Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/virtualenv/create/debug.py [DEBUG creator:182]
319 {
  "sys": {
    "executable": "/Users/aegelhofer/.macos-env/usr/local/bin/python",
    "_base_executable": "/Users/aegelhofer/.macos-env/usr/local/bin/python",
    "prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "base_prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "real_prefix": null,
    "exec_prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "base_exec_prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "path": [
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python37.zip",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload",
      "/Users/aegelhofer/Library/Python/3.7/lib/python/site-packages",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages"
    ],
    "meta_path": [
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "3.7.3 (default, Feb 26 2020, 15:24:45) \n[Clang 11.0.3 (clang-1103.0.32.26)]",
  "os": "<module 'os' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/os.py'>",
  "site": "<module 'site' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site.py'>",
  "datetime": "<module 'datetime' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/datetime.py'>",
  "math": "<module 'math' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload/math.cpython-37m-darwin.so'>",
  "json": "<module 'json' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/json/__init__.py'>"
} [DEBUG session:55]
363 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1) [INFO session:59]
363 Attempting to acquire lock 4452991496 on /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels.lock [DEBUG filelock:270]
363 Lock 4452991496 acquired on /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels.lock [INFO filelock:274]
364 get bundled wheel /Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/virtualenv/seed/embed/wheels/pip-20.0.2-py2.py3-none-any.whl [DEBUG acquire:51]
364 get bundled wheel /Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/virtualenv/seed/embed/wheels/setuptools-46.0.0-py3-none-any.whl [DEBUG acquire:51]
365 get bundled wheel /Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/virtualenv/seed/embed/wheels/wheel-0.34.2-py2.py3-none-any.whl [DEBUG acquire:51]
368 install wheel from wheel /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
368 install setuptools from wheel /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels/setuptools-46.0.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
368 install pip from wheel /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
369 copy /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.virtualenv to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/pip-20.0.2.virtualenv [DEBUG _sync:48]
369 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/setuptools-46.0.0-py3-none-any/setuptools-46.0.0.dist-info to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/setuptools-46.0.0.dist-info [DEBUG _sync:48]
370 copy /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:48]
370 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/pip [DEBUG _sync:48]
371 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/wheel [DEBUG _sync:48]
376 copy /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/setuptools-46.0.0-py3-none-any/setuptools-46.0.0.virtualenv to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/setuptools-46.0.0.virtualenv [DEBUG _sync:48]
377 copy /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/setuptools-46.0.0-py3-none-any/easy_install.py to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/easy_install.py [DEBUG _sync:48]
378 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/setuptools-46.0.0-py3-none-any/setuptools to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/setuptools [DEBUG _sync:48]
385 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:48]
391 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/wheel3 to 755 [INFO util:566]
391 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/wheel-3.7 to 755 [INFO util:566]
392 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/wheel to 755 [INFO util:566]
392 generated console scripts wheel3 wheel wheel-3.7 [DEBUG base:53]
419 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/setuptools-46.0.0-py3-none-any/pkg_resources to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/pkg_resources [DEBUG _sync:48]
429 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/easy_install to 755 [INFO util:566]
430 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/easy_install3 to 755 [INFO util:566]
430 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/easy_install-3.7 to 755 [INFO util:566]
430 generated console scripts easy_install3 easy_install-3.7 easy_install [DEBUG base:53]
500 copy directory /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /Users/aegelhofer/.macos-env/Library/Python/3.7/site-packages/pip-20.0.2.dist-info [DEBUG _sync:48]
503 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/pip3 to 755 [INFO util:566]
503 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/pip to 755 [INFO util:566]
504 changing mode of /Users/aegelhofer/.macos-env/usr/local/bin/pip-3.7 to 755 [INFO util:566]
504 generated console scripts pip pip3 pip-3.7 [DEBUG base:53]
504 Attempting to release lock 4452991496 on /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels.lock [DEBUG filelock:315]
504 Lock 4452991496 released on /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels.lock [INFO filelock:318]
504 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:65]
507 write /Users/aegelhofer/.macos-env/pyvenv.cfg [DEBUG pyenv_cfg:34]
507 	home = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
508 	implementation = CPython [DEBUG pyenv_cfg:38]
508 	version_info = 3.7.3.final.0 [DEBUG pyenv_cfg:38]
508 	virtualenv = 20.0.10 [DEBUG pyenv_cfg:38]
508 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
508 	base-prefix = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
508 	base-exec-prefix = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
508 	base-executable = /Library/Developer/CommandLineTools/usr/bin/python3 [DEBUG pyenv_cfg:38]
508 created virtual environment CPython3.7.3.final.0-64 in 345ms
  creator CPython3Posix(dest=/Users/aegelhofer/.macos-env, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:21]

The lock file is empty:

% ls -la /Users/aegelhofer/Library/Application\ Support/virtualenv/*/*/3.7/wheels.lock
-rwxr-xr-x  1   0 Mar 10 14:02 /Users/aegelhofer/Library/Application Support/virtualenv/seed-app-data/v1/3.7/wheels.lock
% cat /Users/aegelhofer/Library/Application\ Support/virtualenv/*/*/3.7/wheels.lock
@arlaneenalra

This comment has been minimized.

Copy link

@arlaneenalra arlaneenalra commented Mar 10, 2020

@andrewegel You might want to check for a file like:

/Users/aegelhofer/Library/Python/3.7/lib/python/site-packages/distutils/distutils.cfg   /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/distutils/distutils.cfg
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/distutils/distutils.cfg

or some variant of those .. There are other config fields in distutils.cfg that could potentially have similar impact https://docs.python.org/3.7/install/index.html?highlight=distutils%20cfg#distutils-configuration-files

Could also be something else, but knowing if a similar file exists and it's contents could be helpful.

In theory, a similar file could be injected in to the docker container just like I did for my test case.

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 10, 2020

@andrewegel can you post the output of this script https://github.com/pypa/virtualenv/blob/master/src/virtualenv/discovery/py_info.py against /Library/Developer/CommandLineTools/usr/bin/python3? It should be a JSON.

@biochimia

This comment has been minimized.

Copy link

@biochimia biochimia commented Mar 11, 2020

I'm seeing a similar issue using Xcode's python3. Here's the information I gathered from my system.

Python3, provided by Xcode Command Line Tools, macOS Catalina (10.15.2)

$ /usr/bin/python3 --version
Python 3.7.3
$ /usr/bin/python3
Python 3.7.3 (default, Dec 13 2019, 19:58:14)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Virtualenv 20.0.10

$ /usr/bin/python3 virtualenv.pyz --version
virtualenv 20.0.10 from virtualenv.pyz/virtualenv/__init__.py

New virtual environment

$ /usr/bin/python3 virtualenv.pyz venv-test
created virtual environment CPython3.7.3.final.0-64 in 229ms
  creator CPython3Posix(dest=/Users/joao/temp/venv-test, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/joao/Library/Application Support/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Directory structure for generated virtual environment

$ tree -d -L 4 venv-test
venv-test
├── Library
│   └── Python
│       └── 3.7
│           └── site-packages
├── lib
│   └── python3.7
└── usr
    └── local
        └── bin

9 directories

Directory structure (?) breaks virtual environment

$ venv-test/usr/local/bin/pip
Traceback (most recent call last):
  File "venv-test/usr/local/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip._internal.cli.main'

Python information from py_info.py script

$ /usr/bin/python3 py_info.py
{
  "platform": "darwin",
  "implementation": "CPython",
  "version_info": {
    "major": 3,
    "minor": 7,
    "micro": 3,
    "releaselevel": "final",
    "serial": 0
  },
  "architecture": 64,
  "version": "3.7.3 (default, Dec 13 2019, 19:58:14) \n[Clang 11.0.0 (clang-1100.0.33.17)]",
  "os": "posix",
  "prefix": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7",
  "base_prefix": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7",
  "real_prefix": null,
  "base_exec_prefix": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7",
  "exec_prefix": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7",
  "executable": "/Applications/Xcode.app/Contents/Developer/usr/bin/python3",
  "original_executable": "/Applications/Xcode.app/Contents/Developer/usr/bin/python3",
  "system_executable": "/Applications/Xcode.app/Contents/Developer/usr/bin/python3",
  "has_venv": true,
  "path": [
    "/Users/joao/ade/cli/run-python",
    "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python37.zip",
    "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7",
    "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload",
    "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages"
  ],
  "file_system_encoding": "utf-8",
  "stdout_encoding": "UTF-8",
  "sysconfig_paths": {
    "stdlib": "{installed_base}/lib/python{py_version_short}",
    "platstdlib": "{platbase}/lib/python{py_version_short}",
    "purelib": "/Library/Python/{py_version_short}/site-packages",
    "platlib": "/Library/Python/{py_version_short}/site-packages",
    "include": "/Library/Python/{py_version_short}{abiflags}/include",
    "scripts": "/usr/local/bin",
    "data": "/Library/Python/{py_version_short}"
  },
  "sysconfig_vars": {
    "installed_base": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7",
    "py_version_short": "3.7",
    "abiflags": "m",
    "platbase": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7",
    "PYTHONFRAMEWORK": "Python3"
  },
  "distutils_install": {
    "purelib": "Library/Python/3.7/site-packages",
    "platlib": "Library/Python/3.7/site-packages",
    "headers": "Library/Python/3.7m/include/UNKNOWN",
    "scripts": "usr/local/bin",
    "data": "Library/Python/3.7"
  },
  "system_stdlib": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7",
  "system_stdlib_platform": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7",
  "max_size": 9223372036854775807,
  "_creators": null,
  "_stdlib_paths": null
}
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 11, 2020

macOS Mojava to which I have access no longer ships python 3, so cannot validate. But the issue seems the distutils_install being off. Can you post:

python3.7 -c 'import os; from distutils.dist import Distribution; d=Distribution();  i = d.get_command_obj("install", create=True); i.prefix = os.sep;  i.finalize_options(); import pprint; pprint.pprint(vars(i))'
@biochimia

This comment has been minimized.

Copy link

@biochimia biochimia commented Mar 11, 2020

$ /usr/bin/python3 -c 'import os; from distutils.dist import Distribution; d=Distribution();  i = d.get_command_obj("install", create=True); i.prefix = os.sep;  i.finalize_options(); import pprint; pprint.pprint(vars(i))'
{'_dry_run': None,
 'build_base': 'build',
 'build_lib': 'build/lib',
 'compile': None,
 'config_vars': {'abiflags': 'm',
                 'base': '/Library/Python',
                 'dist_fullname': 'UNKNOWN-0.0.0',
                 'dist_name': 'UNKNOWN',
                 'dist_version': '0.0.0',
                 'exec_prefix': '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7',
                 'platbase': '/Library/Python',
                 'prefix': '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7',
                 'py_version': '3.7.3',
                 'py_version_nodot': '37',
                 'py_version_short': '3.7',
                 'sys_exec_prefix': '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7',
                 'sys_prefix': '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7',
                 'userbase': '/Users/joao/Library/Python/3.7',
                 'usersite': '/Users/joao/Library/Python/3.7/lib/python/site-packages'},
 'distribution': <distutils.dist.Distribution object at 0x107356518>,
 'exec_prefix': None,
 'extra_dirs': '',
 'extra_path': None,
 'finalized': 0,
 'force': None,
 'help': 0,
 'home': None,
 'install_base': '/Library/Python',
 'install_data': '/Library/Python/3.7',
 'install_headers': '/Library/Python/3.7m/include/UNKNOWN',
 'install_lib': '/Library/Python/3.7/site-packages/',
 'install_libbase': '/Library/Python/3.7/site-packages',
 'install_path_file': 1,
 'install_platbase': '/Library/Python',
 'install_platlib': '/Library/Python/3.7/site-packages',
 'install_purelib': '/Library/Python/3.7/site-packages',
 'install_scripts': '/usr/local/bin',
 'install_userbase': '/Users/joao/Library/Python/3.7',
 'install_usersite': '/Users/joao/Library/Python/3.7/lib/python/site-packages',
 'optimize': None,
 'path_file': None,
 'prefix': '/',
 'record': None,
 'root': None,
 'skip_build': 0,
 'user': 0,
 'verbose': 1,
 'warn_dir': 1}
@biochimia

This comment has been minimized.

Copy link

@biochimia biochimia commented Mar 11, 2020

Btw, for me this only seems to affect python3 as shipped with Xcode Command Line Tools in macOS Catalina. Virtual environments created with homebrew's python3 setup seem to work just fine.

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 11, 2020

@biochimia can you try with https://github.com/pypa/virtualenv/pull/1711/files? Turns out there's a lot more at fault than just bad paths. That PR though should be a rough estimate of solution.

@biochimia

This comment has been minimized.

Copy link

@biochimia biochimia commented Mar 11, 2020

Checking out a12ed2a from #1711, it looks much better:

$ /usr/bin/python3
Python 3.7.3 (default, Dec 13 2019, 19:58:14)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

(Using virtualenv.pyz built with tasks/make_zipapp.py, using a virtual environment created with python3 -m venv and the latest pip, setuptools, wheel, and packaging packages.)

$ /usr/bin/python3 virtualenv.pyz --version
virtualenv 20.0.11.dev3+ga12ed2a from virtualenv.pyz/virtualenv/__init__.py
$ /usr/bin/python3 virtualenv.pyz xcode-py3-venv
created virtual environment CPython3.7.3.final.0-64 in 228ms
  creator CPython3macOsFramework(dest=/Users/joao/code/virtualenv/xcode-py3-venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/joao/Library/Application Support/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ tree -d -L 4 xcode-py3-venv
xcode-py3-venv
├── bin
└── lib
    └── python3.7
        └── site-packages
            ├── pip
            ├── pip-20.0.2.dist-info
            ├── pkg_resources
            ├── setuptools
            ├── setuptools-46.0.0.dist-info
            ├── wheel
            └── wheel-0.34.2.dist-info

11 directories
xcode-py3-venv/bin/pip --version
pip 20.0.2 from /Users/joao/code/virtualenv/xcode-py3-venv/lib/python3.7/site-packages/pip (python 3.7)

So, #1711 seems to work great here with Xcode's Command Line Tools' python3.

For comparison, in the same system and the same virtualenv.pyz, I get the tree below:

$ tree -d -L 4 homebrew-py3-venv
homebrew-py3-venv
├── bin
├── include
└── lib
    └── python3.7
        └── site-packages
            ├── pip
            ├── pip-20.0.2.dist-info
            ├── pkg_resources
            ├── setuptools
            ├── setuptools-46.0.0.dist-info
            ├── wheel
            └── wheel-0.34.2.dist-info

12 directories
@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 16, 2020

Resolved via #1711, we'll release it soon.

@gaborbernat

This comment has been minimized.

Copy link
Contributor Author

@gaborbernat gaborbernat commented Mar 19, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.11; see https://pypi.org/project/virtualenv/20.0.11/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-11-2020-03-18). Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.
thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.