Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

framework build CPython macOs issues #1561

Closed
rpkilby opened this issue Feb 10, 2020 · 40 comments · Fixed by #1571, #1588, #1641 or #1653
Closed

framework build CPython macOs issues #1561

rpkilby opened this issue Feb 10, 2020 · 40 comments · Fixed by #1571, #1588, #1641 or #1653

Comments

@rpkilby
Copy link

@rpkilby rpkilby commented Feb 10, 2020

Hi. I'm trying to create a Python 2 virtualenv on macOS Mojave and am running into an error.

bagel$ pip3 install --user virtualenv
  <snipped>
  WARNING: The script virtualenv is installed in '/Users/bagel/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed virtualenv-20.0.1

bagel$ pip3 list
Package            Version
------------------ -------
appdirs            1.4.3  
filelock           3.0.12 
importlib-metadata 1.5.0  
pip                19.3.1 
setuptools         42.0.2 
six                1.14.0 
virtualenv         20.0.1 
wheel              0.33.6 
zipp               2.2.0  

bagel$ which python2.7
/usr/bin/python2.7

bagel$ ~/Library/Python/3.7/bin/virtualenv -p python2.7 venv
PermissionError: [Errno 1] Operation not permitted: '/Users/bagel/Documents/projects/test/venv/bin/python'

It's worth noting that Python 2 is the system Python, while Python 3 is installed from Homebrew.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 10, 2020

Can you please show the output with -vvv? Thanks!

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 10, 2020

bagel$ ~/Library/Python/3.7/bin/virtualenv -vvv -p python2.7 venv
85 setup logging to NOTSET [DEBUG report:43]
97 find interpreter for spec PythonSpec(implementation=CPython, major=2, minor=7) [INFO builtin:44]
97 proposed PythonInfo(spec=CPython3.7.6.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.6 (default, Dec 30 2019, 19:38:28) \n[Clang 11.0.0 (clang-1100.0.33.16)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
97 discover PATH[0]=/usr/local/bin [DEBUG builtin:77]
98 filesystem is not case-sensitive [DEBUG info:28]
99 discover PATH[1]=/usr/local/sbin [DEBUG builtin:77]
99 discover PATH[2]=/usr/local/bin [DEBUG builtin:77]
99 discover PATH[3]=/usr/bin [DEBUG builtin:77]
100 Attempting to acquire lock 4505262672 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
100 Lock 4505262672 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
100 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:73]
101 Attempting to release lock 4505262672 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
101 Lock 4505262672 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
102 Attempting to acquire lock 4505327312 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:270]
102 Lock 4505327312 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:274]
102 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.json for /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG cached_py_info:73]
103 Attempting to release lock 4505327312 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:315]
104 Lock 4505327312 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:318]
104 proposed PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
104 accepted PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:52]
131 create virtual environment via CPython2Posix(dest=/Users/bagel/Documents/projects/test/venv, clear=False, global=False) [INFO session:24]
131 create folder /Users/bagel/Documents/projects/test/venv/bin [DEBUG _sync:22]
131 create folder /Users/bagel/Documents/projects/test/venv/include [DEBUG _sync:22]
132 create folder /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages [DEBUG _sync:22]
132 write /Users/bagel/Documents/projects/test/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
132 	home = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
132 	implementation = CPython [DEBUG pyenv_cfg:38]
132 	version_info = 2.7.10.final.0 [DEBUG pyenv_cfg:38]
132 	virtualenv = 20.0.1 [DEBUG pyenv_cfg:38]
132 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
132 	base-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
132 	base-exec-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
132 	base-executable = /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG pyenv_cfg:38]
132 copy /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python to /Users/bagel/Documents/projects/test/venv/bin/python [DEBUG _sync:49]
134 done in 132ms [INFO __main__:23]
134 PermissionError: [Errno 1] Operation not permitted: '/Users/bagel/Documents/projects/test/venv/bin/python' [ERROR __main__:30]

Also:

Sherpa-3:test bagel$ which python2.7
/usr/bin/python2.7
Sherpa-3:test bagel$ ls /usr/bin/python2.7
lrwxr-xr-x 1 root wheel 75 Nov 16  2018 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

@szmark001
Copy link

@szmark001 szmark001 commented Feb 11, 2020

I am seeing the same issue with version 20.0.1.

@mctrafik
Copy link

@mctrafik mctrafik commented Feb 11, 2020

+1 on Mac OS 10.15.3 Catalina. Same exact issue.

@gaborbernat gaborbernat changed the title Unable to create Python 2 virtualenvs on macOS macPython 2 not supported Feb 11, 2020
@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 11, 2020

So basically we do not support with the rewrite the MacOs framework python. We'll need to fix this. In this particular case it seems that copying the file stats of the system python is not allowed, hence the above error.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 11, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.2/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-2-2020-02-11) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 11, 2020

I don't think this is fixed entirely. Virtualenv creation doesn't fail, but there are a lot of unexpected packages present. Also, is the venv creation output expected?

bagel$ ~/Library/Python/3.7/bin/virtualenv -p python2.7 venv 
created virtual environment CPython2Posix(dest=/Users/bagel/Documents/projects/test/venv, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1 via=copy

bagel$ source venv/bin/activate

(venv) bagel$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                                Version 
-------------------------------------- --------
altgraph                               0.10.2  
bdist-mpkg                             0.5.0   
bonjour-py                             0.3     
macholib                               1.5.1   
matplotlib                             1.3.1   
modulegraph                            0.10.4  
numpy                                  1.8.0rc1
pip                                    20.0.2  
py2app                                 0.7.3   
pyobjc-core                            2.5.1   
pyobjc-framework-Accounts              2.5.1   
pyobjc-framework-AddressBook           2.5.1   
pyobjc-framework-AppleScriptKit        2.5.1   
pyobjc-framework-AppleScriptObjC       2.5.1   
pyobjc-framework-Automator             2.5.1   
pyobjc-framework-CFNetwork             2.5.1   
pyobjc-framework-Cocoa                 2.5.1   
pyobjc-framework-Collaboration         2.5.1   
pyobjc-framework-CoreData              2.5.1   
pyobjc-framework-CoreLocation          2.5.1   
pyobjc-framework-CoreText              2.5.1   
pyobjc-framework-DictionaryServices    2.5.1   
pyobjc-framework-EventKit              2.5.1   
pyobjc-framework-ExceptionHandling     2.5.1   
pyobjc-framework-FSEvents              2.5.1   
pyobjc-framework-InputMethodKit        2.5.1   
pyobjc-framework-InstallerPlugins      2.5.1   
pyobjc-framework-InstantMessage        2.5.1   
pyobjc-framework-LatentSemanticMapping 2.5.1   
pyobjc-framework-LaunchServices        2.5.1   
pyobjc-framework-Message               2.5.1   
pyobjc-framework-OpenDirectory         2.5.1   
pyobjc-framework-PreferencePanes       2.5.1   
pyobjc-framework-PubSub                2.5.1   
pyobjc-framework-QTKit                 2.5.1   
pyobjc-framework-Quartz                2.5.1   
pyobjc-framework-ScreenSaver           2.5.1   
pyobjc-framework-ScriptingBridge       2.5.1   
pyobjc-framework-SearchKit             2.5.1   
pyobjc-framework-ServiceManagement     2.5.1   
pyobjc-framework-Social                2.5.1   
pyobjc-framework-SyncServices          2.5.1   
pyobjc-framework-SystemConfiguration   2.5.1   
pyobjc-framework-WebKit                2.5.1   
pyOpenSSL                              0.13.1  
pyparsing                              2.0.1   
python-dateutil                        1.5     
pytz                                   2013.7  
scipy                                  0.13.0b1
setuptools                             44.0.0  
six                                    1.4.1   
vboxapi                                1.0     
wheel                                  0.34.2  
xattr                                  0.6.4   

However, the contents of the venv's site-packages are:

bagel$ ls venv/lib/python2.7/site-packages/
total 4.0K
drwxr-xr-x  7 bagel staff  224 Feb 11 10:42 pip
drwxr-xr-x  9 bagel staff  288 Feb 11 10:41 pip-20.0.2.dist-info
drwxr-xr-x  6 bagel staff  192 Feb 11 10:41 pkg_resources
drwxr-xr-x 43 bagel staff 1.4K Feb 11 10:41 setuptools
drwxr-xr-x 11 bagel staff  352 Feb 11 10:41 setuptools-44.0.0.dist-info
drwxr-xr-x 13 bagel staff  416 Feb 11 10:41 wheel
drwxr-xr-x  9 bagel staff  288 Feb 11 10:41 wheel-0.34.2.dist-info
-rw-r--r--  1 bagel staff  126 Feb 11 10:41 easy_install.py
-rw-r--r--  1 bagel staff    0 Feb 11 10:41 pip-20.0.2.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 11 10:41 setuptools-44.0.0.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 11 10:41 wheel-0.34.2.dist-info.virtualenv

@gaborbernat gaborbernat reopened this Feb 11, 2020
@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 11, 2020

Thanks I'll take a look why this happens 🤔 seems the system python does not respects ignore global site packages on site.py 🤔

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

#1588 fixes this now.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.3/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-3-2020-02-12) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

@YannickJadoul
Copy link
Member

@YannickJadoul YannickJadoul commented Feb 12, 2020

Something's still going wrong with Python 2 on macOS. As far as I notice, scripts are still installed globally. Is this possible?

This is output from Python 3 and Python 2, installed with HomeBrew.

Python 3, correctly installing the cibuildwheel script in the virtualenv's bin:

$ python3 -m virtualenv --version
virtualenv 20.0.3 from /usr/local/lib/python3.7/site-packages/virtualenv/__init__.py
$ python3 -m virtualenv TMPENV3
created virtual environment in 182ms CPython3Posix(dest=/Users/yannick/TMPENV3, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1 via=copy
$ . TMPENV3/bin/activate
(TMPENV3) $ pip install cibuildwheel
Collecting cibuildwheel
  Using cached cibuildwheel-1.1.0-py2.py3-none-any.whl (24 kB)
Processing ./Library/Caches/pip/wheels/53/49/86/a05c97f04697533da8e8d0e7e4a88cfeeb8e5aea5616a65e4a/bashlex-0.14-py3-none-any.whl
Installing collected packages: bashlex, cibuildwheel
Successfully installed bashlex-0.14 cibuildwheel-1.1.0
(TMPENV3) $ which cibuildwheel
/Users/yannick/TMPENV3/bin/cibuildwheel
(TMPENV3) $ deactivate

Python 2, installing the script globally, it seems:

$ python -m virtualenv --version
virtualenv 20.0.3 from /usr/local/lib/python2.7/site-packages/virtualenv/__init__.pyc
$ python -m virtualenv TMPENV
created virtual environment in 183ms CPython2Posix(dest=/Users/yannick/TMPENV, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1 via=copy
$ . TMPENV/bin/activate
(TMPENV) $ which pip
/Users/yannick/TMPENV/bin/pip
(TMPENV) $ pip install cibuildwheel
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting cibuildwheel
  Using cached cibuildwheel-1.1.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: bashlex!=0.13 in /usr/local/lib/python2.7/site-packages (from cibuildwheel) (0.14)
Requirement already satisfied: enum34 in /usr/local/lib/python2.7/site-packages (from bashlex!=0.13->cibuildwheel) (1.1.6)
Installing collected packages: cibuildwheel
Successfully installed cibuildwheel-1.1.0
(TMPENV) $ which cibuildwheel
/usr/local/bin/cibuildwheel

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

Does it install globally even if you don't use the activate, but invoke pip directly?

@gaborbernat gaborbernat reopened this Feb 12, 2020
@YannickJadoul
Copy link
Member

@YannickJadoul YannickJadoul commented Feb 12, 2020

Does it install globally even if you don't use the activate, but invoke pip directly?

Like this, you mean? Then yes:

ai31:~ yannick$ python -m virtualenv TMPENV_BIS
created virtual environment in 184ms CPython2Posix(dest=/Users/yannick/TMPENV_BIS, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1 via=copy
ai31:~ yannick$ TMPENV_BIS/bin/pip install cibuildwheel
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting cibuildwheel
  Using cached cibuildwheel-1.1.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: bashlex!=0.13 in /usr/local/lib/python2.7/site-packages (from cibuildwheel) (0.14)
Requirement already satisfied: enum34 in /usr/local/lib/python2.7/site-packages (from bashlex!=0.13->cibuildwheel) (1.1.6)
Installing collected packages: cibuildwheel
Successfully installed cibuildwheel-1.1.0
ai31:~ yannick$ which cibuildwheel
/usr/local/bin/cibuildwheel

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

Will check what's happening, thanks!

@YannickJadoul
Copy link
Member

@YannickJadoul YannickJadoul commented Feb 12, 2020

Will check what's happening, thanks!

Thank yóu for the quick follow-up. If you still need anything from me, do let me know!

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 12, 2020

Just throwing this out there - I wonder if this is somehow related to #1095.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.5; see https://pypi.org/project/virtualenv/20.0.5/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-5-2020-02-21) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

@YannickJadoul
Copy link
Member

@YannickJadoul YannickJadoul commented Feb 21, 2020

Thank you, @gaborbernat!

I'm trying out the new release at pypa/cibuildwheel#274, right now, and half of our problem is already gone (see first commit, succeedig). But something's still seems to be happening with the CircleCI build, running our tests in a virtualenv (see second commit, failing) :-/ I'll try to look further into it and see if I can make a minimal example.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

@YannickJadoul can't seem to understand what's happening in there, seems pypy and pypy3 is mixed in together making it hard to see what's happening. If you can create a minimal report that would be great.

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 21, 2020

Hi @gaborbernat. I've upgraded to virtualenv 20.0.5, but am still seeing this issue:
#1561 (comment)

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

Can you post a fresh trace exposing the issue?

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 21, 2020

bagel$ ~/Library/Python/3.7/bin/virtualenv --version
virtualenv 20.0.5 from /Users/bagel/Library/Python/3.7/lib/python/site-packages/virtualenv/__init__.py

bagel$ ~/Library/Python/3.7/bin/virtualenv -p python2.7 venv
created virtual environment CPython2.7.10.final.0-64 in 235ms
  creator CPython2macOsFramework(dest=/Users/bagel/Documents/projects/test/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator

bagel$ source venv/bin/activate

bagel$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                                Version 
-------------------------------------- --------
altgraph                               0.10.2  
bdist-mpkg                             0.5.0   
bonjour-py                             0.3     
macholib                               1.5.1   
matplotlib                             1.3.1   
modulegraph                            0.10.4  
numpy                                  1.8.0rc1
pip                                    20.0.2  
py2app                                 0.7.3   
pyobjc-core                            2.5.1   
pyobjc-framework-Accounts              2.5.1   
pyobjc-framework-AddressBook           2.5.1   
pyobjc-framework-AppleScriptKit        2.5.1   
pyobjc-framework-AppleScriptObjC       2.5.1   
pyobjc-framework-Automator             2.5.1   
pyobjc-framework-CFNetwork             2.5.1   
pyobjc-framework-Cocoa                 2.5.1   
pyobjc-framework-Collaboration         2.5.1   
pyobjc-framework-CoreData              2.5.1   
pyobjc-framework-CoreLocation          2.5.1   
pyobjc-framework-CoreText              2.5.1   
pyobjc-framework-DictionaryServices    2.5.1   
pyobjc-framework-EventKit              2.5.1   
pyobjc-framework-ExceptionHandling     2.5.1   
pyobjc-framework-FSEvents              2.5.1   
pyobjc-framework-InputMethodKit        2.5.1   
pyobjc-framework-InstallerPlugins      2.5.1   
pyobjc-framework-InstantMessage        2.5.1   
pyobjc-framework-LatentSemanticMapping 2.5.1   
pyobjc-framework-LaunchServices        2.5.1   
pyobjc-framework-Message               2.5.1   
pyobjc-framework-OpenDirectory         2.5.1   
pyobjc-framework-PreferencePanes       2.5.1   
pyobjc-framework-PubSub                2.5.1   
pyobjc-framework-QTKit                 2.5.1   
pyobjc-framework-Quartz                2.5.1   
pyobjc-framework-ScreenSaver           2.5.1   
pyobjc-framework-ScriptingBridge       2.5.1   
pyobjc-framework-SearchKit             2.5.1   
pyobjc-framework-ServiceManagement     2.5.1   
pyobjc-framework-Social                2.5.1   
pyobjc-framework-SyncServices          2.5.1   
pyobjc-framework-SystemConfiguration   2.5.1   
pyobjc-framework-WebKit                2.5.1   
pyOpenSSL                              0.13.1  
pyparsing                              2.0.1   
python-dateutil                        1.5     
pytz                                   2013.7  
scipy                                  0.13.0b1
setuptools                             44.0.0  
six                                    1.4.1   
vboxapi                                1.0     
wheel                                  0.34.2  
xattr                                  0.6.4   

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 21, 2020

And for completeness

bagel$ ls venv/lib/python2.7/site-packages/
total 4.0K
drwxr-xr-x  7 bagel staff  224 Feb 21 11:33 pip
drwxr-xr-x  9 bagel staff  288 Feb 21 11:32 pip-20.0.2.dist-info
drwxr-xr-x  8 bagel staff  256 Feb 21 11:33 pkg_resources
drwxr-xr-x 56 bagel staff 1.8K Feb 21 11:33 setuptools
drwxr-xr-x 11 bagel staff  352 Feb 21 11:32 setuptools-44.0.0.dist-info
drwxr-xr-x 13 bagel staff  416 Feb 21 11:32 wheel
drwxr-xr-x  9 bagel staff  288 Feb 21 11:32 wheel-0.34.2.dist-info
-rw-r--r--  1 bagel staff  126 Feb 21 11:32 easy_install.py
-rw-r--r--  1 bagel staff    0 Feb 21 11:32 pip-20.0.2.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 21 11:32 setuptools-44.0.0.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 21 11:32 wheel-0.34.2.dist-info.virtualenv

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

The usual caveats apply add v three times to the creation so we can see what happened 👍and ideally with --clear.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

Also in your case would help if you can display the -m site of virtual environment 👍 and the global python you're creating the venv from.

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 21, 2020

Note that I installed virtualenv with pip3 install --user virtualenv but am creating a Python 2 virtualenv.

bagel$ ~/Library/Python/3.7/bin/virtualenv -vvv -p python2.7 venv --clear
55 setup logging to NOTSET [DEBUG report:43]
61 find interpreter for spec PythonSpec(implementation=CPython, major=2, minor=7) [INFO builtin:43]
61 proposed PythonInfo(spec=CPython3.7.6.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.6 (default, Dec 30 2019, 19:38:28) \n[Clang 11.0.0 (clang-1100.0.33.16)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:49]
61 discover PATH[0]=/usr/local/bin [DEBUG builtin:76]
63 filesystem is not case-sensitive [DEBUG info:28]
63 discover PATH[1]=/usr/local/sbin [DEBUG builtin:76]
63 discover PATH[2]=/usr/local/bin [DEBUG builtin:76]
63 discover PATH[3]=/usr/bin [DEBUG builtin:76]
64 Attempting to acquire lock 4530209296 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
64 Lock 4530209296 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
64 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:72]
65 Attempting to release lock 4530209296 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
65 Lock 4530209296 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
65 Attempting to acquire lock 4530325136 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:270]
65 Lock 4530325136 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:274]
65 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.json for /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG cached_py_info:72]
66 Attempting to release lock 4530325136 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:315]
66 Lock 4530325136 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:318]
66 proposed PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:49]
66 accepted PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:51]
79 create virtual environment via CPython2macOsFramework(dest=/Users/bagel/Documents/projects/test/venv, clear=True, global=False) [INFO session:51]
80 create folder /Users/bagel/Documents/projects/test/venv/bin [DEBUG _sync:20]
80 create folder /Users/bagel/Documents/projects/test/venv/include [DEBUG _sync:20]
80 create folder /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages [DEBUG _sync:20]
80 write /Users/bagel/Documents/projects/test/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
80 	home = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
80 	implementation = CPython [DEBUG pyenv_cfg:38]
80 	version_info = 2.7.10.final.0 [DEBUG pyenv_cfg:38]
80 	virtualenv = 20.0.5 [DEBUG pyenv_cfg:38]
80 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
80 	base-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
80 	base-exec-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
80 	base-executable = /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG pyenv_cfg:38]
81 copy /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python to /Users/bagel/Documents/projects/test/venv/bin/python [DEBUG _sync:47]
81 symlink /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py to /Users/bagel/Documents/projects/test/venv/lib/python2.7/os.py [DEBUG _sync:39]
81 symlink /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc to /Users/bagel/Documents/projects/test/venv/lib/python2.7/os.pyc [DEBUG _sync:39]
82 symlink directory /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 to /Users/bagel/Documents/projects/test/venv/include/python2.7 [DEBUG _sync:39]
82 symlink directory /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload to /Users/bagel/Documents/projects/test/venv/lib/python2.7/lib-dynload [DEBUG _sync:39]
82 symlink /System/Library/Frameworks/Python.framework/Versions/2.7/Python to /Users/bagel/Documents/projects/test/venv/.Python [DEBUG _sync:39]
82 change Mach-O for /Users/bagel/Documents/projects/test/venv/bin/python from /System/Library/Frameworks/Python.framework/Versions/2.7/Python to @executable_path/../.Python [DEBUG mac_os:77]
83 ============================== target debug ============================== [DEBUG session:53]
83 debug via /Users/bagel/Documents/projects/test/venv/bin/python /Users/bagel/Library/Python/3.7/lib/python/site-packages/virtualenv/create/debug.py [DEBUG creator:191]
83 {
  "sys": {
    "executable": "/Users/bagel/Documents/projects/test/venv/bin/python",
    "_base_executable": null,
    "prefix": "/Users/bagel/Documents/projects/test/venv",
    "base_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
    "real_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
    "exec_prefix": "/Users/bagel/Documents/projects/test/venv",
    "base_exec_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
    "path": [
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload",
      "/Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages",
      "/Library/Python/2.7/site-packages",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC"
    ],
    "meta_path": [],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]",
  "os": "<module 'os' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc'>",
  "site": "<module 'site' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc'>",
  "datetime": "<module 'datetime' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so'>",
  "math": "<module 'math' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so'>",
  "json": "<module 'json' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc'>"
} [DEBUG session:54]
215 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1) [INFO session:58]
215 Attempting to acquire lock 4531077264 on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:270]
216 Lock 4531077264 acquired on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:274]
217 install pip from wheel /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
217 install setuptools from wheel /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels/setuptools-44.0.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
218 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info.virtualenv to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pip-20.0.2.dist-info.virtualenv [DEBUG _sync:47]
218 install wheel from wheel /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
218 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/easy_install.py to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/easy_install.py [DEBUG _sync:47]
218 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/wheel [DEBUG _sync:47]
219 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pip [DEBUG _sync:47]
219 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info.virtualenv to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/setuptools-44.0.0.dist-info.virtualenv [DEBUG _sync:47]
220 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/setuptools [DEBUG _sync:47]
230 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info.virtualenv to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/wheel-0.34.2.dist-info.virtualenv [DEBUG _sync:47]
231 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:47]
292 changing mode of /Users/bagel/Documents/projects/test/venv/bin/wheel to 755 [INFO util:566]
293 changing mode of /Users/bagel/Documents/projects/test/venv/bin/wheel3 to 755 [INFO util:566]
293 changing mode of /Users/bagel/Documents/projects/test/venv/bin/wheel-3.7 to 755 [INFO util:566]
293 generated console scripts wheel wheel-3.7 wheel3 [DEBUG base:48]
321 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/pkg_resources to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pkg_resources [DEBUG _sync:47]
332 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/setuptools-44.0.0.dist-info [DEBUG _sync:47]
338 changing mode of /Users/bagel/Documents/projects/test/venv/bin/easy_install3 to 755 [INFO util:566]
338 changing mode of /Users/bagel/Documents/projects/test/venv/bin/easy_install-3.7 to 755 [INFO util:566]
339 changing mode of /Users/bagel/Documents/projects/test/venv/bin/easy_install to 755 [INFO util:566]
339 generated console scripts easy_install-3.7 easy_install easy_install3 [DEBUG base:48]
441 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pip-20.0.2.dist-info [DEBUG _sync:47]
445 changing mode of /Users/bagel/Documents/projects/test/venv/bin/pip to 755 [INFO util:566]
445 changing mode of /Users/bagel/Documents/projects/test/venv/bin/pip-3.7 to 755 [INFO util:566]
446 changing mode of /Users/bagel/Documents/projects/test/venv/bin/pip3 to 755 [INFO util:566]
446 generated console scripts pip-3.7 pip3 pip [DEBUG base:48]
446 Attempting to release lock 4531077264 on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:315]
446 Lock 4531077264 released on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:318]
446 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
449 write /Users/bagel/Documents/projects/test/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
449 	home = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
449 	implementation = CPython [DEBUG pyenv_cfg:38]
450 	version_info = 2.7.10.final.0 [DEBUG pyenv_cfg:38]
450 	virtualenv = 20.0.5 [DEBUG pyenv_cfg:38]
450 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
450 	base-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
450 	base-exec-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
450 	base-executable = /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG pyenv_cfg:38]
450 created virtual environment CPython2.7.10.final.0-64 in 395ms
  creator CPython2macOsFramework(dest=/Users/bagel/Documents/projects/test/venv, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:21]

Global Python 3 site (note: virtualenv is installed to my user directory using pip3)

bagel$ python3 -m site
sys.path = [
    '/Users/bagel/Documents/projects/test',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python37.zip',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload',
    '/Users/bagel/Library/Python/3.7/lib/python/site-packages',
    '/usr/local/lib/python3.7/site-packages',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages',
]
USER_BASE: '/Users/bagel/Library/Python/3.7' (exists)
USER_SITE: '/Users/bagel/Library/Python/3.7/lib/python/site-packages' (exists)
ENABLE_USER_SITE: True

Global Python 2 site:

bagel$ python -m site
sys.path = [
    '/Users/bagel/Documents/projects/test',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/Library/Python/2.7/site-packages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
]
USER_BASE: '/Users/bagel/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/bagel/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

venv site:

(venv) bagel$ python -m site
sys.path = [
    '/Users/bagel/Documents/projects/test',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages',
    '/Library/Python/2.7/site-packages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
]
USER_BASE: '/Users/bagel/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/bagel/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

@gaborbernat gaborbernat reopened this Feb 21, 2020
@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

Oh another differently customized python under System path this time 🤷‍♂️macOs is wonderful. If you install brew version that will work.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

Note for some reason we're picking up /usr/bin/python2.7, not the global python you're showcasing.

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 21, 2020

For context I've installed Python 3.7 via brew, but Python 2.7 is what shipped with Mojave.

Note for some reason we're picking up /usr/bin/python2.7, not the global python you're showcasing.

It's just a symlink to the framework Python.

bagel$ ls /usr/bin/python2.7
lrwxr-xr-x 1 root wheel 75 Nov 16  2018 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

I'm fairly certain the issue here is that site.py of the shipped python adds the global python to the sys path unconditionally (ignoring blatantly the enable global site packages flag within site.py 🤦‍♂️🤷‍♂️ macOS) so you're picking up things as that. We'll need to escape this in the custom shim site.py we ship 👍

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 21, 2020

Idk if it helps, but here are the contents of the site module.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py
"""Append module search paths for third-party packages to sys.path.

****************************************************************
* This module is automatically imported during initialization. *
****************************************************************

In earlier versions of Python (up to 1.5a3), scripts or modules that
needed to use site-specific modules would place ``import site''
somewhere near the top of their code.  Because of the automatic
import, this is no longer necessary (but code that does it still
works).

This will append site-specific paths to the module search path.  On
Unix (including Mac OSX), it starts with sys.prefix and
sys.exec_prefix (if different) and appends
lib/python<version>/site-packages as well as lib/site-python.
On other platforms (such as Windows), it tries each of the
prefixes directly, as well as with lib/site-packages appended.  The
resulting directories, if they exist, are appended to sys.path, and
also inspected for path configuration files.

A path configuration file is a file whose name has the form
<package>.pth; its contents are additional directories (one per line)
to be added to sys.path.  Non-existing directories (or
non-directories) are never added to sys.path; no directory is added to
sys.path more than once.  Blank lines and lines beginning with
'#' are skipped. Lines starting with 'import' are executed.

For example, suppose sys.prefix and sys.exec_prefix are set to
/usr/local and there is a directory /usr/local/lib/python2.5/site-packages
with three subdirectories, foo, bar and spam, and two path
configuration files, foo.pth and bar.pth.  Assume foo.pth contains the
following:

  # foo package configuration
  foo
  bar
  bletch

and bar.pth contains:

  # bar package configuration
  bar

Then the following directories are added to sys.path, in this order:

  /usr/local/lib/python2.5/site-packages/bar
  /usr/local/lib/python2.5/site-packages/foo

Note that bletch is omitted because it doesn't exist; bar precedes foo
because bar.pth comes alphabetically before foo.pth; and spam is
omitted because it is not mentioned in either path configuration file.

After these path manipulations, an attempt is made to import a module
named sitecustomize, which can perform arbitrary additional
site-specific customizations.  If this import fails with an
ImportError exception, it is silently ignored.

"""

import sys
import os
import __builtin__
import traceback

# Prefixes for site-packages; add additional prefixes like /usr/local here
PREFIXES = [sys.prefix, sys.exec_prefix]
# Enable per user site-packages directory
# set it to False to disable the feature or True to force the feature
ENABLE_USER_SITE = None

# for distutils.commands.install
# These values are initialized by the getuserbase() and getusersitepackages()
# functions, through the main() function when Python starts.
USER_SITE = None
USER_BASE = None


def makepath(*paths):
    dir = os.path.join(*paths)
    try:
        dir = os.path.abspath(dir)
    except OSError:
        pass
    return dir, os.path.normcase(dir)


def abs__file__():
    """Set all module' __file__ attribute to an absolute path"""
    for m in sys.modules.values():
        if hasattr(m, '__loader__'):
            continue   # don't mess with a PEP 302-supplied __file__
        try:
            m.__file__ = os.path.abspath(m.__file__)
        except (AttributeError, OSError):
            pass


def removeduppaths():
    """ Remove duplicate entries from sys.path along with making them
    absolute"""
    # This ensures that the initial path provided by the interpreter contains
    # only absolute pathnames, even if we're running from the build directory.
    L = []
    known_paths = set()
    for dir in sys.path:
        # Filter out duplicate paths (on case-insensitive file systems also
        # if they only differ in case); turn relative paths into absolute
        # paths.
        dir, dircase = makepath(dir)
        if not dircase in known_paths:
            L.append(dir)
            known_paths.add(dircase)
    sys.path[:] = L
    return known_paths


def _init_pathinfo():
    """Return a set containing all existing directory entries from sys.path"""
    d = set()
    for dir in sys.path:
        try:
            if os.path.isdir(dir):
                dir, dircase = makepath(dir)
                d.add(dircase)
        except TypeError:
            continue
    return d


def addpackage(sitedir, name, known_paths):
    """Process a .pth file within the site-packages directory:
       For each line in the file, either combine it with sitedir to a path
       and add that to known_paths, or execute it if it starts with 'import '.
    """
    if known_paths is None:
        _init_pathinfo()
        reset = 1
    else:
        reset = 0
    fullname = os.path.join(sitedir, name)
    try:
        f = open(fullname, "rU")
    except IOError:
        return
    with f:
        for n, line in enumerate(f):
            if line.startswith("#"):
                continue
            try:
                if line.startswith(("import ", "import\t")):
                    exec line
                    continue
                line = line.rstrip()
                dir, dircase = makepath(sitedir, line)
                if not dircase in known_paths and os.path.exists(dir):
                    sys.path.append(dir)
                    known_paths.add(dircase)
            except Exception as err:
                print >>sys.stderr, "Error processing line {:d} of {}:\n".format(
                    n+1, fullname)
                for record in traceback.format_exception(*sys.exc_info()):
                    for line in record.splitlines():
                        print >>sys.stderr, '  '+line
                print >>sys.stderr, "\nRemainder of file ignored"
                break
    if reset:
        known_paths = None
    return known_paths


def addsitedir(sitedir, known_paths=None):
    """Add 'sitedir' argument to sys.path if missing and handle .pth files in
    'sitedir'"""
    if known_paths is None:
        known_paths = _init_pathinfo()
        reset = 1
    else:
        reset = 0
    sitedir, sitedircase = makepath(sitedir)
    if not sitedircase in known_paths:
        sys.path.append(sitedir)        # Add path component
    try:
        names = os.listdir(sitedir)
    except os.error:
        return
    dotpth = os.extsep + "pth"
    names = [name for name in names if name.endswith(dotpth)]
    for name in sorted(names):
        addpackage(sitedir, name, known_paths)
    if reset:
        known_paths = None
    return known_paths


def check_enableusersite():
    """Check if user site directory is safe for inclusion

    The function tests for the command line flag (including environment var),
    process uid/gid equal to effective uid/gid.

    None: Disabled for security reasons
    False: Disabled by user (command line option)
    True: Safe and enabled
    """
    if sys.flags.no_user_site:
        return False

    if hasattr(os, "getuid") and hasattr(os, "geteuid"):
        # check process uid == effective uid
        if os.geteuid() != os.getuid():
            return None
    if hasattr(os, "getgid") and hasattr(os, "getegid"):
        # check process gid == effective gid
        if os.getegid() != os.getgid():
            return None

    return True

def getuserbase():
    """Returns the `user base` directory path.

    The `user base` directory can be used to store data. If the global
    variable ``USER_BASE`` is not initialized yet, this function will also set
    it.
    """
    global USER_BASE
    if USER_BASE is not None:
        return USER_BASE
    from sysconfig import get_config_var
    USER_BASE = get_config_var('userbase')
    return USER_BASE

def getusersitepackages():
    """Returns the user-specific site-packages directory path.

    If the global variable ``USER_SITE`` is not initialized yet, this
    function will also set it.
    """
    global USER_SITE
    user_base = getuserbase() # this will also set USER_BASE

    if USER_SITE is not None:
        return USER_SITE

    from sysconfig import get_path
    import os

    if sys.platform == 'darwin':
        from sysconfig import get_config_var
        if get_config_var('PYTHONFRAMEWORK'):
            USER_SITE = get_path('purelib', 'osx_framework_user')
            return USER_SITE

    USER_SITE = get_path('purelib', '%s_user' % os.name)
    return USER_SITE

def addusersitepackages(known_paths):
    """Add a per user site-package to sys.path

    Each user has its own python directory with site-packages in the
    home directory.
    """
    # get the per user site-package path
    # this call will also make sure USER_BASE and USER_SITE are set
    user_site = getusersitepackages()

    if ENABLE_USER_SITE and os.path.isdir(user_site):
        addsitedir(user_site, known_paths)
    return known_paths

def getsitepackages():
    """Returns a list containing all global site-packages directories
    (and possibly site-python).

    For each directory present in the global ``PREFIXES``, this function
    will find its `site-packages` subdirectory depending on the system
    environment, and will return a list of full paths.
    """
    sitepackages = []
    seen = set()

    for prefix in PREFIXES:
        if not prefix or prefix in seen:
            continue
        seen.add(prefix)

        if sys.platform in ('os2emx', 'riscos'):
            sitepackages.append(os.path.join(prefix, "Lib", "site-packages"))
        elif os.sep == '/':
            sitepackages.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "site-packages"))
            sitepackages.append(os.path.join(prefix, "lib", "site-python"))
        else:
            sitepackages.append(prefix)
            sitepackages.append(os.path.join(prefix, "lib", "site-packages"))
        if sys.platform == "darwin":
            # for framework builds *only* we add the standard Apple
            # locations.
            from sysconfig import get_config_var
            framework = get_config_var("PYTHONFRAMEWORK")
            if framework:
                sitepackages.append(
                        os.path.join("/Library", framework,
                            sys.version[:3], "site-packages"))
    return sitepackages

def addsitepackages(known_paths):
    """Add site-packages (and possibly site-python) to sys.path"""
    for sitedir in getsitepackages():
        if os.path.isdir(sitedir):
            addsitedir(sitedir, known_paths)

    return known_paths

def setBEGINLIBPATH():
    """The OS/2 EMX port has optional extension modules that do double duty
    as DLLs (and must use the .DLL file extension) for other extensions.
    The library search path needs to be amended so these will be found
    during module import.  Use BEGINLIBPATH so that these are at the start
    of the library search path.

    """
    dllpath = os.path.join(sys.prefix, "Lib", "lib-dynload")
    libpath = os.environ['BEGINLIBPATH'].split(';')
    if libpath[-1]:
        libpath.append(dllpath)
    else:
        libpath[-1] = dllpath
    os.environ['BEGINLIBPATH'] = ';'.join(libpath)


def setquit():
    """Define new builtins 'quit' and 'exit'.

    These are objects which make the interpreter exit when called.
    The repr of each object contains a hint at how it works.

    """
    if os.sep == ':':
        eof = 'Cmd-Q'
    elif os.sep == '\\':
        eof = 'Ctrl-Z plus Return'
    else:
        eof = 'Ctrl-D (i.e. EOF)'

    class Quitter(object):
        def __init__(self, name):
            self.name = name
        def __repr__(self):
            return 'Use %s() or %s to exit' % (self.name, eof)
        def __call__(self, code=None):
            # Shells like IDLE catch the SystemExit, but listen when their
            # stdin wrapper is closed.
            try:
                sys.stdin.close()
            except:
                pass
            raise SystemExit(code)
    __builtin__.quit = Quitter('quit')
    __builtin__.exit = Quitter('exit')


class _Printer(object):
    """interactive prompt objects for printing the license text, a list of
    contributors and the copyright notice."""

    MAXLINES = 23

    def __init__(self, name, data, files=(), dirs=()):
        self.__name = name
        self.__data = data
        self.__files = files
        self.__dirs = dirs
        self.__lines = None

    def __setup(self):
        if self.__lines:
            return
        data = None
        for dir in self.__dirs:
            for filename in self.__files:
                filename = os.path.join(dir, filename)
                try:
                    fp = file(filename, "rU")
                    data = fp.read()
                    fp.close()
                    break
                except IOError:
                    pass
            if data:
                break
        if not data:
            data = self.__data
        self.__lines = data.split('\n')
        self.__linecnt = len(self.__lines)

    def __repr__(self):
        self.__setup()
        if len(self.__lines) <= self.MAXLINES:
            return "\n".join(self.__lines)
        else:
            return "Type %s() to see the full %s text" % ((self.__name,)*2)

    def __call__(self):
        self.__setup()
        prompt = 'Hit Return for more, or q (and Return) to quit: '
        lineno = 0
        while 1:
            try:
                for i in range(lineno, lineno + self.MAXLINES):
                    print self.__lines[i]
            except IndexError:
                break
            else:
                lineno += self.MAXLINES
                key = None
                while key is None:
                    key = raw_input(prompt)
                    if key not in ('', 'q'):
                        key = None
                if key == 'q':
                    break

def setcopyright():
    """Set 'copyright' and 'credits' in __builtin__"""
    __builtin__.copyright = _Printer("copyright", sys.copyright)
    if sys.platform[:4] == 'java':
        __builtin__.credits = _Printer(
            "credits",
            "Jython is maintained by the Jython developers (www.jython.org).")
    else:
        __builtin__.credits = _Printer("credits", """\
    Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
    for supporting Python development.  See www.python.org for more information.""")
    here = os.path.dirname(os.__file__)
    __builtin__.license = _Printer(
        "license", "See https://www.python.org/psf/license/",
        ["LICENSE.txt", "LICENSE"],
        [os.path.join(here, os.pardir), here, os.curdir])


class _Helper(object):
    """Define the builtin 'help'.
    This is a wrapper around pydoc.help (with a twist).

    """

    def __repr__(self):
        return "Type help() for interactive help, " \
               "or help(object) for help about object."
    def __call__(self, *args, **kwds):
        import pydoc
        return pydoc.help(*args, **kwds)

def sethelper():
    __builtin__.help = _Helper()

def aliasmbcs():
    """On Windows, some default encodings are not provided by Python,
    while they are always available as "mbcs" in each locale. Make
    them usable by aliasing to "mbcs" in such a case."""
    if sys.platform == 'win32':
        import locale, codecs
        enc = locale.getdefaultlocale()[1]
        if enc.startswith('cp'):            # "cp***" ?
            try:
                codecs.lookup(enc)
            except LookupError:
                import encodings
                encodings._cache[enc] = encodings._unknown
                encodings.aliases.aliases[enc] = 'mbcs'

def setencoding():
    """Set the string encoding used by the Unicode implementation.  The
    default is 'ascii', but if you're willing to experiment, you can
    change this."""
    encoding = "ascii" # Default value set by _PyUnicode_Init()
    if 0:
        # Enable to support locale aware default string encodings.
        import locale
        loc = locale.getdefaultlocale()
        if loc[1]:
            encoding = loc[1]
    if 0:
        # Enable to switch off string to Unicode coercion and implicit
        # Unicode to string conversion.
        encoding = "undefined"
    if encoding != "ascii":
        # On Non-Unicode builds this will raise an AttributeError...
        sys.setdefaultencoding(encoding) # Needs Python Unicode build !


def execsitecustomize():
    """Run custom site specific code, if available."""
    try:
        import sitecustomize
    except ImportError:
        pass
    except Exception:
        if sys.flags.verbose:
            sys.excepthook(*sys.exc_info())
        else:
            print >>sys.stderr, \
                "'import sitecustomize' failed; use -v for traceback"


def execusercustomize():
    """Run custom user specific code, if available."""
    try:
        import usercustomize
    except ImportError:
        pass
    except Exception:
        if sys.flags.verbose:
            sys.excepthook(*sys.exc_info())
        else:
            print>>sys.stderr, \
                "'import usercustomize' failed; use -v for traceback"


def main():
    global ENABLE_USER_SITE

    abs__file__()
    known_paths = removeduppaths()
    if ENABLE_USER_SITE is None:
        ENABLE_USER_SITE = check_enableusersite()
    known_paths = addusersitepackages(known_paths)
    known_paths = addsitepackages(known_paths)
    if sys.platform == 'os2emx':
        setBEGINLIBPATH()
    setquit()
    setcopyright()
    sethelper()
    aliasmbcs()
    setencoding()
    execsitecustomize()
    if ENABLE_USER_SITE:
        execusercustomize()
    # Remove sys.setdefaultencoding() so that users cannot change the
    # encoding after initialization.  The test for presence is needed when
    # this module is run as a script, because this code is executed twice.
    if hasattr(sys, "setdefaultencoding"):
        del sys.setdefaultencoding

main()

def _script():
    help = """\
    %s [--user-base] [--user-site]

    Without arguments print some useful information
    With arguments print the value of USER_BASE and/or USER_SITE separated
    by '%s'.

    Exit codes with --user-base or --user-site:
      0 - user site directory is enabled
      1 - user site directory is disabled by user
      2 - uses site directory is disabled by super user
          or for security reasons
     >2 - unknown error
    """
    args = sys.argv[1:]
    if not args:
        print "sys.path = ["
        for dir in sys.path:
            print "    %r," % (dir,)
        print "]"
        print "USER_BASE: %r (%s)" % (USER_BASE,
            "exists" if os.path.isdir(USER_BASE) else "doesn't exist")
        print "USER_SITE: %r (%s)" % (USER_SITE,
            "exists" if os.path.isdir(USER_SITE) else "doesn't exist")
        print "ENABLE_USER_SITE: %r" %  ENABLE_USER_SITE
        sys.exit(0)

    buffer = []
    if '--user-base' in args:
        buffer.append(USER_BASE)
    if '--user-site' in args:
        buffer.append(USER_SITE)

    if buffer:
        print os.pathsep.join(buffer)
        if ENABLE_USER_SITE:
            sys.exit(0)
        elif ENABLE_USER_SITE is False:
            sys.exit(1)
        elif ENABLE_USER_SITE is None:
            sys.exit(2)
        else:
            sys.exit(3)
    else:
        import textwrap
        print textwrap.dedent(help % (sys.argv[0], os.pathsep))
        sys.exit(10)

if __name__ == '__main__':
    _script()

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 21, 2020

Yeah see the hardcoded part at

framework = get_config_var("PYTHONFRAMEWORK") if framework: sitepackages.append( os.path.join("/Library", framework, sys.version[:3], "site-packages"

@YannickJadoul
Copy link
Member

@YannickJadoul YannickJadoul commented Feb 21, 2020

@gaborbernat I've managed to reproduce my remaining issue in a simpler way:

mkdir tmp_dir
cd tmp_dir

curl -L https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-osx64.tar.bz2 | tar xj
curl -L https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-osx64.tar.bz2 | tar xj
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

python3 -m virtualenv --version
python3 -m virtualenv env
source env/bin/activate

pip install pytest

cat << EOF > test.py
import os
import subprocess

def test2():
    subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', 'get-pip.py', '-v'])
    subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', '-m', 'site'])
    print(os.listdir('pypy2.7-v7.3.0-osx64/bin'))
    assert False

def test3():
    subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', 'get-pip.py', '-v'])
    subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', '-m', 'site'])
    print(os.listdir('pypy3.6-v7.3.0-osx64/bin'))
    assert False

EOF

pytest test.py
Output

ai31:~ yannick$ ./minimal_example.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 20.5M  100 20.5M    0     0  1263k      0  0:00:16  0:00:16 --:--:-- 1707k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 20.2M  100 20.2M    0     0  1406k      0  0:00:14  0:00:14 --:--:-- 1607k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1764k  100 1764k    0     0  6561k      0 --:--:-- --:--:-- --:--:-- 6561k
virtualenv 20.0.5 from /usr/local/lib/python3.7/site-packages/virtualenv/__init__.py
created virtual environment CPython3.7.5.final.0-64 in 199ms
  creator CPython3Posix(dest=/Users/yannick/tmp_dir/env, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Collecting pytest
  Using cached pytest-5.3.5-py3-none-any.whl (235 kB)
Collecting packaging
  Using cached packaging-20.1-py2.py3-none-any.whl (36 kB)
Collecting importlib-metadata>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0
  Using cached more_itertools-8.2.0-py3-none-any.whl (43 kB)
Collecting py>=1.5.0
  Using cached py-1.8.1-py2.py3-none-any.whl (83 kB)
Collecting pluggy<1.0,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting wcwidth
  Using cached wcwidth-0.1.8-py2.py3-none-any.whl (17 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.6-py2.py3-none-any.whl (67 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting zipp>=0.5
  Using cached zipp-3.0.0-py3-none-any.whl (4.8 kB)
Installing collected packages: pyparsing, six, packaging, zipp, importlib-metadata, more-itertools, py, pluggy, attrs, wcwidth, pytest
Successfully installed attrs-19.3.0 importlib-metadata-1.5.0 more-itertools-8.2.0 packaging-20.1 pluggy-0.13.1 py-1.8.1 pyparsing-2.4.6 pytest-5.3.5 six-1.14.0 wcwidth-0.1.8 zipp-3.0.0
======================================== test session starts ========================================
platform darwin -- Python 3.7.5, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/yannick/tmp_dir
collected 2 items                                                                                   

test.py FF                                                                                    [100%]

============================================= FAILURES ==============================================
_______________________________________________ test2 _______________________________________________

    def test2():
        subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', 'get-pip.py', '-v'])
        subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', '-m', 'site'])
        print(os.listdir('pypy2.7-v7.3.0-osx64/bin'))
>       assert False
E       assert False

test.py:8: AssertionError
--------------------------------------- Captured stdout call ----------------------------------------
Non-user install because site-packages writeable
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-ephem-wheel-cache-Dzm69W
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d
Initialized build tracking at /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d
Created build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d
Entered build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-install-yhafaR
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0
  Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2

...

Installing collected packages: pip, setuptools, wheel
  Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-xnir60

  changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/pip to 755
  changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/pip2 to 755
  changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/pip2.7 to 755
  Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-qGsleO

  changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/easy_install to 755
  changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/easy_install-2.7 to 755
  Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-erEv1A

  changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/wheel to 755
Successfully installed pip-20.0.2 setuptools-44.0.0 wheel-0.34.2
Cleaning up...
Removed build tracker: '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d'
sys.path = [
    '/Users/yannick/tmp_dir',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib_pypy/__extensions__',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib_pypy',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/lib-tk',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/plat-darwin',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/plat-mac',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/plat-mac/lib-scriptpackages',
    '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/site-packages',
]
USER_BASE: '/Users/yannick/.local' (doesn't exist)
USER_SITE: '/Users/yannick/.local/lib/pypy2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
['libpypy-c.dylib', 'pip2', 'easy_install', 'wheel', 'pip', 'pypy', 'easy_install-2.7', 'pip2.7']
--------------------------------------- Captured stderr call ----------------------------------------
DEPRECATION: A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
_______________________________________________ test3 _______________________________________________

    def test3():
        subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', 'get-pip.py', '-v'])
        subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', '-m', 'site'])
        print(os.listdir('pypy3.6-v7.3.0-osx64/bin'))
>       assert False
E       assert False

test.py:14: AssertionError
--------------------------------------- Captured stdout call ----------------------------------------
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-ephem-wheel-cache-p_j2cq6p
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b
Initialized build tracking at /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b
Created build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b
Entered build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b
Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-install-08qoikav
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0
  Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2

...

Installing collected packages: pip, setuptools, wheel
  Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-_wruh4x9

  changing mode of /Users/yannick/tmp_dir/env/bin/pip to 755
  changing mode of /Users/yannick/tmp_dir/env/bin/pip3 to 755
  changing mode of /Users/yannick/tmp_dir/env/bin/pip3.6 to 755
  Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-15knx9f1

  changing mode of /Users/yannick/tmp_dir/env/bin/easy_install to 755
  changing mode of /Users/yannick/tmp_dir/env/bin/easy_install-3.6 to 755
  Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-n8_gcseu

  changing mode of /Users/yannick/tmp_dir/env/bin/wheel to 755
Successfully installed pip-20.0.2 setuptools-45.2.0 wheel-0.34.2
Cleaning up...
Removed build tracker: '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b'
sys.path = [
    '/Users/yannick/tmp_dir',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy/__extensions__',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/lib-tk',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-darwin',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac/lib-scriptpackages',
    '/Users/yannick/tmp_dir/env/site-packages',
]
USER_BASE: '/Users/yannick/.local' (doesn't exist)
USER_SITE: '/Users/yannick/.local/lib/pypy3.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
['libpypy3-c.dylib', 'pypy3']
======================================== 2 failed in 23.66s =========================================

I think the main thing to notice in that whole output, is that when running PyPy 3.6, v7.3.0 from within pytest from within a virtualenv (oh yes, and somehow only on macOS; and oh yes, on virtualenv<20, it works), -m site gives this output:

sys.path = [
    '/Users/yannick/tmp_dir',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy/__extensions__',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/lib-tk',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-darwin',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac/lib-scriptpackages',
    '/Users/yannick/tmp_dir/env/site-packages',
]
USER_BASE: '/Users/yannick/.local' (doesn't exist)
USER_SITE: '/Users/yannick/.local/lib/pypy3.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

while if I afterwards run the same command outside of pytest, I get:

ai31:~ yannick$ cd tmp_dir/
ai31:tmp_dir yannick$ source env/bin/activate
(env) ai31:tmp_dir yannick$ pypy3.6-v7.3.0-osx64/bin/pypy3 -m site
sys.path = [
    '/Users/yannick/tmp_dir',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy/__extensions__',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/lib-tk',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-darwin',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac/lib-scriptpackages',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/site-packages',
]
USER_BASE: '/Users/yannick/.local' (doesn't exist)
USER_SITE: '/Users/yannick/.local/lib/pypy3.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

Somehow, the same doesn't happen with PyPy 2.7.

EDIT: I'm really sorry for finding such a nasty bug, that needs to be reproduced in such a horrible way :-/

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 22, 2020

@rpkilby your issue should be addressed with #1653, can you validate? Thanks!

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 22, 2020

@YannickJadoul you're issue is related with pypy, please open a separate issue for that; with a clearer description of what's expected and what happens 👍 this issue is mainly about CPython Framework builds on macOs.

@YannickJadoul
Copy link
Member

@YannickJadoul YannickJadoul commented Feb 22, 2020

@YannickJadoul you're issue is related with pypy, please open a separate issue for that; with a clearer description of what's expected and what happens +1 this issue is mainly about CPython Framework builds on macOs.

Yeah, sorry, I came across these two issues at the same time, so I kind of assumed they were linked. I've opened #1654 now, hopefully that's more clear :-)

@rpkilby
Copy link
Author

@rpkilby rpkilby commented Feb 22, 2020

@gaborbernat #1653 seems to work :D

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 26, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.6; see https://pypi.org/project/virtualenv/20.0.6/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-6-2020-02-26) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.
thanks

@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.