Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[20.0.5] pypy 2.x site initialization strangeness #1652

Closed
asottile opened this issue Feb 21, 2020 · 5 comments · Fixed by #1656
Closed

[20.0.5] pypy 2.x site initialization strangeness #1652

asottile opened this issue Feb 21, 2020 · 5 comments · Fixed by #1656

Comments

@asottile
Copy link
Contributor

This is affecting builds -- it makes pip fail in a strange way when invoking shlex:

import shlex

__import__('pprint').pprint(__import__('sys').path)
print(repr(shlex.shlex(posix=True).wordchars))
print(shlex.split(u'-e .'))

virtualenv creation:

$ virtualenv venv -p pypy -vvv
51 setup logging to NOTSET [DEBUG report:43]
71 find interpreter for spec PythonSpec(implementation=pypy) [INFO builtin:43]
71 discover system for PythonInfo(spec=CPython3.6.9.final.0-64, exe=/home/asottile/opt/venv/bin/python3, platform=linux, version='3.6.9 (default, Nov  7 2019, 10:44:02) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) in /usr [DEBUG py_info:336]
71 filesystem is case-sensitive [DEBUG info:28]
72 Attempting to acquire lock 140540686132672 on /home/asottile/.local/share/virtualenv/py-info/20.0.5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:270]
72 Lock 140540686132672 acquired on /home/asottile/.local/share/virtualenv/py-info/20.0.5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:274]
72 get PythonInfo from /home/asottile/.local/share/virtualenv/py-info/20.0.5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.json for /usr/bin/python3.6 [DEBUG cached_py_info:72]
72 Attempting to release lock 140540686132672 on /home/asottile/.local/share/virtualenv/py-info/20.0.5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [DEBUG filelock:315]
73 Lock 140540686132672 released on /home/asottile/.local/share/virtualenv/py-info/20.0.5/30295c5bec572e859485b1ffa5e89b8b3e2022ef6e3e739c1ac40f143a557caf.lock [INFO filelock:318]
73 proposed PythonInfo(spec=CPython3.6.9.final.0-64, system=/usr/bin/python3.6, exe=/home/asottile/opt/venv/bin/python3, platform=linux, version='3.6.9 (default, Nov  7 2019, 10:44:02) \n[GCC 8.3.0]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:49]
73 discover PATH[0]=/home/asottile/bin [DEBUG builtin:76]
73 Attempting to acquire lock 140540686132448 on /home/asottile/.local/share/virtualenv/py-info/20.0.5/322f70e2973bbcb5ad96e04ecb7f3ba9de600fafc76f86be6088dabb2fcc505e.lock [DEBUG filelock:270]
73 Lock 140540686132448 acquired on /home/asottile/.local/share/virtualenv/py-info/20.0.5/322f70e2973bbcb5ad96e04ecb7f3ba9de600fafc76f86be6088dabb2fcc505e.lock [INFO filelock:274]
73 get PythonInfo from /home/asottile/.local/share/virtualenv/py-info/20.0.5/322f70e2973bbcb5ad96e04ecb7f3ba9de600fafc76f86be6088dabb2fcc505e.json for /home/asottile/bin/pypy [DEBUG cached_py_info:72]
74 Attempting to release lock 140540686132448 on /home/asottile/.local/share/virtualenv/py-info/20.0.5/322f70e2973bbcb5ad96e04ecb7f3ba9de600fafc76f86be6088dabb2fcc505e.lock [DEBUG filelock:315]
74 Lock 140540686132448 released on /home/asottile/.local/share/virtualenv/py-info/20.0.5/322f70e2973bbcb5ad96e04ecb7f3ba9de600fafc76f86be6088dabb2fcc505e.lock [INFO filelock:318]
74 proposed PathPythonInfo(spec=PyPy2.7.13.final.42-64, exe=/home/asottile/bin/pypy, platform=linux2, version='2.7.13 (724f1a7d62e8, Dec 23 2019, 15:36:24)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]', encoding_fs_io=ANSI_X3.4-1968-None) [INFO builtin:49]
74 accepted PathPythonInfo(spec=PyPy2.7.13.final.42-64, exe=/home/asottile/bin/pypy, platform=linux2, version='2.7.13 (724f1a7d62e8, Dec 23 2019, 15:36:24)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]', encoding_fs_io=ANSI_X3.4-1968-None) [DEBUG builtin:51]
93 create virtual environment via PyPy2Posix(dest=/tmp/z/venv, clear=False, global=False) [INFO session:51]
93 create folder /tmp/z/venv/bin [DEBUG _sync:20]
93 create folder /tmp/z/venv/lib-python/2.7 [DEBUG _sync:20]
94 create folder /tmp/z/venv/lib_pypy [DEBUG _sync:20]
94 create folder /tmp/z/venv/site-packages [DEBUG _sync:20]
94 write /tmp/z/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
94 	home = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
94 	implementation = PyPy [DEBUG pyenv_cfg:38]
94 	version_info = 2.7.13.final.42 [DEBUG pyenv_cfg:38]
94 	virtualenv = 20.0.5 [DEBUG pyenv_cfg:38]
94 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
94 	base-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
94 	base-exec-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
94 	base-executable = /home/asottile/bin/pypy [DEBUG pyenv_cfg:38]
94 copy /home/asottile/bin/pypy to /tmp/z/venv/bin/pypy [DEBUG _sync:47]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/os.py to /tmp/z/venv/lib-python/2.7/os.py [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/os.pyc to /tmp/z/venv/lib-python/2.7/os.pyc [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/copy_reg.py to /tmp/z/venv/lib-python/2.7/copy_reg.py [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/copy_reg.pyc to /tmp/z/venv/lib-python/2.7/copy_reg.pyc [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/genericpath.py to /tmp/z/venv/lib-python/2.7/genericpath.py [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/genericpath.pyc to /tmp/z/venv/lib-python/2.7/genericpath.pyc [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/linecache.py to /tmp/z/venv/lib-python/2.7/linecache.py [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/linecache.pyc to /tmp/z/venv/lib-python/2.7/linecache.pyc [DEBUG _sync:39]
95 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/stat.py to /tmp/z/venv/lib-python/2.7/stat.py [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/stat.pyc to /tmp/z/venv/lib-python/2.7/stat.pyc [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/UserDict.py to /tmp/z/venv/lib-python/2.7/UserDict.py [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/UserDict.pyc to /tmp/z/venv/lib-python/2.7/UserDict.pyc [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/warnings.py to /tmp/z/venv/lib-python/2.7/warnings.py [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/warnings.pyc to /tmp/z/venv/lib-python/2.7/warnings.pyc [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/posixpath.py to /tmp/z/venv/lib-python/2.7/posixpath.py [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/posixpath.pyc to /tmp/z/venv/lib-python/2.7/posixpath.pyc [DEBUG _sync:39]
96 symlink /home/asottile/opt/pypy2.7-v7.3.0-linux64/bin/libpypy-c.so to /tmp/z/venv/bin/libpypy-c.so [DEBUG _sync:39]
96 symlink directory /home/asottile/opt/pypy2.7-v7.3.0-linux64/include to /tmp/z/venv/include [DEBUG _sync:39]
96 symlink directory /home/asottile/opt/pypy2.7-v7.3.0-linux64/lib to /tmp/z/venv/lib [DEBUG _sync:39]
97 ============================== target debug ============================== [DEBUG session:53]
97 debug via /tmp/z/venv/bin/pypy /home/asottile/opt/venv/lib/python3.6/site-packages/virtualenv/create/debug.py [DEBUG creator:191]
97 {
  "sys": {
    "executable": "/tmp/z/venv/bin/pypy",
    "_base_executable": null,
    "prefix": "/tmp/z/venv",
    "base_prefix": "/home/asottile/opt/pypy2.7-v7.3.0-linux64",
    "real_prefix": "/home/asottile/opt/pypy2.7-v7.3.0-linux64",
    "exec_prefix": "/tmp/z/venv",
    "base_exec_prefix": "/home/asottile/opt/pypy2.7-v7.3.0-linux64",
    "path": [
      "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy/__extensions__",
      "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy",
      "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7",
      "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/lib-tk",
      "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/plat-linux2",
      "/tmp/z/venv/site-packages"
    ],
    "meta_path": [],
    "fs_encoding": "UTF-8",
    "io_encoding": null
  },
  "version": "2.7.13 (724f1a7d62e8, Dec 23 2019, 15:36:24)\n[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]",
  "os": "<module 'os' from '/tmp/z/venv/lib-python/2.7/os.pyc'>",
  "site": "<module 'site' from '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/site.pyc'>",
  "datetime": "<module 'datetime' from '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy/datetime.pyc'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/json/__init__.pyc'>"
} [DEBUG session:54]
157 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/asottile/.local/share/virtualenv/seed-v1) [INFO session:58]
157 Attempting to acquire lock 140540685686712 on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:270]
157 Lock 140540685686712 acquired on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:274]
159 install setuptools from wheel /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels/setuptools-44.0.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
159 install pip from wheel /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
159 install wheel from wheel /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
159 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info to /tmp/z/venv/site-packages/setuptools-44.0.0.dist-info [DEBUG _sync:47]
160 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /tmp/z/venv/site-packages/pip [DEBUG _sync:47]
161 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /tmp/z/venv/site-packages/wheel [DEBUG _sync:47]
163 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/pkg_resources to /tmp/z/venv/site-packages/pkg_resources [DEBUG _sync:47]
166 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /tmp/z/venv/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:47]
169 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info.virtualenv to /tmp/z/venv/site-packages/wheel-0.34.2.dist-info.virtualenv [DEBUG _sync:47]
179 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/easy_install.py to /tmp/z/venv/site-packages/easy_install.py [DEBUG _sync:47]
180 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info.virtualenv to /tmp/z/venv/site-packages/setuptools-44.0.0.dist-info.virtualenv [DEBUG _sync:47]
182 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools to /tmp/z/venv/site-packages/setuptools [DEBUG _sync:47]
205 changing mode of /tmp/z/venv/bin/wheel to 755 [INFO util:566]
205 changing mode of /tmp/z/venv/bin/wheel3 to 755 [INFO util:566]
206 changing mode of /tmp/z/venv/bin/wheel-3.6 to 755 [INFO util:566]
206 generated console scripts wheel-3.6 wheel3 wheel [DEBUG base:48]
222 changing mode of /tmp/z/venv/bin/easy_install to 755 [INFO util:566]
223 changing mode of /tmp/z/venv/bin/easy_install-3.6 to 755 [INFO util:566]
223 changing mode of /tmp/z/venv/bin/easy_install3 to 755 [INFO util:566]
223 generated console scripts easy_install-3.6 easy_install easy_install3 [DEBUG base:48]
241 copy /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info.virtualenv to /tmp/z/venv/site-packages/pip-20.0.2.dist-info.virtualenv [DEBUG _sync:47]
242 copy directory /home/asottile/.local/share/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /tmp/z/venv/site-packages/pip-20.0.2.dist-info [DEBUG _sync:47]
243 changing mode of /tmp/z/venv/bin/pip to 755 [INFO util:566]
243 changing mode of /tmp/z/venv/bin/pip-3.6 to 755 [INFO util:566]
243 changing mode of /tmp/z/venv/bin/pip3 to 755 [INFO util:566]
243 generated console scripts pip pip3 pip-3.6 [DEBUG base:48]
243 Attempting to release lock 140540685686712 on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:315]
243 Lock 140540685686712 released on /home/asottile/.local/share/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:318]
243 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
245 write /tmp/z/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
245 	home = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
245 	implementation = PyPy [DEBUG pyenv_cfg:38]
245 	version_info = 2.7.13.final.42 [DEBUG pyenv_cfg:38]
245 	virtualenv = 20.0.5 [DEBUG pyenv_cfg:38]
245 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
245 	base-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
245 	base-exec-prefix = /home/asottile/opt/pypy2.7-v7.3.0-linux64 [DEBUG pyenv_cfg:38]
245 	base-executable = /home/asottile/bin/pypy [DEBUG pyenv_cfg:38]
245 created virtual environment PyPy2.7.13.final.42-64 in 194ms
  creator PyPy2Posix(dest=/tmp/z/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/asottile/.local/share/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:21]

script difference:

$ ./venv/bin/pypy t.py
['/tmp/z',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy/__extensions__',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/lib-tk',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/plat-linux2',
 '/tmp/z/venv/site-packages']
'abcdfeghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd8\xd9\xda\xdb\xdc\xdd\xde'
Traceback (most recent call last):
  File "t.py", line 5, in <module>
    print(shlex.split(u'-e .'))
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 279, in split
    return list(lex)
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 150, in read_token
    elif nextchar in self.wordchars:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 63: ordinal not in range(128)
$ pypy t.py
['/tmp/z',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy/__extensions__',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/lib-tk',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/plat-linux2',
 '/home/asottile/opt/pypy2.7-v7.3.0-linux64/site-packages']
'abcdfeghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd8\xd9\xda\xdb\xdc\xdd\xde'
['-e', '.']

the actual failure I'm seeing:

$ echo '-e . ' > requirements-dev.txt
$ echo 'import setuptools; setuptools.setup()' > setup.py
$ ./venv/bin/pip install -r requirements-dev.txt 
DEPRECATION: A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/z/venv/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/tmp/z/venv/site-packages/pip/_internal/commands/install.py", line 301, in run
    wheel_cache
  File "/tmp/z/venv/site-packages/pip/_internal/cli/req_command.py", line 270, in populate_requirement_set
    use_pep517=options.use_pep517):
  File "/tmp/z/venv/site-packages/pip/_internal/req/req_file.py", line 137, in parse_requirements
    for parsed_line in parser.parse(filename, constraint):
  File "/tmp/z/venv/site-packages/pip/_internal/req/req_file.py", line 282, in parse
    for line in self._parse_and_recurse(filename, constraint):
  File "/tmp/z/venv/site-packages/pip/_internal/req/req_file.py", line 287, in _parse_and_recurse
    for line in self._parse_file(filename, constraint):
  File "/tmp/z/venv/site-packages/pip/_internal/req/req_file.py", line 329, in _parse_file
    args_str, opts = self._line_parser(line)
  File "/tmp/z/venv/site-packages/pip/_internal/req/req_file.py", line 365, in parse_line
    shlex.split(options_str), defaults)  # type: ignore
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 279, in split
    return list(lex)
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib-python/2.7/shlex.py", line 150, in read_token
    elif nextchar in self.wordchars:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 63: ordinal not in range(128)

Investigation:

This seems to be the main difference:

$ venv/bin/python -c 'import cStringIO; print(cStringIO)'
<module 'cStringIO' from '/home/asottile/opt/pypy2.7-v7.3.0-linux64/lib_pypy/cStringIO.pyc'>
$ pypy -c 'import cStringIO; print(cStringIO)'
<module 'cStringIO' (built-in)>

leading to this difference:

$ pypy -c 'import cStringIO; print(repr(cStringIO.StringIO(u"-e n").read()))'
'-e n'
$ venv/bin/pypy -c 'import cStringIO; print(repr(cStringIO.StringIO(u"-e n").read()))'
u'-e n'
@gaborbernat
Copy link
Contributor

Seems strange, did you open an issue with pypy, could help to understand what's the difference between the two modules 🤔

@gaborbernat
Copy link
Contributor

gaborbernat commented Feb 22, 2020

Some investigation, seems the meta path is different; but the paths otherwise the same.

pypy -c 'import sys; print("meta paths"); print("\n".join(repr(i) for i in sys.meta_path)); print(""); print("path hooks"); print("\n".join(repr(i) for i in sys.path_hooks)); print("\npath"); print("\n".join(repr(i) for i in sys.path));print("\ntest import"); import cStringIO; print(cStringIO);'                                                                                                                    167ms  Sat 22 Feb 08:59:53 2020
meta paths


path hooks
<type 'zipimporter'>

path
''
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib_pypy/__extensions__'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib_pypy'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/lib-tk'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/plat-darwin'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/plat-mac'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/plat-mac/lib-scriptpackages'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/site-packages'

test import
<module 'cStringIO' (built-in)>
venv/bin/pypy -c 'import sys; print("meta paths"); print("\n".join(repr(i) for i in sys.meta_path)); print(""); print("path hooks"); print("\n".join(repr(i) for i in sys.path_hooks)); print("\npath"); print("\n".join(repr(i) for i in sys.path));print("\ntest import"); import cStringIO; print(cStringIO);'                                                                                                               231ms  Sat 22 Feb 08:59:58 2020
meta paths
<setuptools.extern.VendorImporter instance at 0x00000001059e9960>
<setuptools._vendor.six._SixMetaPathImporter object at 0x00000001059e1d70>
<pkg_resources.extern.VendorImporter instance at 0x000000010612b1e0>
<pkg_resources._vendor.six._SixMetaPathImporter object at 0x000000010612db08>

path hooks
<type 'zipimporter'>

path
''
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib_pypy/__extensions__'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib_pypy'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/lib-tk'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/plat-darwin'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/plat-mac'
'/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib-python/2.7/plat-mac/lib-scriptpackages'
'/Users/bgabor8/git/github/virtualenv/venv/site-packages'

test import
<module 'cStringIO' from '/Users/bgabor8/.pyenv/versions/pypy2.7-7.3.0/lib_pypy/cStringIO.pyc'>

Opened upstream issue https://foss.heptapod.net/pypy/pypy/issues/3174 to understand the difference.

@gaborbernat
Copy link
Contributor

gaborbernat commented Feb 22, 2020

Looking at that file:

#
# StringIO-based cStringIO implementation.
#

# Note that PyPy also contains a built-in module 'cStringIO' which will hide
# this one if compiled in.

from StringIO import *
from StringIO import __doc__

class StringIO(StringIO):
    def reset(self):
        """
        reset() -- Reset the file position to the beginning
        """
        self.seek(0, 0)

So I guess the question remains why is the built-in not discovered within the venv but is outside. Side question would be why the lib-pypy behaves differently than the builtin 🤷‍♂

@mattip
Copy link
Contributor

mattip commented Feb 23, 2020

There is this in the pypy 2.7 code:

            # a "real builtin module xx" shadows every file "xx.py" there
            # could possibly be; a "pseudo-extension module" does not, and
            # is only loaded at the point in sys.path where we find
            # '.../lib_pypy/__extensions__'.
            if modulename in space.MODULES_THAT_ALWAYS_SHADOW:
                return delayed_builtin

where MODULES_THAT_ALWAYS_SHADOW is defined

    # These modules are treated like CPython treats built-in modules,
    # i.e. they always shadow any xx.py.  The other modules are treated
    # like CPython treats extension modules, and are loaded in sys.path
    # order by the fake entry '.../lib_pypy/__extensions__'.
    MODULES_THAT_ALWAYS_SHADOW = dict.fromkeys([
        '__builtin__', '__pypy__', '_ast', '_codecs', '_sre', '_warnings',
        '_weakref', 'errno', 'exceptions', 'gc', 'imp', 'marshal',
        'posix', 'nt', 'pwd', 'signal', 'sys', 'thread', 'zipimport',
    ], None)

So by changing sys.path[0] which wass path/to/lib_pypy/__extensions__ and is changed to path/in/old/pypy/lib_pypy/__extensions__ in rewrite_standard_library_sys_path virtualenv breaks this mechanism. If I modify that to skip lib_pypy/__extensions__ like it does exe_dir, then cStringIO is always the built-in one.

So in conclusion:

  • the PyPy mechanism is too brittle
  • for backward compatibility virtualenv might want to change something
  • going forward, PyPy should change to be less brittle.

gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Feb 23, 2020
Resolves pypa#1652.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit to gaborbernat/virtualenv that referenced this issue Feb 23, 2020
Resolves pypa#1652.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
gaborbernat added a commit that referenced this issue Feb 23, 2020
* fix pypy2 builtins are imported as stdlib modules

Resolves #1652.

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>

* test failure

Signed-off-by: Bernat Gabor <bgabor8@bloomberg.net>
@gaborbernat
Copy link
Contributor

Hello, a fix for this issue has been released via virtualenv 20.0.6; see https://pypi.org/project/virtualenv/20.0.6/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-6-2020-02-26) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.
thanks

@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants