Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

version 1.6.9 fails with null module __file__ attribute #756

Closed
gshuflin opened this issue Aug 13, 2019 · 1 comment · Fixed by #757

Comments

@gshuflin
Copy link

commented Aug 13, 2019

On my arch linux system, after the pants upgrade to version 1.6.9, a number of pieces of pants functionality that make use of pex started failing with the following error message:

                     Traceback (most recent call last):
                       File "/home/gregs/code/pants/.pants.d/test/pytest-prep/CPython-3.7.3/7c81bb6ec656c4064b548c392d519f0695716e81/.bootstrap/pex/pex.py", line 356, in execute
                         self.patch_sys(pex_inherit_path)
                       File "/home/gregs/code/pants/.pants.d/test/pytest-prep/CPython-3.7.3/7c81bb6ec656c4064b548c392d519f0695716e81/.bootstrap/pex/pex.py", line 285, in patch_sys
                         new_sys_path, new_sys_path_importer_cache, new_sys_modules = self.minimum_sys(inherit_path)
                       File "/home/gregs/code/pants/.pants.d/test/pytest-prep/CPython-3.7.3/7c81bb6ec656c4064b548c392d519f0695716e81/.bootstrap/pex/pex.py", line 259, in minimum_sys
                         sys_modules = cls.minimum_sys_modules(site_libs)
                       File "/home/gregs/code/pants/.pants.d/test/pytest-prep/CPython-3.7.3/7c81bb6ec656c4064b548c392d519f0695716e81/.bootstrap/pex/pex.py", line 179, in minimum_sys_modules
                         if cls._tainted_path(module_file, site_libs):
                       File "/home/gregs/code/pants/.pants.d/test/pytest-prep/CPython-3.7.3/7c81bb6ec656c4064b548c392d519f0695716e81/.bootstrap/pex/pex.py", line 157, in _tainted_path
                         paths = frozenset([path, os.path.realpath(path)])
                       File "/home/gregs/code/pants/build-support/pants_dev_deps.py37.venv/lib/python3.7/posixpath.py", line 394, in realpath
                         filename = os.fspath(filename)
                     TypeError: expected str, bytes or os.PathLike object, not NoneType

It looks like what's going on here is that https://github.com/pantsbuild/pex/blob/master/pex/pex.py#L178 is iterating over every module and extracting the __file__ attribute of that mdoule. However, some module on my system seems to have a None value for that attribute, which is causing code downstream of that line to fail. It looks like the commit b5f7e41 might be relevant.

@jsirois

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

For posterity, this was a specific issue with namespace packages in Python 3.7+:

$ python3.6 -mvenv /tmp/twitter-3.6
$ python3.7 -mvenv /tmp/twitter-3.7
$ /tmp/twitter-3.6/bin/pip install -U pip twitter.common.collections twitter.common.dirutil
Collecting pip
  Using cached https://files.pythonhosted.org/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl
Collecting twitter.common.collections
Collecting twitter.common.dirutil
Collecting twitter.common.lang==0.3.11 (from twitter.common.collections)
Installing collected packages: pip, twitter.common.lang, twitter.common.collections, twitter.common.dirutil
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.2.2 twitter.common.collections-0.3.11 twitter.common.dirutil-0.3.11 twitter.common.lang-0.3.11
$ /tmp/twitter-3.7/bin/pip install -U pip twitter.common.collections twitter.common.dirutil
Collecting pip
  Using cached https://files.pythonhosted.org/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl
Collecting twitter.common.collections
Collecting twitter.common.dirutil
Collecting twitter.common.lang==0.3.11 (from twitter.common.collections)
Installing collected packages: pip, twitter.common.lang, twitter.common.collections, twitter.common.dirutil
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.2.2 twitter.common.collections-0.3.11 twitter.common.dirutil-0.3.11 twitter.common.lang-0.3.11
$ /tmp/twitter-3.6/bin/python
Python 3.6.9 (default, Jul  8 2019, 20:56:32) 
[GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import twitter
>>> print(twitter.__file__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'twitter' has no attribute '__file__'
>>> 
$ /tmp/twitter-3.7/bin/python
Python 3.7.4 (default, Jul 16 2019, 07:12:58) 
[GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import twitter
>>> print(twitter.__file__)
None
>>> 

In particular, this was a CPython cleanup noted here: https://docs.python.org/3/whatsnew/3.7.html#other-cpython-implementation-changes as a bug fix for https://bugs.python.org/issue32305.

@jsirois jsirois closed this in #757 Aug 19, 2019

jsirois added a commit that referenced this issue Aug 19, 2019
@jsirois jsirois referenced this issue Aug 19, 2019
1 of 1 task complete

@jsirois jsirois removed the in progress label Aug 19, 2019

jsirois added a commit to jsirois/pants that referenced this issue Aug 20, 2019
Upgrade to Pex 1.6.10.
This pulls in a fix for the pexes Pants generates when run under CPython
3.7+ and namespace packages are involved.

See: pantsbuild/pex#756

Before the fix both our own tool pexes and pexes we generated for end
users could fail like so:
```
$ ./pants --python-setup-interpreter-constraints="['CPython==3.7.*']" \
  test.pytest \
  --cache-ignore \
  tests/python/pants_test/backend/graph_info/tasks:cloc
...
17:16:00 00:26   [test]
...
17:16:00 00:26     [pytest]
                   Invalidated 1 target.
                   scrubbed PYTHONPATH=/home/jsirois/dev/pantsbuild/pants/src/python: from pytest environment
17:16:00 00:26       [run]
                     Traceback (most recent call last):
...
                       File "/home/jsirois/dev/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.7.4/e307788111116c190fe04c233d334b12cdc63f77/.bootstrap/pex/pex.py", line 179, in minimum_sys_modules
                         if cls._tainted_path(module_file, site_libs):
                       File "/home/jsirois/dev/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.7.4/e307788111116c190fe04c233d334b12cdc63f77/.bootstrap/pex/pex.py", line 157, in _tainted_path
                         paths = frozenset([path, os.path.realpath(path)])
                       File "/home/jsirois/dev/pantsbuild/jsirois-pants/build-support/pants_dev_deps.py37.venv/lib/python3.7/posixpath.py", line 394, in realpath
                         filename = os.fspath(filename)
                     TypeError: expected str, bytes or os.PathLike object, not NoneType
...
```
jsirois added a commit to pantsbuild/pants that referenced this issue Aug 20, 2019
Upgrade to Pex 1.6.10. (#8187)
This pulls in a fix for the pexes Pants generates when run under CPython
3.7+ and namespace packages are involved.

See: pantsbuild/pex#756

Before the fix both our own tool pexes and pexes we generated for end
users could fail like so:
```
$ ./pants --python-setup-interpreter-constraints="['CPython==3.7.*']" \
  test.pytest \
  --cache-ignore \
  tests/python/pants_test/backend/graph_info/tasks:cloc
...
17:16:00 00:26   [test]
...
17:16:00 00:26     [pytest]
                   Invalidated 1 target.
                   scrubbed PYTHONPATH=/home/jsirois/dev/pantsbuild/pants/src/python: from pytest environment
17:16:00 00:26       [run]
                     Traceback (most recent call last):
...
                       File "/home/jsirois/dev/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.7.4/e307788111116c190fe04c233d334b12cdc63f77/.bootstrap/pex/pex.py", line 179, in minimum_sys_modules
                         if cls._tainted_path(module_file, site_libs):
                       File "/home/jsirois/dev/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.7.4/e307788111116c190fe04c233d334b12cdc63f77/.bootstrap/pex/pex.py", line 157, in _tainted_path
                         paths = frozenset([path, os.path.realpath(path)])
                       File "/home/jsirois/dev/pantsbuild/jsirois-pants/build-support/pants_dev_deps.py37.venv/lib/python3.7/posixpath.py", line 394, in realpath
                         filename = os.fspath(filename)
                     TypeError: expected str, bytes or os.PathLike object, not NoneType
...
```
patliu85 added a commit to twitter/pants that referenced this issue Aug 20, 2019
Upgrade to Pex 1.6.10. (pantsbuild#8187)
This pulls in a fix for the pexes Pants generates when run under CPython
3.7+ and namespace packages are involved.

See: pantsbuild/pex#756

Before the fix both our own tool pexes and pexes we generated for end
users could fail like so:
```
$ ./pants --python-setup-interpreter-constraints="['CPython==3.7.*']" \
  test.pytest \
  --cache-ignore \
  tests/python/pants_test/backend/graph_info/tasks:cloc
...
17:16:00 00:26   [test]
...
17:16:00 00:26     [pytest]
                   Invalidated 1 target.
                   scrubbed PYTHONPATH=/home/jsirois/dev/pantsbuild/pants/src/python: from pytest environment
17:16:00 00:26       [run]
                     Traceback (most recent call last):
...
                       File "/home/jsirois/dev/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.7.4/e307788111116c190fe04c233d334b12cdc63f77/.bootstrap/pex/pex.py", line 179, in minimum_sys_modules
                         if cls._tainted_path(module_file, site_libs):
                       File "/home/jsirois/dev/pantsbuild/pants/.pants.d/test/pytest-prep/CPython-3.7.4/e307788111116c190fe04c233d334b12cdc63f77/.bootstrap/pex/pex.py", line 157, in _tainted_path
                         paths = frozenset([path, os.path.realpath(path)])
                       File "/home/jsirois/dev/pantsbuild/jsirois-pants/build-support/pants_dev_deps.py37.venv/lib/python3.7/posixpath.py", line 394, in realpath
                         filename = os.fspath(filename)
                     TypeError: expected str, bytes or os.PathLike object, not NoneType
...
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.