namespaced dependencies cannot be found in a custom PYTHONPATH #8505

kodawah · 2020-06-27T10:09:19Z

Describe the bug
Using the latest stable version of pip, I am trying to install twisted (which depends on zope.interface), on a custom PYTHONPATH location.

pip install --ignore-installed --prefix=$STACK_PREFIX twisted

This env variable is set to

$ echo $PYTHONPATH
/home/user/stack/lib/python3.6/site-packages:/home/user/stack/lib64/python2.7/site-packages/:/home/user/stack/lib/python2.7/site-packages

and both twisted and and zope.interface are installed correctly

$ find stack -name zope
stack/lib64/python2.7/site-packages/zope
$ ls stack/lib64/python2.7/site-packages/zope
interface
$ find stack -name twisted
stack/lib64/python2.7/site-packages/twisted

The package twisted can be loaded just fine -- zope.interface however cannot be found

Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twisted
>>> import zope.interface
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: No module named zope.interface

Following the advice from here pyinstaller/pyinstaller#502 if I manually add an empty __init__.py to stack/lib64/python2.7/site-packages/zope then the module becomes discoverable

$ touch stack/lib64/python2.7/site-packages/zope/__init__.py 
$ ls  stack/lib64/python2.7/site-packages/zope
__init__.py  interface
$ python
Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twisted
>>> import zope.interface
>>>

Expected behavior
I don't think it should be required to manually add the init file to the PYTHONPATH location. Even this is a namespaced module it should be discoverable in the custom PYTHONPATH location like any other module.

To Reproduce

set PYTHONPATH to a custom location
install twisted pip install --ignore-installed --prefix=$STACK_PREFIX twisted
import zope.interface in python

My Platform

$pip -V
pip 20.1.1 from /home/user/stack/lib/python2.7/site-packages/pip (python 2.7)
$ uname -a
Linux b582ede63e34 4.19.76-linuxkit pypi/warehouse#1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

uranusjr · 2020-06-27T14:24:09Z

Without looking deep into it, this feels like a Python import logic issue, since pip does not handle namespace packages specially in any way afaik.

kodawah · 2020-06-30T16:53:55Z

@uranusjr so the namespace packages just happen to work when they are not in a custom PYTHONPATH?

I'm not too familiar with python import logic, should I post a ticket somewhere else? Thanks for your advice

uranusjr · 2020-06-30T18:54:22Z

One important thing to do here is to reduce the example to minimise the moving parts in it, and determine whether pip or Python is at fault here. The reduced example should contain as little as possible, and be reliably reproducible from scratch.

For example, this is what I tried from reading your description:

$ python3 -m pip install twisted --target ./target
Collecting twisted
  Downloading Twisted-20.3.0-cp37-cp37m-macosx_10_6_intel.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 605 kB/s 
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting PyHamcrest!=1.10.0,>=1.9.0
  Downloading PyHamcrest-2.0.2-py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 135 kB/s 
Collecting incremental>=16.10.1
  Downloading incremental-17.5.0-py2.py3-none-any.whl (16 kB)
Collecting Automat>=0.3.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting hyperlink>=17.1.1
  Downloading hyperlink-19.0.0-py2.py3-none-any.whl (38 kB)
Collecting zope.interface
  Downloading zope.interface-5.1.0-cp37-cp37m-macosx_10_9_x86_64.whl (192 kB)
     |████████████████████████████████| 192 kB 383 kB/s
Collecting attrs>=19.2.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting idna>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 649 kB/s 
Collecting setuptools
  Downloading setuptools-47.3.1-py3-none-any.whl (582 kB)
     |████████████████████████████████| 582 kB 375 kB/s
Installing collected packages: constantly, PyHamcrest, incremental, attrs, six, Automat, idna, hyperlink, setuptools, zope.interface, twisted
Successfully installed Automat-20.2.0 PyHamcrest-2.0.2 attrs-19.3.0 constantly-15.1.0 hyperlink-19.0.0 idna-2.10 incremental-17.5.0 setuptools-47.3.1 six-1.15.0 twisted-20.3.0 zope.interface-5.1.0
$ PYTHONPATH=$PWD/target python3 -c 'import zope.interface; print(zope.interface)'
<module 'zope.interface' from '/.../target/zope/interface/__init__.py'>

Everything installs and imports correctly, so it seems that neither --target nor the namespace package is at fault here. This means that there is something in the setup not mentioned in your description that makes things fail. You need to hunt that thing down first by adding/tweaking things in the setup piece by piece.

kodawah · 2020-06-30T21:09:13Z

hi, thanks for your suggestion, I tried to minimize the moving parts in the "Steps to reproduce" section

set PYTHONPATH to a custom location
install twisted pip install --ignore-installed --prefix=$STACK_PREFIX twisted
import zope.interface in python

You may note that i used --ignore-installed and --prefix instead of --target - I tried your script and i can confirm it works. Here are the two command lines to hopefully reproduce the reported problem on your env as well

$ python3 -m pip install twisted --ignore-installed --prefix=./target
$ PYTHONPATH=$PWD/target python3 -c 'import zope.interface; print(zope.interface)'

which result in

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'zope'

jankatins · 2021-08-26T14:02:41Z

Not sure if this a new problem or just the same:

[15:59:24] λ  rm -rf /tmp/zope_problem


[15:59:44] λ  /home/jan.katins/.virtualenvs/pyinfra/bin/python3 -m pip --disable-pip-version-check install --target /tmp/zope_problem zope.interface zope.event
Collecting zope.interface
  Using cached zope.interface-5.4.0-cp39-cp39-manylinux2010_x86_64.whl (255 kB)
Collecting zope.event
  Using cached zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB)
Collecting setuptools
  Using cached setuptools-57.4.0-py3-none-any.whl (819 kB)
Installing collected packages: setuptools, zope.interface, zope.event
  WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  distutils: /tmp/pip-target-irw4lnin/lib64/python
  sysconfig: /tmp/pip-target-irw4lnin/lib/python
  WARNING: Additional context:
  user = False
  home = '/tmp/pip-target-irw4lnin'
  root = None
  prefix = None
Successfully installed setuptools-57.4.0 zope.event-4.5.0 zope.interface-5.4.0
WARNING: Target directory /tmp/zope_problem/zope already exists. Specify --upgrade to force replacement.


[15:59:55] λ  find  /tmp/zope_problem/ |grep /zope/ |grep "\.py$"
/tmp/zope_problem/zope/event/tests.py
/tmp/zope_problem/zope/event/classhandler.py
/tmp/zope_problem/zope/event/__init__.py


[16:00:04] λ  /home/jan.katins/.virtualenvs/pyinfra/bin/python3 -m pip --disable-pip-version-check install --target /tmp/zope_problem zope.interface zope.event --upgrade
Collecting zope.interface
  Using cached zope.interface-5.4.0-cp39-cp39-manylinux2010_x86_64.whl (255 kB)
Collecting zope.event
  Using cached zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB)
Collecting setuptools
  Using cached setuptools-57.4.0-py3-none-any.whl (819 kB)
Installing collected packages: setuptools, zope.interface, zope.event
  WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  distutils: /tmp/pip-target-3u8y7xfp/lib64/python
  sysconfig: /tmp/pip-target-3u8y7xfp/lib/python
  WARNING: Additional context:
  user = False
  home = '/tmp/pip-target-3u8y7xfp'
  root = None
  prefix = None
Successfully installed setuptools-57.4.0 zope.event-4.5.0 zope.interface-5.4.0


[16:00:17] λ  find  /tmp/zope_problem/ |grep /zope/ |grep "\.py$"
/tmp/zope_problem/zope/interface/tests/test_sorting.py
[...]
/tmp/zope_problem/zope/interface/_flatten.py

For whatever reason, only one namespace package content survives but the dist-info files are there for both:

[16:03:58] λ  ls  /tmp/zope_problem/ | grep zope
zope
zope.event-4.5.0.dist-info
zope.event-4.5.0-py3.6-nspkg.pth
zope.interface-5.4.0.dist-info
zope.interface-5.4.0-py3.9-nspkg.pth

Installing it without --target ... works as expected:

[16:06:18] λ  /home/jan.katins/.virtualenvs/pyinfra/bin/python3 -m pip --disable-pip-version-check install  zope.event zope.interface --upgrade
Requirement already satisfied: zope.event in ./.virtualenvs/pyinfra/lib/python3.9/site-packages (4.5.0)
Requirement already satisfied: zope.interface in ./.virtualenvs/pyinfra/lib/python3.9/site-packages (5.4.0)
Requirement already satisfied: setuptools in ./.virtualenvs/pyinfra/lib/python3.9/site-packages (from zope.event) (53.0.0)

~  🐍 v3.9.6 (venv:pyinfra)
[16:06:41] λ  ls  ./.virtualenvs/pyinfra/lib/python3.9/site-packages/zope
event  interface

uranusjr · 2021-08-27T18:07:07Z

The original report was specific to Python 2, so I'm going to close this. (It's a limitation in Python's import system and not related to pip.) @jankatins I am not sure if your issue is the same and you should probably open a new issue.

di transferred this issue from pypi/warehouse Jun 27, 2020

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Jun 27, 2020

FRidh mentioned this issue Jul 4, 2020

python2.7-python-otr-1.2.0: Test failure: ImportError: No module named zope.interface NixOS/nixpkgs#74619

Closed

jankatins mentioned this issue Aug 26, 2021

zope.event is not packaged properly when installed together with zope.interface linkedin/shiv#187

Open

uranusjr closed this as completed Aug 27, 2021

uranusjr added resolution: invalid Invalid issue/PR and removed S: needs triage Issues/PRs that need to be triaged labels Aug 27, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

namespaced dependencies cannot be found in a custom PYTHONPATH #8505

namespaced dependencies cannot be found in a custom PYTHONPATH #8505

kodawah commented Jun 27, 2020

uranusjr commented Jun 27, 2020 •

edited

Loading

kodawah commented Jun 30, 2020

uranusjr commented Jun 30, 2020

kodawah commented Jun 30, 2020

jankatins commented Aug 26, 2021 •

edited

Loading

uranusjr commented Aug 27, 2021

namespaced dependencies cannot be found in a custom PYTHONPATH #8505

namespaced dependencies cannot be found in a custom PYTHONPATH #8505

Comments

kodawah commented Jun 27, 2020

uranusjr commented Jun 27, 2020 • edited Loading

kodawah commented Jun 30, 2020

uranusjr commented Jun 30, 2020

kodawah commented Jun 30, 2020

jankatins commented Aug 26, 2021 • edited Loading

uranusjr commented Aug 27, 2021

uranusjr commented Jun 27, 2020 •

edited

Loading

jankatins commented Aug 26, 2021 •

edited

Loading