pip install --editable and pip install clash for namespace packages #3

Closed
mitsuhiko opened this Issue Mar 14, 2011 · 41 comments

Comments

Projects
None yet
Contributor

mitsuhiko commented Mar 14, 2011

A namespace package installed editable and another one installed regularly don't work well together.

Contributor

harobed commented May 2, 2011

Same issue here :(

Contributor

harobed commented May 3, 2011

Contributor

carljm commented Jul 26, 2011

The root of the problem here is that setuptools includes two methods for making namespace packages work: the __init__.py method (which is documented, and usable by humans), and the ...-nspkg.pth file method, which is only used with --single-version-externally-managed (which pip uses). The two methods are not compatible with each other.

After some discussion with mitsuhiko and others on IRC, it seems the best (partial) solution for pip is to have "pip install -e" add a modified version of the standard setuptools ...nspkg.pth file for each develop-installed package. (The necessary modifications are to use the develop egg-link path instead of sitedir, and to skip the check for init.py entirely, since a develop-installed namespace package will have an init.py). This means pip will at least be compatible with itself (pip-installed and pip-editable-installed namespace packages will work together). Pip-installed and setuptools/distribute-installed namespace packages will not be compatible.

Contributor

embray commented Feb 16, 2012

I recently got bitten by this too. Another problem caused by it is that since an __init__.py isn't installed for the namespace package (assuming you've only installed one package in that namespace, and it was installed --single-version-externally managed) it breaks nose's module search. Maybe that's nose's fault, but I still think it's reasonable to expect if a directory doesn't contain an __init__.py that it's not a Python package.

I think pip should somehow just kill the nspkg.pth thing altogether and install the __init__.py. As long as the package itself is designed correctly (i.e. the __init__.py is empty except for extend_path/declare_namespace) it shouldn't be a problem. --single-version-externally-managed was designed with system packagers in mind, but they're not going to be using pip in the first place. So I wonder if there's some way to disable this behavior altogether without being too intrusive.

tseaver commented Mar 25, 2012

+1 for switching away from '--single-version-externally-managed': that option is not intended for pip's use cases at all. If pip can't do at least as good a job installing stuff as easy_install, what is its purpose?

Contributor

carljm commented Mar 26, 2012

Switching away from --single-version-externally-managed entirely is not going to happen; flat installs are a feature. Switching away from it in the case of namespace packages only is a possibility I'd consider, to avoid these issues. I don't like it, but there doesn't seem to be any great option, given the inherent incompatibility between the two types of namespace package support built in to setuptools/distribute.

Saying that --single-version-externally-managed is "not intended for pip's use cases" is somewhere between wrong and a red herring. The flag is intended to allow flat single-version installations, managed by some tool other than easy_install. That's precisely what pip uses it for; the fact that the setuptools author originally (and incorrectly) thought that only system packagers would be interested in such a feature is irrelevant.

The breakage here is an inherent bug in the technique setuptools uses for namespace packages with that flag, and the bug is just as present when the flag is used by its originally intended audience, system packagers (I first saw the bug in interaction between a "setup.py develop" installed package and a system-package-installed namespace package).

I'm experiencing this bug as well. I guess this is one of those unresolvable bugs that's here to stay. Oh well.

Contributor

carljm commented Apr 16, 2012

I would accept a patch to avoid --single-version-externally-managed when a namespace package is involved, which I think would solve this issue. I don't currently have any plans to write that patch.

k4ml commented May 19, 2012

Why not an option that we can pass to pip install not to use --single-version-externally-managed ? I test by commenting out that in ./pip/req.py:568 and all packages now installed into the egg dir, just like easy_install does. Sometime I want to do this if I used both pip and easy_install so that my site-packages does not mix with some packages install flat and some not.

Contributor

carljm commented May 19, 2012

I don't see a problem with a --egg option to opt out of --single-version-externally-managed.

k4ml commented May 20, 2012

k4ml/pip@93cd6b1

Can someone comment if I'm moving into the right direction ? This is my first time hacking on pip code so this is based on my few minutes glimpse into part of it, just to get pip install --egg somepackages install everything into single egg dir.

Contributor

carljm commented May 21, 2012

Can someone comment if I'm moving into the right direction ?

Looks good to me, just needs a test. Tests are mostly high-level
integration tests using ScriptTest, should be easy to write one based on
existing examples. In this case you'll want to just test that installing
a sample package results in a .egg file, not a flat installation, in
site-packages. You should install a package from the local filesystem,
not the network: tests/packages/FSPkg will probably work fine. Adding
the test to tests/test_basic.py is ok.

Thanks!

k4ml commented May 22, 2012

Pull requests - #541

I'm also thinking of making it possible to set this flag in pip.conf. What you think about that ?

Contributor

carljm commented May 22, 2012

I'm also thinking of making it possible to set this flag in pip.conf. What you think about that ?

Let's discuss on the pull request.

Contributor

carljm commented May 24, 2012

Merged pull request #541, which provides one workaround. Thanks @k4ml!

Contributor

qwcode commented Apr 24, 2014

Notice: the --egg option, which was added to fix this issue may potentially be removed, with no alternative workaround offered. See discussion in #1749

Member

Ivoz commented May 13, 2014

Here is a script to actually reproduce this issue

https://gist.github.com/Ivoz/d9bff05069e0ec53e6ea

@dmp42 dmp42 referenced this issue in docker/docker-registry May 29, 2014

Closed

Next #395

i'm not a huge fan of the --egg solution, but there needs to be a less cumbersome workaround for this.

without a workaround, while developing, I can't use --editable once and then just edit and test. every time I save my code I've got to run pip install -I --no-deps . which gets really old and is fragile (read: sometimes I forget).

part of the problem with the way this entire issue manifests is that it is silent. you just can't import a module even though it is installed and pip tells you it is there.

it would be a positive step just to make pip complain if trying to install editable packages as part of a namespace where a non editable package in that namespace is already installed. or alternately to complain if trying to install a non editable package that is part of a namespace where an editable package in that namespace is already installed.

another option, if we actually want to make namespace packages and editable installs always work, could be that if a namespace package is installed as editable, pip could reorganize all the other packages in that namespace as editable along side the desired package.

timmow added a commit to alphagov/performanceplatform-collector that referenced this issue Jun 25, 2014

Contributor

hongqn commented Aug 29, 2014

I submitted 2 proposals to fix this issue in setuptools https://bitbucket.org/pypa/setuptools/issue/250/develop-and-install-single-version. FYI.

Contributor

rbtcollins commented Sep 22, 2014

putting

import pkg_resources; pkg_resources.fixup_namespace_packages('')

in a .pth file in site-packages seems sufficient to make pip install -e installed things work correctly.

itsafire commented Feb 4, 2015

This also affects two packages with the same top namespace with both installed as editable. The second package install is broken.

cbrand commented Aug 14, 2015

We use a global namespace for all our applications and thus this issue bit hard (+ the issue that setuptools doesn't support wheels, which are necessary due to the lack of a compiler on our windows deployment platforms). Also all recommended solutions seem to fail in our setup with Python 3.4.

After realizing that pip install -e does run setup.py develop for the project which should be editable, I hooked into the setuptools procedure and wrote a *.pth file, which "introduces" the namespace packages when the python process starts. This resolves the issue for us when using setuptools 18 and pip 7.1.0.

An example setup.py file, which shows how this can be achieved, can be found here:
https://gist.github.com/cbrand/a1624ac3e9c81ce45fcb

I hope this is helpful for other people as well.

wichert commented Sep 3, 2015

I ran into this one today as well, and it's blocking me from switching from buildout to virtualenv+pip.

I created a small demo demonstrate the problem. To test it unpack the .tar.gz and run the {{run-me}} script inside it. Might be useful when testing a fix.

Contributor

sbidoul commented Nov 12, 2015

I'm also hitting this issue.

While trying to understand, I came across the solution proposed by @carljm in #3 (comment), ie have "pip install -e" add a modified version of the standard setuptools ...nspkg.pth file for each develop-installed package.

I experimented with that approach manually and it seems to work well. It looks it would at the same time solve the question of shallow source trees with missing namespace packages as described in #3160.

So I'm wondering if that approach is still considered valid, if there were any attempt to implement it, and if a patch for this would be considered?

Ashald commented Dec 4, 2015

So... no hope on this one to be fixed in near future?

Some side effect of pkg_resources.get_distribution() makes the import work. We discovered this when code that loaded entry points worked correctly while the python shell failed.

This may also explain why an ipython shell has no problems importing the packages.

I'm amazed this bug is still open with no clear work around after 5 years.

If your job is to glue to frameworks together into one unified framework, pip makes your job suck. I honestly have no idea how to proceed forward with my work due to this bug.

Contributor

RonnyPfannschmidt commented Feb 9, 2016

This is a hard to fix issue in primarily volunteer done projects,
Feel free to add the necessary support in setuptools so pip can do this correct

lelit commented Feb 9, 2016

Brandon Github notifications@github.com writes:

I'm amazed this bug is still open with no clear work around after 5 years.

Yeah, it's a pity.

If your job is to glue to frameworks together into one unified framework,
pip makes your job suck. I honestly have no idea how to proceed forward with
my work due to this bug.

I recently applied the fixup_namespace_packages('') hack mentioned above and
it seems to work: I created a z.pth inside the venv's site-packages
containing just import pkg_resources; pkg_resources.fixup_namespace_packages('')

Worth a try, IMHO.

Just yet another bump here!

I'll just mention that every single marrow package utilizes namespaces (some packages populating up to four or five) and that this particular issue has been a bit of a bane of my existence when it comes to helping new users get set up with development tools. I've got just shy of 60 packages using this. The entry_points note is also interesting, as virtually every package contributing to a namespace also contributes entry_points.

The pip approach only works, in my experience, if every package is installed, editable, from disk, that cooperates on the namespace. If one package is installed non-editable via pip, the whole ball of yarn begins to unravel with some very, very obtuse symptoms for new users. (Such as intermittently importable modules; a general sanity check of importing the parent namespace and verifying namespace.__path___ has rapidly identified the culprit borked package in testing.) Mixing properly pip installed packages and pure setup.py develop'd source ones (avoiding pip) appears, however, to work.

We also seem to be able to get into strange situations where a single namespace-contributing package can be installed three different ways, sometimes all simultaneously. (Repeated calls to pip uninstall, each finding more files, is as hilarious to see as it is unfortunate.) Dependency installation when using pip install -e appears to be inconsistent, as well. In a majority of cases we end up with properly unpacked namespaces (installed), pth-link files (installed editable), and in one case we somehow managed to get a zipped .egg installed, despite zip_safe being False on that dependent package and no .egg distribution being provided on Pypi.

Frustration with namespace issues reaches a peak after the fourth time a virtual environment is nuked to start over from scratch. ;)

k4ml commented Nov 15, 2016

Not solution but I just put some notes. When mixing between 'editable' packages, namespaced packages and normal packages, I have more luck with buildout + mr.developer.

derekbekoe added a commit to derekbekoe/azure-cli that referenced this issue Dec 1, 2016

Prevent the use of ‘component’ commands in dev mode
- Also, the dev versions have all command modules installed by default
- If the user really wants to install another module, from some unknown location, they can use pip directly
- * pip doesn’t support mixing editable packages with regular packages *
pip install --editable and pip install clash for namespace packages #3
pypa/pip#3

And, Fix nightly build version patching as well

derekbekoe added a commit to derekbekoe/azure-cli that referenced this issue Dec 1, 2016

Prevent the use of ‘component’ commands in dev mode
- Also, the dev versions have all command modules installed by default
- If the user really wants to install another module, from some unknown location, they can use pip directly
- * pip doesn’t support mixing editable packages with regular packages *
pip install --editable and pip install clash for namespace packages #3
pypa/pip#3

And, Fix nightly build version patching as well

derekbekoe added a commit to derekbekoe/azure-cli that referenced this issue Dec 1, 2016

Prevent the use of ‘component’ commands in dev mode
- Also, the dev versions have all command modules installed by default
- If the user really wants to install another module, from some unknown location, they can use pip directly
- * pip doesn’t support mixing editable packages with regular packages *
pip install --editable and pip install clash for namespace packages #3
pypa/pip#3

And, Fix nightly build version patching as well

derekbekoe added a commit to Azure/azure-cli that referenced this issue Dec 7, 2016

Prevent the use of ‘component’ commands in dev mode (#1485)
- Also, the dev versions have all command modules installed by default
- If the user really wants to install another module, from some unknown location, they can use pip directly
- * pip doesn’t support mixing editable packages with regular packages *
pip install --editable and pip install clash for namespace packages #3
pypa/pip#3

And, Fix nightly build version patching as well

ilendir commented Dec 22, 2016

While the fixup hack @lelit mentioned seems to work for some packages, it also breaks some packages/builds for us.

After some playing around with pip install and editable mode, I came up with the same idea as @carljm (adding a -nspkg.pth file for every editable package), but nobody seems to have implemented that (or was this the now deprecated --egg option?).

Is this still an issue on newer versions of python with PEP420 implemented? (read: would it help to switch to a newer version of python?)

lelit commented Dec 22, 2016

Using PEP420 style effectively helped me several times now wrt the editable mode.

Unfortunately it comes with its own glitches: for example, setuptools' find_packages does not support it, so my newer packages contains the following hack:

-    packages=find_packages('src'),
+    packages=['toplevel.child.' + package
+              for package in find_packages('src/toplevel/child/')],
Owner

jaraco commented Dec 22, 2016

nobody seems to have implemented adding a -nspkg.pth for every editable package.

Check out Setuptools 31 which adds support for this feature, and also co-exists with PEP-420 packages on Python 3.5+.

Member

jonparrott commented Feb 2, 2017

For anyone curious, here's an exhaustive list how namespace packages work across pip install, pip install -e, python setup.py install, and python setup.py develop:

https://github.com/jonparrott/namespace-pkg-tests/blob/master/table.md

tl;dr: You can use pip install -e and python setup.py develop as long as every other package in your namespace has been installed using pip.

@lvdmaaten lvdmaaten referenced this issue in facebookresearch/visdom Mar 17, 2017

Closed

Issue with pip install -e . #16

Owner

dstufft commented Mar 18, 2017

I'm going to close this issue. It appears that setuptools 31 has fixed this for our reproduction scripts, and beyond that there isn't anything pip can do here.

@dstufft dstufft closed this Mar 18, 2017

piotr-dobrogost commented Mar 18, 2017

@jonparrott
What versions of pip and setuptools were used to obtain results at https://github.com/jonparrott/namespace-pkg-tests/blob/master/table.md ?

@dstufft While I appreciate that it's supposedly fixed, I'd like to see @jonparrott 's compatibility table re-run as proof. I hope for the best, plan for the worst on tickets that are nearly 7 years old and have such widely scoped failure scenarios. Confidence is not high given this problem's extensive history, and re-running the nox suite myself, the failures would indicate a lack of actual resolution.

Owner

dstufft commented Mar 18, 2017

In @jonparrott's examples, the failing cases could be distilled down to "using python setup.py install directly (excluding the PEP 420 failures on Python 2, which are expected). This is failing even in cases where pip is not involved at all (such as python setup.py install + python setup.py develop).

This ticket was for combinations of pip install . and pip install -e or python setup.py develop.

The graph already posted by @jonparrott bears this out that this ticket is resolved and any remaining issues here is an issue with python setup.py install and not with pip.

Member

jonparrott commented Mar 20, 2017

I just re-ran my tests and everything is the same as I initially ran it. I agree with @dstufft's assessment - pip has done everything it can do to resolve this.

@piotr-dobrogost the test uses the latest versions of both. As of this writing, pip 9.0.1 and setuptools 34.3.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment