allow some system packages #300

Closed
akumria opened this Issue Jun 6, 2011 · 14 comments

Projects

None yet

7 participants

@akumria

Hi,

Whilst this issue is related to a hypothetical virtualenv setup, after some dicussion with Jannis Leidel in person at DjangoCon.EU, he suggested that this problem be filled here.

However, good practice is to use 'virtualenv --no-site-packages' to ensure that packages that are not required are not installed on a production system. However, some packages, (e.g. PIL), require compilation. And they can be difficult to get installed properly on some platforms (i.e. Windows).

It would be nice if:

$ pip install --copy-from-system

would copy the system installed version of a package and install into the virtualenv (typically by copying the .pth file).

@carljm

This is in general a good idea, I think (in fact, I filed it as a feature request several years ago on the original, pre-Bitbucket, pip issue tracker). There are some things to figure out yet about the correct implementation; copying pth files is only relevant to egg or develop installs and won't work if the global package is installed "flat" (pip-style). Symlinks might be a nice, general approach, but won't work on Windows. So just plain copying the entire package might be necessary in some cases - and I'm guessing there are some cases out there where things are installed with absolute paths for some reason, and so even copying could break.

In other words, I don't think the patch here will be easy or simple, but I'm open to the feature if a patch looks good.

@ncoghlan

I actually need this right now (M2Crypto won't build for me, so I need to use the system one - I'll probably hack something together with some manual symlinks for the moment). It would actually be good if PIP attempted to satisfy dependencies from the system Python installation by default rather than having to be explicitly told to do so.

@carljm

@ncoghlan - that'd be an interesting discussion to have. But first we need an actual implementation of "copy-from-system" that works reliably :-)

@alex
Python Packaging Authority member

Copy from system would be an incredibly dangerous default, it would break whenever your venv Python had a different ABI or API from your system Python and there were C extensions (read: any difference in even micro-releases before CPython 3.3, or altnernative implementations).

@carljm

@alex You've got the wrong definition of "system Python" in mind. The only "system Python" that is relevant to a virtualenv is the Python that the virtualenv Python binary was actually copied from, which by definition always has the same ABI and API - it's the same binary! Virtualenv will never include a feature that goes out and looks at packages installed in any other system Python.

@carljm

Actually, the version of Python a virtualenv was created from could in theory have an in-place micro-version update after the virtualenv was created. This would already today have the potential to break C extensions used in a non --no-site-packages virtualenv; in practice I don't think we've ever heard from anybody having that issue. And the fix would be easy: just copy the updated binary into your virtualenv.

@ncoghlan

Yeah, I meant what Carl said :)

I'm currently working on a script that handles the symlinking approach for some cases (i.e. at least M2Crypto), hopefully that will give us a foundation to start building something robust enough for inclusion in pip itself.

@alex
Python Packaging Authority member

Indeed, sorry for the confusion.

@ncoghlan

My dodgy hack is here: https://bitbucket.org/ncoghlan/misc/src/default/venv_link.py

It qualifies as "good enough" for what I need, but it's completely wrong for pip, since it works at the wrong layer (i.e. individual imports rather than distribution metadata). pip would instead want to look at the database of installed packages and work at the same level it does normally.

@davidfraser

Adjusted ncoghlan's dogy hack: https://gist.github.com/davidfraser/6555916

This uses the system python to find the original package, and pip locations to link it in... It doesn't use distribute to try and locate package resources yet - that would be an improvement

@thedrow

@davidfraser You meant setuptools right? distribute is deprecated.

@davidfraser

@thedrow - I missed that, but of course I mean whatever-the-python-packaging-people-are-currently-trying :)

@thedrow

@davidfraser LOL yeh it's finally being fixed now.

@dstufft
Python Packaging Authority member

Closing this, while it would be useful/nice it's not possible to do across all OSs that pip supports. In order to make this happen we need changes from Python itself to make it possible to have a generic "link" file (similar to egg-link) that points to the importable item instead of the parent directory.

@dstufft dstufft closed this Mar 28, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment