Skip to content


Browse files Browse the repository at this point in the history
Significantly enhanced support and docs for "non-root" installation,
including both "virtual" and PYTHONPATH-based installs.  The activation
precedence of distributions has also changed so that PYTHONPATH-based
non-root installs can include eggs that override system-defined packages
(whether managed or unmanaged).  This version should eliminate most
common installation complaints from non-root Python users.
Note: this version includes a hacked '' to support processing
.pth files in directories that come *before* site-packages on sys.path.
However, because of its placement, it should only come into play when
a user puts the setuptools .egg file *directly* on PYTHONPATH, so it
doesn't affect "virtual" or "root" installations.  It's strictly to
provide support for luddites who refuse to give up their
existing non-root PYTHONPATH setup unless you pry it from their cold,
dead hands.  :)

branch : setuptools
extra : convert_revision : svn%3A6015fed2-1504-0410-9fe1-9d1591cc4771/sandbox/trunk/setuptools%4041262
  • Loading branch information
PJ Eby committed Oct 17, 2005
1 parent c23b0fb commit 3df2aab
Show file tree
Hide file tree
Showing 7 changed files with 683 additions and 28 deletions.
96 changes: 84 additions & 12 deletions EasyInstall.txt
Expand Up @@ -25,6 +25,8 @@ Using "Easy Install"

.. _installation instructions:

Installing "Easy Install"

Expand Down Expand Up @@ -718,12 +720,46 @@ configuration file will work correctly no matter what Python version you use,
now or in the future.)

If you are on a Linux, BSD, Cygwin, or other similar Unix-like operating
system, you should create a ``~/lib/python2.x/site-packages`` directory
instead. (Note: Ian Bicking has created a script that can automate most of the
process that follows; see
for details.)
system, you have a couple of different options. You can create a "virtual"
Python installation, which uses its own library directories and some symlinks
to the site-wide Python. Or, you can use a "traditional" ``PYTHONPATH``-based
installation, which isn't as flexible, but which you may find more familiar,
especially if you already have a custom ``PYTHONPATH`` set up.

Creating a "Virtual" Python

In the simplest case, your virtual Python installation will live under the
``~/lib/python2.x``, ``~/include/python2.x``, and ``~/bin`` directories. Just
download ``_ and run it using the site-wide Python. If you
want to customize the location, you can use the ``--prefix`` option to specify
an installation base directory in place of ``~``. (Use ``--help`` to get the
complete list of options.)


When you're done, you'll have a ``~/bin/python`` executable that's linked to
the local Python installation and inherits all its current libraries, but which
allows you to add as many new libraries as you want. Simply use this new
Python in place of your system-defined one, and you can modify it as you like
without breaking anything that relies on the system Python. You'll also still
need to follow the standard `installation instructions`_ to install setuptools
and EasyInstall, using your new ``~/bin/python`` executable in place of the
system Python.

Note that if you were previously setting a ``PYTHONPATH`` and/or had other
special configuration options in your ``~/.pydistutils.cfg``, you may need to
remove these settings *before* running ````. You should
also make sure that the ``~/bin`` directory (or whatever directory you choose)
is on your ``PATH``, because that is where EasyInstall will install new Python

If you'd prefer to do the installation steps by hand, or just want to know what
the script will do, here are the steps. (If you don't care how it works, you
can just skip the rest of this section.)

You will need to know your Python version's ``sys.prefix`` and
First, you will need to know your Python version's ``sys.prefix`` and
``sys.exec_prefix``, which you can find out by running::

python -c "import sys; print sys.prefix; print sys.exec_prefix"
Expand Down Expand Up @@ -761,13 +797,43 @@ is on a different filesystem), you should use ``copy -p`` instead of ``ln``.
Do NOT use a symlink! The Python binary must be copied or hardlinked,
otherwise it will use the system ``site-packages`` directory and not yours.

Note that if you were previously setting a ``PYTHONPATH`` and/or had other
special configuration options in your ``~/.pydistutils.cfg``, you may need to
remove these settings and relocate any older installed modules to your
new ``~/lib/python2.x/site-packages`` directory. Also note that you must now
make sure to use the ``~/bin/python`` executable instead of the system Python,
and ideally you should put the ``~/bin`` directory first on your ``PATH`` as
well, because that is where EasyInstall will install new Python scripts.
You can now proceed with the standard `installation instructions`_.

"Traditional" ``PYTHONPATH``-based Installation

This installation method is not as robust or as flexible as `creating a
"virtual" python`_ installation, as it uses various tricks to fool Python into
processing ``.pth`` files where it normally wouldn't. We suggest you try the
virtual Python approach first, as we are providing this method mainly for
people who just can't get past their unshakeable belief that creating a virtual
python is somehow "Not Right", or that putting stuff on ``PYTHONPATH`` "Should
Just Work, Darnit." So, if you're not one of those people, you don't
need these instructions. :-)

Assuming that you want to install packages in a directory called ``~/py-lib``,
and scripts in ``~/bin``, here's what you need to do:

First, edit ``~/.pydistutils.cfg`` to include these settings::

install_lib = ~/py-lib
install_scripts = ~/bin

site_dirs = ~/py_lib

Be sure to do this *before* you try to run the ```` installation
script. Then, follow the standard `installation instructions`_, but take
careful note of the full pathname of the ``.egg`` file that gets installed, so
that you can add it to your ``PYTHONPATH``, along with ``~/py_lib``.

You *must* add the setuptools egg file to your ``PYTHONPATH`` manually, or it
will not work, and neither will any other packages you install with
EasyInstall. You will not, however, have to manually add any other
packages to the ``PYTHONPATH``; EasyInstall will take care of them for you, as
long as the setuptools egg is explicitly listed in ``PYTHONPATH``.

Release Notes/Change History
Expand All @@ -778,6 +844,12 @@ Known Issues
time out or be missing a file.

* Added support for "traditional" PYTHONPATH-based non-root installation, and
also the convenient ```` script, based on a contribution
by Ian Bicking. The setuptools egg now contains a hacked ``site`` module
that makes the PYTHONPATH-based approach work with .pth files, so that you
can get the full EasyInstall feature set on such installations.

* Added ``--no-deps`` option.

* Improved Windows ``.exe`` script wrappers so that the script can have the
Expand Down
10 changes: 5 additions & 5 deletions api_tests.txt
Expand Up @@ -182,12 +182,12 @@ once)::
['', '...pkg_resources...', '...pkg_resources...']

And you can specify the path entry a distribution was found under, using the
optional second parameter to ``add()``
optional second parameter to ``add()``::

>>> ws = WorkingSet([])
>>> ws.add(dist,"foo")
>>> ws.add(dist,"bar")
>>> ws.entries
['', ..., 'foo', 'bar']

But even if a distribution is found under multiple path entries, it still only
shows up once when iterating the working set:
Expand Down Expand Up @@ -222,14 +222,14 @@ again for new distributions added thereafter::
>>> def added(dist): print "Added", dist
>>> ws.subscribe(added)
Added Bar 0.9
>>> foo12 = Distribution(project_name="Foo", version="1.2")
>>> foo12 = Distribution(project_name="Foo", version="1.2", location="f12")
>>> ws.add(foo12)
Added Foo 1.2

Note, however, that only the first distribution added for a given project name
will trigger a callback, even during the initial ``subscribe()`` callback::

>>> foo14 = Distribution(project_name="Foo", version="1.4")
>>> foo14 = Distribution(project_name="Foo", version="1.4", location="f14")
>>> ws.add(foo14) # no callback, because Foo 1.2 is already active

>>> ws = WorkingSet([])
Expand Down
61 changes: 51 additions & 10 deletions
Expand Up @@ -356,7 +356,7 @@ def add_entry(self, entry):
self.entry_keys.setdefault(entry, [])
for dist in find_distributions(entry, True):
self.add(dist, entry)
self.add(dist, entry, False)

def __contains__(self,dist):
Expand Down Expand Up @@ -421,7 +421,7 @@ def __iter__(self):
yield self.by_key[key]

def add(self, dist, entry=None):
def add(self, dist, entry=None, insert=True):
"""Add `dist` to working set, associated with `entry`
If `entry` is unspecified, it defaults to the ``.location`` of `dist`.
Expand All @@ -432,23 +432,23 @@ def add(self, dist, entry=None):
doesn't already have a distribution in the set. If it's added, any
callbacks registered with the ``subscribe()`` method will be called.
if insert:
dist.insert_on(self.entries, entry)

if entry is None:
entry = dist.location

if entry not in self.entry_keys:
self.entry_keys[entry] = []
keys = self.entry_keys.setdefault(entry,[])

if dist.key in self.by_key:
return # ignore hidden distros

self.by_key[dist.key] = dist
keys = self.entry_keys[entry]
if dist.key not in keys:


def resolve(self, requirements, env=None, installer=None):
"""List all distributions needed to (recursively) meet `requirements`
Expand Down Expand Up @@ -1837,12 +1837,12 @@ def _get_metadata(self,name):
def activate(self,path=None):
"""Ensure distribution is importable on `path` (default=sys.path)"""
if path is None: path = sys.path
if self.location not in path:
if path is sys.path:
map(declare_namespace, self._get_metadata('namespace_packages.txt'))

def egg_name(self):
"""Return what this distribution's standard .egg filename should be"""
filename = "%s-%s-py%s" % (
Expand Down Expand Up @@ -1907,6 +1907,47 @@ def get_entry_info(self, group, name):
"""Return the EntryPoint object for `group`+`name`, or ``None``"""
return self.get_entry_map(group).get(name)

def insert_on(self, path, loc = None):
"""Insert self.location in path before its nearest parent directory"""
loc = loc or self.location
if not loc: return
if path is sys.path:
best, pos = 0, -1
for p,item in enumerate(path):
if loc.startswith(item) and len(item)>best and loc<>item:
best, pos = len(item), p
if pos==-1:
if loc not in path: path.append(loc)
elif loc not in path[:pos+1]:
while loc in path: path.remove(loc)

def check_version_conflict(self):
if self.key=='setuptools':
return # ignore the inevitable setuptools self-conflicts :(

nsp = dict.fromkeys(self._get_metadata('namespace_packages.txt'))

for modname in self._get_metadata('top_level.txt'):
if modname not in sys.modules or modname in nsp:

fn = getattr(sys.modules[modname], '__file__', None)
if fn and fn.startswith(self.location):

from warnings import warn
"Module %s was already imported from %s, but %s is being added"
" to sys.path" % (modname, fn, self.location)

Expand Down Expand Up @@ -2165,9 +2206,9 @@ def _initialize(g):
add_activation_listener = working_set.subscribe
run_script = working_set.run_script
run_main = run_script # backward compatibility

# Activate all distributions already on sys.path, and ensure that
# all distributions added to the working set in the future (e.g. by
# calling ``require()``) will get activated as well.
add_activation_listener(lambda dist: dist.activate())
working_set.entries=[]; map(working_set.add_entry,sys.path) # match order

11 changes: 11 additions & 0 deletions pkg_resources.txt
Expand Up @@ -1488,6 +1488,17 @@ File/Path Utilities
Release Notes/Change History

* Activated distributions are now inserted in ``sys.path`` (and the working
set) just before the directory that contains them, instead of at the end.
This allows e.g. eggs in ``site-packages`` to override unmanged modules in
the same location, and allows eggs found earlier on ``sys.path`` to override
ones found later.

* When a distribution is activated, it now checks whether any contained
non-namespace modules have already been imported and issues a warning if
a conflicting module has already been imported.

* Fix a bug in ``WorkingSet.resolve()`` that was introduced in 0.6a3.

Expand Down
2 changes: 1 addition & 1 deletion
Expand Up @@ -37,7 +37,7 @@ def get_description():
test_suite = 'setuptools.tests.test_suite',
packages = find_packages(),
package_data = {'setuptools': ['*.exe']},
py_modules = ['pkg_resources', 'easy_install'],
py_modules = ['pkg_resources', 'easy_install', 'site'],

zip_safe = False, # We want 'python -m easy_install' to work, for now :(
entry_points = {
Expand Down

0 comments on commit 3df2aab

Please sign in to comment.