Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add vendor library management support to pip #519

Closed
wants to merge 3 commits into from

4 participants

@ianb
Owner

Do a lookup for config files, looking from the current directory for pip.conf/.pip.conf/pip.ini
Do some limited %() substitution in config files
Handle empty lines in config files better

This is a refreshing of #491 (with bad commits removed)

Original description:

This pull request makes pip look in the current and parent directories for pip.conf and loads it. Also %(here)s gets substituted in config values.

The motivation here is to make directory-local vendor libraries easier, with a config file like:

[install]
install_option = --install-purelib=%(here)s/vendor
    --install-platlib=%(here)s/vendor-binary
    --install-scripts=%(here)s/vendor/bin

Or something like that... the idea isn't complete. Feedback appreciated.

ianb added some commits
@ianb ianb Add --force/-f option to pip uninstall.
Do a lookup for config files, looking from the current directory for pip.conf/.pip.conf/pip.ini
Do some limited %() substitution in config files
Handle empty lines in config files better
bfaf257
@ianb ianb clarify what happens to scripts 1e7c3fc
@ianb ianb Add --script-fixup option to install, that lets you rewrite all scrip…
…ts that are installed.
620a84c
@ianb
Owner

The second commit adds a new option which can be used to facilitate the creation of scripts that use vendor libraries

@pnasrat
Owner

This generally looks good to me, but I'd like to have some tests anyway.

@gvalkov gvalkov referenced this pull request
Closed

Rework command handling #463

@avinoamr

@pypa this would be a great addition as it will make the maintenance of self-encapsulating projects easier. Can you please update on the status of this pull-request? I'd love to help fill in the gaps if you think anything's missing.

@pnasrat
Owner

@avinoamr as before it needs tests - it also needs to apply cleanly if you want to take that up feel free.

@avinoamr

@ianb ian, what's the use-case here? why did you add the 'sys.path' option to the global config?

Owner

Another change allows for non-global config files, e.g., project-specific config files. In the project-specific case you might have project-specific locations where you are installing a library (e.g., using --install-option="--install-lib=..."). That puts the package in, but doesn't let pip find the package, both for uninstallation, and to see what dependencies might already be installed.

@dstufft
Owner

I'm going to close this, It's been awhile and it has no active work being done on it. I'm also not entirely sure that it's a good idea in general.

@dstufft dstufft closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 26, 2012
  1. @ianb

    Add --force/-f option to pip uninstall.

    ianb authored
    Do a lookup for config files, looking from the current directory for pip.conf/.pip.conf/pip.ini
    Do some limited %() substitution in config files
    Handle empty lines in config files better
  2. @ianb

    clarify what happens to scripts

    ianb authored
Commits on Apr 27, 2012
  1. @ianb
This page is out of date. Refresh to see the latest.
View
20 docs/configuration.txt
@@ -52,7 +52,7 @@ global setting with the same name will be overridden; e.g. decreasing the
[global]
timeout = 60
-
+
[freeze]
timeout = 10
@@ -88,6 +88,9 @@ On Unix and Mac OS X the configuration file is: :file:`$HOME/.pip/pip.conf`
And on Windows, the configuration file is: :file:`%HOME%\\pip\\pip.ini`
+Also pip will search in the current and parent directories for a file
+called ``pip.conf`` or ``.pip.conf``, or just ``pip.ini`` on Windows.
+
Environment variables
-----------------------
@@ -139,3 +142,18 @@ configuration file::
mirrors =
http://d.pypi.python.org
http://b.pypi.python.org
+
+Path management
+***************
+
+You can have entries added to ``sys.path`` in your config file,
+generally to support cases when you are installing libraries somewhere
+not normally on the path, but you want to be able to uninstall these
+libraries, or have pip detect them when upgrading/etc. To do this
+use::
+
+ [global]
+ sys.path = %(here)s/vendor/
+
+You can include multiple paths, each on their own line. These are
+added using ``site.addsitedir()`` so ``.pth`` files are respected.
View
14 docs/news.txt
@@ -39,6 +39,20 @@ develop (unreleased)
* Fixed issue #427 - clearer error message on a malformed VCS url. Thanks
Thomas Fenzl.
+* Search for config files (``pip.ini`` on Windows, ``pip.conf`` or
+ ``.pip.conf`` on other platforms) in the current and parent
+ directories.
+
+* Substitute ``%(here)s``, ``%(cwd)s`` and ``%(file)s`` in config file
+ values.
+
+* Add ``-f/--force`` option to ``pip uninstall``, which allows
+ uninstallation of files that aren't inside the virtualenv.
+
+* Add ``--script-fixup``, a hook on installation that allows you to rewrite
+ any scripts that were installed. This can be helpful if you want to make
+ sure the environment is initialized before any scripts run (this is
+ implicitly what virtualenv does, with just the ``#!`` line).
1.1 (2012-02-16)
----------------
View
2  pip/basecommand.py
@@ -62,6 +62,7 @@ def setup_logging(self):
def main(self, args, initial_options):
options, args = self.parser.parse_args(args)
self.merge_options(initial_options, options)
+ self.parser.update_sys_path()
level = 1 # Notify
level += options.verbose
@@ -190,4 +191,3 @@ def load_all_commands():
def command_names():
names = set((pkg[1] for pkg in walk_packages(path=commands.__path__)))
return list(names)
-
View
69 pip/baseparser.py
@@ -6,7 +6,7 @@
import os
from distutils.util import strtobool
from pip.backwardcompat import ConfigParser, string_types
-from pip.locations import default_config_file, default_log_file
+from pip.locations import default_config_file, default_log_file, default_config_file_name, add_explicit_path
class UpdatingDefaultsHelpFormatter(optparse.IndentedHelpFormatter):
@@ -28,7 +28,7 @@ def __init__(self, *args, **kwargs):
self.config = ConfigParser.RawConfigParser()
self.name = kwargs.pop('name')
self.files = self.get_config_files()
- self.config.read(self.files)
+ self.read_config_files(self.files)
assert self.name
optparse.OptionParser.__init__(self, *args, **kwargs)
@@ -36,7 +36,50 @@ def get_config_files(self):
config_file = os.environ.get('PIP_CONFIG_FILE', False)
if config_file and os.path.exists(config_file):
return [config_file]
- return [default_config_file]
+ files = []
+ search_path = os.path.abspath(os.getcwd())
+ last_search_path = None
+ while search_path != last_search_path:
+ path = os.path.join(search_path, default_config_file_name)
+ if os.path.exists(path):
+ files.append(path)
+ if sys.platform != 'win32':
+ path = os.path.join(search_path, '.' + default_config_file_name)
+ if os.path.exists(path):
+ files.append(path)
+ last_search_path = search_path
+ search_path = os.path.dirname(search_path)
+ files.append(default_config_file)
+ # Put local files at the end of the list, where they get highest priority:
+ files.reverse()
+ return files
+
+ def read_config_files(self, files):
+ existing = {}
+ for file in files:
+ if not self.config.read([file]):
+ # This means nothing was actually read
+ continue
+ for section in self.config.sections():
+ for name in self.config.options(section):
+ value = self.config.get(section, name)
+ if existing.get(section, {}).get(name) == value:
+ continue
+ # It's a new option, and we need to do substitution
+ value = self.substitute_config_value(value, file)
+ existing.setdefault(section, {})[name] = value
+ self.config.set(section, name, value)
+
+ def substitute_config_value(self, value, file):
+ file = os.path.abspath(file)
+ subs = [
+ ('%(file)s', file),
+ ('%(here)s', os.path.dirname(file)),
+ ('%(cwd)s', os.getcwd()),
+ ]
+ for var, sub in subs:
+ value = value.replace(var, sub)
+ return value
def update_defaults(self, defaults):
"""Updates the given defaults with values from the config files and
@@ -58,7 +101,10 @@ def update_defaults(self, defaults):
continue
# handle multiline configs
if option.action == 'append':
- val = val.split()
+ if '\n' in val:
+ val = [v.strip() for v in val.splitlines() if v.strip()]
+ else:
+ val = val.split()
else:
option.nargs = 1
if option.action in ('store_true', 'store_false', 'count'):
@@ -111,6 +157,21 @@ def get_default_values(self):
defaults[option.dest] = option.check_value(opt_str, default)
return optparse.Values(defaults)
+ def update_sys_path(self):
+ import pkg_resources
+ prev_sys_path = list(sys.path)
+ if self.config.has_option('global', 'sys.path'):
+ value = self.config.get('global', 'sys.path')
+ value = [v.strip() for v in value.splitlines() if v.strip()]
+ for path in value:
+ import site
+ site.addsitedir(path)
+ for path in sys.path:
+ if path not in prev_sys_path:
+ add_explicit_path(path)
+ pkg_resources.working_set.add_entry(path)
+
+
try:
pip_dist = pkg_resources.get_distribution('pip')
version = '%s from %s (python %s)' % (
View
73 pip/commands/install.py
@@ -2,6 +2,7 @@
import sys
import tempfile
import shutil
+import re
from pip.req import InstallRequirement, RequirementSet
from pip.req import parse_requirements
from pip.log import logger
@@ -89,7 +90,7 @@ def __init__(self):
dest='target_dir',
metavar='DIR',
default=None,
- help='Install packages into DIR.')
+ help='Install packages into DIR (any scripts are thrown away!)')
self.parser.add_option(
'-d', '--download', '--download-dir', '--download-directory',
dest='download_dir',
@@ -165,6 +166,13 @@ def __init__(self):
action='store_true',
help='Install to user-site')
+ self.parser.add_option(
+ '--script-fixup',
+ metavar="file.py:function or module.name:function",
+ dest='script_fixup',
+ help="Calls the given function with a list of all scripts created during installation, like "
+ "function([(req1, script1), (req2, script2)]). You can use this to rewrite scripts.")
+
def _build_package_finder(self, options, index_urls):
"""
Create a package finder appropriate to this install command.
@@ -200,6 +208,11 @@ def run(self, options, args):
finder = self._build_package_finder(options, index_urls)
+ if options.script_fixup:
+ script_fixup = ScriptFixup(options.script_fixup)
+ else:
+ script_fixup = None
+
requirement_set = RequirementSet(
build_dir=options.build_dir,
src_dir=options.src_dir,
@@ -208,7 +221,8 @@ def run(self, options, args):
upgrade=options.upgrade,
ignore_installed=options.ignore_installed,
ignore_dependencies=options.ignore_dependencies,
- force_reinstall=options.force_reinstall)
+ force_reinstall=options.force_reinstall,
+ script_fixup=script_fixup)
for name in args:
requirement_set.add_requirement(
InstallRequirement.from_line(name, None))
@@ -275,5 +289,58 @@ def run(self, options, args):
shutil.rmtree(temp_target_dir)
return requirement_set
-
InstallCommand()
+
+
+class ScriptFixup(object):
+
+ valid_function_re = re.compile(r'^[a-zA-Z_][a-zA-Z_0-9]*$')
+ valid_module_re = re.compile(r'^[a-zA-Z_][a-zA-Z0-9_.]*$')
+
+ def __init__(self, script):
+ self.script = script
+ if ':' not in script:
+ raise CommandError("--script-fixup=%s must be in the form FILE:FUNCTION or MODULE:FUNCTION")
+ path, function = script.rsplit(':', 1)
+ if not self.valid_function_re.search(function):
+ raise CommandError("Function %s in --script-fixup=%s is not a valid function name" % (function, script))
+ if os.path.exists(path):
+ ns = {
+ '__file__': path,
+ '__name__': '__script_fixup__',
+ }
+ logger.debug('Execing %s' % path)
+ try:
+ execfile(path, ns)
+ except:
+ logger.error('Exception while running script %s from --script-fixup=%s' % (path, script))
+ raise
+ if function not in ns:
+ raise CommandError('File %s from --script-fixup=%s does not define a function %s'
+ % (path, script, function))
+ self.function = ns[function]
+ else:
+ if not self.valid_module_re.search(path):
+ raise CommandError("--script-fixup=%s refers to a file that does not exist, or an invalid module name: %s"
+ % (script, path))
+ logger.debug('Importing %s' % path)
+ try:
+ __import__(path)
+ except:
+ logger.error('Exception importing module %s from --script-fixup=%s' % (path, script))
+ raise
+ mod = sys.modules[path]
+ try:
+ self.function = getattr(mod, function)
+ except NameError:
+ raise CommandError(
+ "Module %s (in %s) from --script-fixup=%s does not define a function or object named %s"
+ % (path, mod.__file__, script, function))
+
+ def __call__(self, args):
+ try:
+ self.function(args)
+ except:
+ logger.error(
+ "Exception when calling --script-fixup=%s" % self.script)
+ raise
View
8 pip/commands/uninstall.py
@@ -23,6 +23,12 @@ def __init__(self):
dest='yes',
action='store_true',
help="Don't ask for confirmation of uninstall deletions.")
+ self.parser.add_option(
+ '-f', '--force',
+ dest='force',
+ action='store_true',
+ help="Uninstall (or try to uninstall) a package even when it does not "
+ "appear to be in the current virtual environment")
def run(self, options, args):
requirement_set = RequirementSet(
@@ -38,6 +44,6 @@ def run(self, options, args):
if not requirement_set.has_requirements:
raise InstallationError('You must give at least one requirement '
'to %(name)s (see "pip help %(name)s")' % dict(name=self.name))
- requirement_set.uninstall(auto_confirm=options.yes)
+ requirement_set.uninstall(auto_confirm=options.yes, force=options.force)
UninstallCommand()
View
10 pip/locations.py
@@ -12,6 +12,10 @@ def running_under_virtualenv():
"""
return hasattr(sys, 'real_prefix')
+explicit_paths = []
+
+def add_explicit_path(path):
+ explicit_paths.append(path)
if running_under_virtualenv():
## FIXME: is build/ a good name?
@@ -36,14 +40,16 @@ def running_under_virtualenv():
# buildout uses 'bin' on Windows too?
if not os.path.exists(bin_py):
bin_py = os.path.join(sys.prefix, 'bin')
+ default_config_file_name = 'pip.ini'
user_dir = os.environ.get('APPDATA', user_dir) # Use %APPDATA% for roaming
default_storage_dir = os.path.join(user_dir, 'pip')
- default_config_file = os.path.join(default_storage_dir, 'pip.ini')
+ default_config_file = os.path.join(default_storage_dir, default_config_file_name)
default_log_file = os.path.join(default_storage_dir, 'pip.log')
else:
bin_py = os.path.join(sys.prefix, 'bin')
+ default_config_file_name = 'pip.conf'
default_storage_dir = os.path.join(user_dir, '.pip')
- default_config_file = os.path.join(default_storage_dir, 'pip.conf')
+ default_config_file = os.path.join(default_storage_dir, default_config_file_name)
default_log_file = os.path.join(default_storage_dir, 'pip.log')
# Forcing to use /usr/local/bin for standard Mac OS X framework installs
# Also log to ~/Library/Logs/ for use with the Console.app log viewer
View
55 pip/req.py
@@ -7,8 +7,7 @@
import tempfile
from pip.locations import bin_py, running_under_virtualenv
from pip.exceptions import (InstallationError, UninstallationError,
- BestVersionAlreadyInstalled,
- DistributionNotFound)
+ BestVersionAlreadyInstalled)
from pip.vcs import vcs
from pip.log import logger
from pip.util import display_path, rmtree
@@ -60,6 +59,8 @@ def __init__(self, req, comes_from, source_dir=None, editable=False,
self.install_succeeded = None
# UninstallPathSet of uninstalled distribution (for possible rollback)
self.uninstalled = None
+ # A tracker that will be called with any filenames of files created by this
+ self.file_tracker = None
@classmethod
def from_editable(cls, editable_req, comes_from=None, default_vcs=None):
@@ -95,7 +96,7 @@ def from_line(cls, name, comes_from=None):
# If the line has an egg= definition, but isn't editable, pull the requirement out.
# Otherwise, assume the name is the req for the non URL/path/archive case.
if link and req is None:
- url = link.url_without_fragment
+ url = link.url_fragment
req = link.egg_fragment
# Handle relative file URLs
@@ -394,7 +395,7 @@ def update_editable(self, obtain=True):
'Unexpected version control type (in %s): %s'
% (self.url, vc_type))
- def uninstall(self, auto_confirm=False):
+ def uninstall(self, auto_confirm=False, force=False):
"""
Uninstall the distribution currently satisfying this requirement.
@@ -484,7 +485,7 @@ def uninstall(self, auto_confirm=False):
paths_to_remove.add(os.path.join(bin_py, name) + '.exe.manifest')
paths_to_remove.add(os.path.join(bin_py, name) + '-script.py')
- paths_to_remove.remove(auto_confirm)
+ paths_to_remove.remove(auto_confirm, force)
self.uninstalled = paths_to_remove
def rollback_uninstall(self):
@@ -600,6 +601,8 @@ def install(self, install_options, global_options=()):
filename = line.strip()
if os.path.isdir(filename):
filename += os.path.sep
+ if self.file_tracker:
+ self.file_tracker(filename)
new_lines.append(make_path_relative(filename, egg_info_dir))
f.close()
f = open(os.path.join(egg_info_dir, 'installed-files.txt'), 'w')
@@ -782,7 +785,8 @@ class RequirementSet(object):
def __init__(self, build_dir, src_dir, download_dir, download_cache=None,
upgrade=False, ignore_installed=False,
- ignore_dependencies=False, force_reinstall=False):
+ ignore_dependencies=False, force_reinstall=False,
+ script_fixup=None):
self.build_dir = build_dir
self.src_dir = src_dir
self.download_dir = download_dir
@@ -790,6 +794,7 @@ def __init__(self, build_dir, src_dir, download_dir, download_cache=None,
self.upgrade = upgrade
self.ignore_installed = ignore_installed
self.force_reinstall = force_reinstall
+ self.script_fixup = script_fixup
self.requirements = Requirements()
# Mapping of alias: real_name
self.requirement_aliases = {}
@@ -858,9 +863,9 @@ def get_requirement(self, project_name):
return self.requirements[self.requirement_aliases[name]]
raise KeyError("No project with the name %r" % project_name)
- def uninstall(self, auto_confirm=False):
+ def uninstall(self, auto_confirm=False, force=False):
for req in self.requirements.values():
- req.uninstall(auto_confirm=auto_confirm)
+ req.uninstall(auto_confirm=auto_confirm, force=force)
req.commit_uninstall()
def locate_files(self):
@@ -911,20 +916,17 @@ def prepare_files(self, finder, force_root_egg_info=False, bundle=False):
req_to_install = reqs.pop(0)
install = True
best_installed = False
- not_found = None
if not self.ignore_installed and not req_to_install.editable:
req_to_install.check_if_exists()
if req_to_install.satisfied_by:
if self.upgrade:
- if not self.force_reinstall and not req_to_install.url:
+ if not self.force_reinstall:
try:
url = finder.find_requirement(
req_to_install, self.upgrade)
except BestVersionAlreadyInstalled:
best_installed = True
install = False
- except DistributionNotFound:
- not_found = sys.exc_info()[1]
else:
# Avoid the need to call find_requirement again
req_to_install.url = url.url
@@ -979,8 +981,6 @@ def prepare_files(self, finder, force_root_egg_info=False, bundle=False):
if not os.path.exists(os.path.join(location, 'setup.py')):
## FIXME: this won't upgrade when there's an existing package unpacked in `location`
if req_to_install.url is None:
- if not_found:
- raise not_found
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
else:
## FIXME: should req_to_install.url already be a link?
@@ -1125,8 +1125,13 @@ def install(self, install_options, global_options=()):
if to_install:
logger.notify('Installing collected packages: %s' % ', '.join([req.name for req in to_install]))
logger.indent += 2
+ if self.script_fixup:
+ files = []
try:
for requirement in to_install:
+ if self.script_fixup:
+ files.append((requirement, []))
+ requirement.file_tracker = files[-1][1].append
if requirement.conflicts_with:
logger.notify('Found existing installation: %s'
% requirement.conflicts_with)
@@ -1148,6 +1153,8 @@ def install(self, install_options, global_options=()):
requirement.remove_temporary_source()
finally:
logger.indent -= 2
+ if self.script_fixup:
+ self.run_script_fixup(files)
self.successfully_installed = to_install
def create_bundle(self, bundle_filename):
@@ -1226,6 +1233,22 @@ def _clean_zip_name(self, name, prefix):
name = name.replace(os.path.sep, '/')
return name
+ def run_script_fixup(self, req_files):
+ args = []
+ for req, files in req_files:
+ for file in files:
+ if not os.path.isfile(file):
+ continue
+ fp = open(file)
+ try:
+ first = fp.readline()
+ if first.startswith('#!'):
+ logger.debug('Found #! script: %s' % file)
+ args.append((req, file))
+ finally:
+ fp.close()
+ self.script_fixup(args)
+
def _make_build_dir(build_dir):
os.makedirs(build_dir)
@@ -1408,10 +1431,10 @@ def _stash(self, path):
return os.path.join(
self.save_dir, os.path.splitdrive(path)[1].lstrip(os.path.sep))
- def remove(self, auto_confirm=False):
+ def remove(self, auto_confirm=False, force=False):
"""Remove paths in ``self.paths`` with confirmation (unless
``auto_confirm`` is True)."""
- if not self._can_uninstall():
+ if not force and not self._can_uninstall():
return
logger.notify('Uninstalling %s:' % self.dist.project_name)
logger.indent += 2
View
9 pip/util.py
@@ -9,7 +9,7 @@
import tarfile
from pip.exceptions import InstallationError, BadCommand
from pip.backwardcompat import WindowsError, string_types, raw_input
-from pip.locations import site_packages, running_under_virtualenv
+from pip.locations import site_packages, running_under_virtualenv, explicit_paths
from pip.log import logger
__all__ = ['rmtree', 'display_path', 'backup_dir',
@@ -279,6 +279,9 @@ def is_local(path):
"""
if not running_under_virtualenv():
return True
+ for p in explicit_paths:
+ if normalize_path(path).startswith(normalize_path(p)):
+ return True
return normalize_path(path).startswith(normalize_path(sys.prefix))
@@ -483,6 +486,7 @@ def cache_download(target_file, temp_location, content_type):
def unpack_file(filename, location, content_type, link):
+ filename = os.path.realpath(filename)
if (content_type == 'application/zip'
or filename.endswith('.zip')
or filename.endswith('.pybundle')
@@ -503,6 +507,3 @@ def unpack_file(filename, location, content_type, link):
logger.fatal('Cannot unpack file %s (downloaded from %s, content-type: %s); cannot detect archive format'
% (filename, location, content_type))
raise InstallationError('Cannot determine archive format of %s' % location)
-
-
-
Something went wrong with that request. Please try again.