Implement pip-sync #181

merged 3 commits into from Jul 20, 2015


None yet

2 participants


I implemented pip-sync. It would be nice to have this available, so if there's anything you'd like me to do in order to get this merged, let me know.

Here's an overview of what it does:

  1. Parse requirements.txt using pip.req.parse_requirements
  2. Fetch currently installed modules using pip.get_installed_distributions
  3. Diff these two sets, by module name and version specifier
  4. install and uninstall the resulting modules using pip.main(["(un)install"] + ...)

I had to create a whitelist to avoid uninstalling the following modules, which are automatically installed when creating a virtualenv and not listed on requirements.txt:

  • pip
  • wheel
  • setuptools

I am assuming that requirements.txt contains all direct and indirect dependencies, as if it was generated by pip-compile, so if an indirect dependency is not present, it will be uninstalled and installed again.

nvie commented Jul 19, 2015

This is great — thanks, @hugopeixoto! I got one error trying this out locally. I had an environment where I had installed Jinja2==2.6 manually, then ran pip-sync with the following input file:

# This file is autogenerated by pip-compile
# Make changes in, then run this to update:
#    pip-compile
itsdangerous==0.24        # via flask
jinja2==2.7.3             # via flask
markupsafe==0.23          # via jinja2
werkzeug==0.10.4          # via flask

Then I got the following error:

> pip-sync --dry-run examples/flask.txt
Traceback (most recent call last):
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/bin/pip-sync", line 9, in <module>
    load_entry_point('pip-tools==1.0dev1', 'console_scripts', 'pip-sync')()
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/", line 644, in main
    rv = self.invoke(ctx)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/", line 464, in invoke
    return callback(*args, **kwargs)
  File "/Users/nvie/Projects/pip-tools/piptools/scripts/", line 27, in cli
    to_be_installed, to_be_uninstalled = sync.diff(requirements, installed)
  File "/Users/nvie/Projects/pip-tools/piptools/", line 25, in diff
    elif module.version in requirements[key].specifier:
TypeError: argument of type 'SpecifierSet' is not iterable

Thank you for giving this a try :)

I was testing with python 3.4 and pip 7.1.0. I see that you are using python 2.7.x.

Can you tell me which version of pip do you have installed?
That's probably the cause of the problem. I didn't test for 2.x.
If you tell me which version of pip you're using I'll try to figure out a way to do it in a more compatible way.


It seems that SpecifierSet.__contains__ (which allows us to do version in specifier) was only added in pip 7.0.0. I changed it to specifier.contains(version), which was added in pip 6.0. This should fix the issue.

nvie commented Jul 20, 2015

This is AWESOME. Thanks, will merge immediately :)

@nvie nvie merged commit 3bf3f65 into nvie:future Jul 20, 2015

1 check failed

continuous-integration/travis-ci/pr The Travis CI build failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment