New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement pip-sync #181

Merged
merged 3 commits into from Jul 20, 2015

Conversation

Projects
None yet
2 participants
@hugopeixoto
Contributor

hugopeixoto commented Jul 17, 2015

I implemented pip-sync. It would be nice to have this available, so if there's anything you'd like me to do in order to get this merged, let me know.

Here's an overview of what it does:

  1. Parse requirements.txt using pip.req.parse_requirements
  2. Fetch currently installed modules using pip.get_installed_distributions
  3. Diff these two sets, by module name and version specifier
  4. install and uninstall the resulting modules using pip.main(["(un)install"] + ...)

I had to create a whitelist to avoid uninstalling the following modules, which are automatically installed when creating a virtualenv and not listed on requirements.txt:

  • pip
  • wheel
  • setuptools

I am assuming that requirements.txt contains all direct and indirect dependencies, as if it was generated by pip-compile, so if an indirect dependency is not present, it will be uninstalled and installed again.

@nvie

This comment has been minimized.

Show comment
Hide comment
@nvie

nvie Jul 19, 2015

Member

This is great — thanks, @hugopeixoto! I got one error trying this out locally. I had an environment where I had installed Jinja2==2.6 manually, then ran pip-sync with the following input file:

#
# This file is autogenerated by pip-compile
# Make changes in flask.in, then run this to update:
#
#    pip-compile flask.in
#
flask==0.10.1
itsdangerous==0.24        # via flask
jinja2==2.7.3             # via flask
markupsafe==0.23          # via jinja2
werkzeug==0.10.4          # via flask

Then I got the following error:

> pip-sync --dry-run examples/flask.txt
Traceback (most recent call last):
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/bin/pip-sync", line 9, in <module>
    load_entry_point('pip-tools==1.0dev1', 'console_scripts', 'pip-sync')()
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/Users/nvie/Projects/pip-tools/piptools/scripts/sync.py", line 27, in cli
    to_be_installed, to_be_uninstalled = sync.diff(requirements, installed)
  File "/Users/nvie/Projects/pip-tools/piptools/sync.py", line 25, in diff
    elif module.version in requirements[key].specifier:
TypeError: argument of type 'SpecifierSet' is not iterable
Member

nvie commented Jul 19, 2015

This is great — thanks, @hugopeixoto! I got one error trying this out locally. I had an environment where I had installed Jinja2==2.6 manually, then ran pip-sync with the following input file:

#
# This file is autogenerated by pip-compile
# Make changes in flask.in, then run this to update:
#
#    pip-compile flask.in
#
flask==0.10.1
itsdangerous==0.24        # via flask
jinja2==2.7.3             # via flask
markupsafe==0.23          # via jinja2
werkzeug==0.10.4          # via flask

Then I got the following error:

> pip-sync --dry-run examples/flask.txt
Traceback (most recent call last):
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/bin/pip-sync", line 9, in <module>
    load_entry_point('pip-tools==1.0dev1', 'console_scripts', 'pip-sync')()
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/nvie/Projects/pip-tools/.direnv/python-2.7.9/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/Users/nvie/Projects/pip-tools/piptools/scripts/sync.py", line 27, in cli
    to_be_installed, to_be_uninstalled = sync.diff(requirements, installed)
  File "/Users/nvie/Projects/pip-tools/piptools/sync.py", line 25, in diff
    elif module.version in requirements[key].specifier:
TypeError: argument of type 'SpecifierSet' is not iterable
@hugopeixoto

This comment has been minimized.

Show comment
Hide comment
@hugopeixoto

hugopeixoto Jul 19, 2015

Contributor

Thank you for giving this a try :)

I was testing with python 3.4 and pip 7.1.0. I see that you are using python 2.7.x.

Can you tell me which version of pip do you have installed?
That's probably the cause of the problem. I didn't test for 2.x.
If you tell me which version of pip you're using I'll try to figure out a way to do it in a more compatible way.

Contributor

hugopeixoto commented Jul 19, 2015

Thank you for giving this a try :)

I was testing with python 3.4 and pip 7.1.0. I see that you are using python 2.7.x.

Can you tell me which version of pip do you have installed?
That's probably the cause of the problem. I didn't test for 2.x.
If you tell me which version of pip you're using I'll try to figure out a way to do it in a more compatible way.

@hugopeixoto

This comment has been minimized.

Show comment
Hide comment
@hugopeixoto

hugopeixoto Jul 19, 2015

Contributor

It seems that SpecifierSet.__contains__ (which allows us to do version in specifier) was only added in pip 7.0.0. I changed it to specifier.contains(version), which was added in pip 6.0. This should fix the issue.

Contributor

hugopeixoto commented Jul 19, 2015

It seems that SpecifierSet.__contains__ (which allows us to do version in specifier) was only added in pip 7.0.0. I changed it to specifier.contains(version), which was added in pip 6.0. This should fix the issue.

@nvie

This comment has been minimized.

Show comment
Hide comment
@nvie

nvie Jul 20, 2015

Member

This is AWESOME. Thanks, will merge immediately :)

Member

nvie commented Jul 20, 2015

This is AWESOME. Thanks, will merge immediately :)

nvie added a commit that referenced this pull request Jul 20, 2015

@nvie nvie merged commit 3bf3f65 into jazzband:future Jul 20, 2015

1 check failed

continuous-integration/travis-ci/pr The Travis CI build failed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment