New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"pip wheel" does nothing if a wheel is available from PyPI #1310

Closed
rassie opened this Issue Nov 8, 2013 · 25 comments

Comments

Projects
None yet
@rassie

rassie commented Nov 8, 2013

Use case: I'm preparing a wheelhouse with dependencies for our product as wheels. The installation afterwards is supposed to run only from these wheels, i.e. also with --no-index.

However, pip wheel fails to actually download a wheel if it's available from PyPI. For example, I would like to have a wheel for setuptools locally:

$ mkvirtualenv wheel
[....]
$ pip install setuptools==1.3.1 wheel https://github.com/pypa/pip/archive/develop.zip
[....]
$ pip wheel setuptools==1.3.1
Downloading/unpacking setuptools==1.3.1
  Downloading setuptools-1.3.1-py2.py3-none-any.whl (549kB): 549kB downloaded
Building wheels for collected packages: setuptools
  Skipping building wheel: https://pypi.python.org/packages/3.3/s/setuptools/setuptools-1.3.1-py2.py3-none-any.whl#md5=e39a9b0e37005b4cd5e0d328cb88fd77
Cleaning up...

I would expect pip wheel to create correct wheels in every possible way: by compiling, downloading or otherwise. It should only skip doing anything if there is a correct wheel file in the target directory.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 8, 2013

Contributor

I agree it should find the setuptools wheel and place it into your wheelhouse. marking this as a bug.

Contributor

qwcode commented Nov 8, 2013

I agree it should find the setuptools wheel and place it into your wheelhouse. marking this as a bug.

@akheron

This comment has been minimized.

Show comment
Hide comment
@akheron

akheron Dec 3, 2013

I'm having the same issue with django-appconf.

akheron commented Dec 3, 2013

I'm having the same issue with django-appconf.

@akheron

This comment has been minimized.

Show comment
Hide comment
@akheron

akheron Dec 3, 2013

Actually, my problem is that I'm trying to update my wheelhouse when requirements.txt is updated. For that, I invoke pip wheel --wheel-dir=wheelhouse --use-wheel --find-links=wheelhouse. This builds all updated requirements as wheels and saves them to wheelhouse, except for those that already have a wheel available on PyPI.

akheron commented Dec 3, 2013

Actually, my problem is that I'm trying to update my wheelhouse when requirements.txt is updated. For that, I invoke pip wheel --wheel-dir=wheelhouse --use-wheel --find-links=wheelhouse. This builds all updated requirements as wheels and saves them to wheelhouse, except for those that already have a wheel available on PyPI.

@jbalonso

This comment has been minimized.

Show comment
Hide comment
@jbalonso

jbalonso Dec 3, 2013

This is a bit of a brutal hack, but this is my workaround:

mkdir -p "${WHEELHOUSE}/cache"
pip wheel --use-wheel -w "${WHEELHOUSE}" -f "${WHEELHOUSE}" -r "${REQUIREMENTS}" --download-cache "${WHEELHOUSE}/cache"

echo CACHE: ${WHEELHOUSE}/cache
for x in "${WHEELHOUSE}/cache"/*.whl; do
        echo ...copying "${x##*%2F}"
        cp $x "${WHEELHOUSE}/${x##*%2F}"
done

jbalonso commented Dec 3, 2013

This is a bit of a brutal hack, but this is my workaround:

mkdir -p "${WHEELHOUSE}/cache"
pip wheel --use-wheel -w "${WHEELHOUSE}" -f "${WHEELHOUSE}" -r "${REQUIREMENTS}" --download-cache "${WHEELHOUSE}/cache"

echo CACHE: ${WHEELHOUSE}/cache
for x in "${WHEELHOUSE}/cache"/*.whl; do
        echo ...copying "${x##*%2F}"
        cp $x "${WHEELHOUSE}/${x##*%2F}"
done
@akheron

This comment has been minimized.

Show comment
Hide comment
@akheron

akheron Dec 4, 2013

@jbalonso Thanks, works like a charm!

akheron commented Dec 4, 2013

@jbalonso Thanks, works like a charm!

@Dr-Bean

This comment has been minimized.

Show comment
Hide comment
@Dr-Bean

Dr-Bean Jan 2, 2014

I have a similar situation, although in our case the issue is not directly related to PyPI, but using --no-index --find-links="wheelhouse".

We have been using pip bundle for application packaging, but my goal now is to set up a central wheelhouse (similar to PyPI) to store wheels after building them.
Using a requirements file, the needed wheels should be copied from the central wheelhouse to a 'package wheelhouse', for distributing together with the application in a package, and ultimately, client-side installation. However, we run into the same message with that approach.

Dr-Bean commented Jan 2, 2014

I have a similar situation, although in our case the issue is not directly related to PyPI, but using --no-index --find-links="wheelhouse".

We have been using pip bundle for application packaging, but my goal now is to set up a central wheelhouse (similar to PyPI) to store wheels after building them.
Using a requirements file, the needed wheels should be copied from the central wheelhouse to a 'package wheelhouse', for distributing together with the application in a package, and ultimately, client-side installation. However, we run into the same message with that approach.

@GaretJax

This comment has been minimized.

Show comment
Hide comment
@GaretJax

GaretJax Jan 11, 2014

+1 for me as well.

GaretJax commented Jan 11, 2014

+1 for me as well.

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Jan 24, 2014

Member

If the behaviour were to change so that pip instead did not ignore packages already found as wheels (i.e. cache all packages requested, as wheels full stop), which I think is a more sensible and intuitive default, I'm wondering if there's a use case to warrant adding a flag to revert to the current behaviour?

Member

Ivoz commented Jan 24, 2014

If the behaviour were to change so that pip instead did not ignore packages already found as wheels (i.e. cache all packages requested, as wheels full stop), which I think is a more sensible and intuitive default, I'm wondering if there's a use case to warrant adding a flag to revert to the current behaviour?

@mitsuhiko

This comment has been minimized.

Show comment
Hide comment
@mitsuhiko

mitsuhiko Jan 25, 2014

Contributor

This is also currently different behavior than the documentation suggests.

Contributor

mitsuhiko commented Jan 25, 2014

This is also currently different behavior than the documentation suggests.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Jan 25, 2014

Contributor

let me add a warning in the docs for now, until this get's fixed.
(6d25e12)

Contributor

qwcode commented Jan 25, 2014

let me add a warning in the docs for now, until this get's fixed.
(6d25e12)

@leo-the-manic

This comment has been minimized.

Show comment
Hide comment
@leo-the-manic

leo-the-manic Jan 29, 2014

If I'm not mistaken, this issue tracks the bug discussed in the article "Python on Wheels?" (See here)

leo-the-manic commented Jan 29, 2014

If I'm not mistaken, this issue tracks the bug discussed in the article "Python on Wheels?" (See here)

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Jan 29, 2014

Contributor

I'm working on this, so should have a PR soon.

Contributor

qwcode commented Jan 29, 2014

I'm working on this, so should have a PR soon.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Feb 2, 2014

Contributor

closing due to merge of #1524
for those following this issue, please try from the develop branch

Contributor

qwcode commented Feb 2, 2014

closing due to merge of #1524
for those following this issue, please try from the develop branch

@qwcode qwcode closed this Feb 2, 2014

@Dr-Bean

This comment has been minimized.

Show comment
Hide comment
@Dr-Bean

Dr-Bean Feb 3, 2014

Tested, works for my use-case (local wheelhouse). Downloading PyPI wheels works too. Thanks!

Dr-Bean commented Feb 3, 2014

Tested, works for my use-case (local wheelhouse). Downloading PyPI wheels works too. Thanks!

@Fiedzia

This comment has been minimized.

Show comment
Hide comment
@Fiedzia

Fiedzia Feb 7, 2014

Any chance to make 1.5.3 with this fix? Right now stable pip version is just broken, and 1.6 may be months away (guessing from release history).

Fiedzia commented Feb 7, 2014

Any chance to make 1.5.3 with this fix? Right now stable pip version is just broken, and 1.6 may be months away (guessing from release history).

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Feb 7, 2014

Contributor

@Fiedzia yes I was thinking the same (along with #1527)

@pfmoore , @dstufft thoughts?

Contributor

qwcode commented Feb 7, 2014

@Fiedzia yes I was thinking the same (along with #1527)

@pfmoore , @dstufft thoughts?

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Feb 7, 2014

Member

@qwcode I'm happy enough to see a new release (but I'm not the one doing the work :-)) It would be good to get this fix into the version shipped with Python 3.4.

Member

pfmoore commented Feb 7, 2014

@qwcode I'm happy enough to see a new release (but I'm not the one doing the work :-)) It would be good to get this fix into the version shipped with Python 3.4.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Feb 7, 2014

Member

Seems reasonable to me to roll this into the 1.5.X branch.

Member

dstufft commented Feb 7, 2014

Seems reasonable to me to roll this into the 1.5.X branch.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Feb 7, 2014

Contributor

ok, I'll push it over later.

Contributor

qwcode commented Feb 7, 2014

ok, I'll push it over later.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Feb 8, 2014

Contributor

I have the push ready, but too many problems atm, due to bitbucket being down
will wait til later.

Contributor

qwcode commented Feb 8, 2014

I have the push ready, but too many problems atm, due to bitbucket being down
will wait til later.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Feb 8, 2014

Contributor

ok, pushed to 1.5.X (along with #1527) with a successful travis build.

Contributor

qwcode commented Feb 8, 2014

ok, pushed to 1.5.X (along with #1527) with a successful travis build.

@sebastianneubauer

This comment has been minimized.

Show comment
Hide comment
@sebastianneubauer

sebastianneubauer Jul 3, 2014

Mmmh, this bug actually was a feature for me: I was happy that in the wheel dir only those packages ended up which are not already found somewhere (--find-links). This is really helpfull if you need to build/maintain a local (no access to pypi.python.org) wheelhouse, in my case a local devpi. Maybe it would be possible to introduce a --no-copy flag if someone only wants to have those wheels which are not already existing and just copied? Best would be if one could specify from where the wheels are copied (outside firewall e.g. pypi.python.org) and from where not (inside firewall, local wheelhouse)

sebastianneubauer commented Jul 3, 2014

Mmmh, this bug actually was a feature for me: I was happy that in the wheel dir only those packages ended up which are not already found somewhere (--find-links). This is really helpfull if you need to build/maintain a local (no access to pypi.python.org) wheelhouse, in my case a local devpi. Maybe it would be possible to introduce a --no-copy flag if someone only wants to have those wheels which are not already existing and just copied? Best would be if one could specify from where the wheels are copied (outside firewall e.g. pypi.python.org) and from where not (inside firewall, local wheelhouse)

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Jul 3, 2014

Contributor

a --no-copy flag seems reasonable to me.

Contributor

qwcode commented Jul 3, 2014

a --no-copy flag seems reasonable to me.

@sublee

This comment has been minimized.

Show comment
Hide comment
@sublee

sublee Dec 2, 2015

+1 for --no-copy.

sublee commented Dec 2, 2015

+1 for --no-copy.

@flaugher

This comment has been minimized.

Show comment
Hide comment
@flaugher

flaugher Jan 20, 2016

I'd like to be able to "pip wheel --wheel-dir=/path -r requirements.txt regardless of whether there's a PypI wheel so that I always have a backup I can use. I'd think this would be desirable since it would reduce PyPI traffic.

flaugher commented Jan 20, 2016

I'd like to be able to "pip wheel --wheel-dir=/path -r requirements.txt regardless of whether there's a PypI wheel so that I always have a backup I can use. I'd think this would be desirable since it would reduce PyPI traffic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment