Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passing --no-cache-dir should not affect whether to use a wheel #5749

Closed
nehaljwani opened this issue Sep 2, 2018 · 12 comments

Comments

Projects
None yet
4 participants
@nehaljwani
Copy link
Contributor

commented Sep 2, 2018

At the moment the help output for --no-cache-dir just says: Disable the cache.

Till a week ago, I was under the impression that it meant that pip wouldn't use a cached wheel, and wouldn't cache a wheel fetched from PyPI.

What I didn't know was that when this option is enabled and a package has to be installed from an sdist, pip doesn't actually create a wheel and ends up creating egg-info directories, instead of dist-info ones.

Quick example:

$ pip install https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz
Collecting https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz
  Downloading https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz (318kB)
    100% |████████████████████████████████| 327kB 17.3MB/s 
Building wheels for collected packages: pytz <------------------------- Builds a 'wheel'
  Running setup.py bdist_wheel for pytz ... done
  Stored in directory: /home/nwani/.cache/pip/wheels/0d/29/29/5c4384ddcad4ea97cb91f7ff4ef13ab3cff23ddaf35ce02d21
Successfully built pytz
Installing collected packages: pytz
Successfully installed pytz-2018.5
$ find $PREFIX -type d | grep pytz-2018.5
/tmp/wani.1535901395/dev/lib/python2.7/site-packages/pytz-2018.5.dist-info  <------ 'dist-info'
$ pip install https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz --no-cache-dir
Collecting https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz
  Downloading https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz (318kB)
    100% |████████████████████████████████| 327kB 52.1MB/s 
Installing collected packages: pytz
  Running setup.py install for pytz ... done  <------------------------ No 'wheel' ?
Successfully installed pytz-2018.5
$ find $PREFIX -type d | grep pytz-2018.5
/tmp/wani.1535901395/dev/lib/python2.7/site-packages/pytz-2018.5-py2.7.egg-info <----- 'egg-info'

In the case of '--no-cache-dir', would it not be possible for pip to still build the wheel but throw it away after installation?

@ihnorton

This comment has been minimized.

Copy link

commented Feb 20, 2019

In my case, this breaks installation of a library via pip install, because the wheel step includes building of a necessary shared library (which otherwise works fine using setup.py bdist_wheel and then pointing pip at the result).

ihnorton added a commit to ihnorton/pip that referenced this issue Feb 21, 2019

ihnorton added a commit to ihnorton/pip that referenced this issue Feb 21, 2019

ihnorton added a commit to ihnorton/pip that referenced this issue Feb 21, 2019

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 21, 2019

In the case of '--no-cache-dir', would it not be possible for pip to still build the wheel but throw it away after installation?

I thought it was already supposed to do this? Can you retry with the latest version of pip (19.0.3)? If it's not building a wheel, it's possible there's a separate reason for that (pip has separate logic to decide whether to build a wheel).

@ihnorton

This comment has been minimized.

Copy link

commented Feb 21, 2019

I was using 19.0.3 (also tried other versions). See

# We don't build wheels for legacy requirements if we
# don't have wheel installed or we don't have a cache dir
try:
import wheel # noqa: F401
build_legacy = bool(options.cache_dir)
except ImportError:
build_legacy = False

@nehaljwani

This comment has been minimized.

Copy link
Contributor Author

commented Feb 21, 2019

19.0.3 also doesn't create the wheel.

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 21, 2019

See also this code which does what you suggest, but under a different scenario:

# Otherwise, build the wheel just for this run using the ephemeral
# cache since we are either in the case of e.g. a local directory, or
# no cache directory is available to use.
return True

@ihnorton

This comment has been minimized.

Copy link

commented Feb 21, 2019

AFAICT that is only applicable to the "pep 517" path. In my case -- and I suspect the repro in the original issue -- pep517_requirements is empty, and the needed package is in legacy_requirements.

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 21, 2019

Yes. Let’s keep this issue open to discuss whether the behavior should be changed.

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 21, 2019

I see your confused emoji. What I'm saying is that instead of documenting the current behavior which is what you did in the PR #6284 that you submitted, let's instead discuss whether pip can be changed in the way that the person opening the issue first suggested:

In the case of '--no-cache-dir', would it not be possible for pip to still build the wheel but throw it away after installation?

Also, to clarify the above, the code I referenced is applicable in both the pep 517 and non-pep 517 cases in some scenarios (e.g. without --no-cache-dir). You can see that the return value of should_use_ephemeral_cache () is used in WheelBuilder.build(), prior to determining whether _build_one_pep517() or _build_one_legacy() should be used. This is also consistent with a wheel being built for the package when --no-cache-dir wasn't provided.

@cjerdonek cjerdonek changed the title Query regarding --no-cache-dir Passing --no-cache-dir should not affect whether to build a wheel Feb 22, 2019

@cjerdonek cjerdonek changed the title Passing --no-cache-dir should not affect whether to build a wheel Passing --no-cache-dir should not affect whether to use a wheel Feb 22, 2019

@ihnorton

This comment has been minimized.

Copy link

commented Feb 22, 2019

Changing the behavior would be great. The current behavior seems contrary to the pip install documentation which states that pip runs bdist_wheel when installing from source. That guarantee is important to me, and probably others, because bdist_wheel is familiar and at least somewhat debuggable (whereas I don't really know how to debug pip build failures because the intermediates are removed even with --no-clean).

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 22, 2019

For historical purposes, I found where this logic was first added: #2780

I think the “ephemeral cache” was added a lot more recently, so they didn’t have as many options to address their issue back then.

@ihnorton

This comment has been minimized.

Copy link

commented Feb 22, 2019

I was hoping that specifying a specific (empty) --cache-dir would help. That does force a rebuild, but appears to skip bdist_wheel in the same way. Whereas deleting the global cache directory and re-installing works.

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Feb 27, 2019

FYI, I just filed PR #6303 to address this.

Also, I just marked this issue as an enhancement. It's possible one could argue it's a bug, but since it's debatable IMO I chose the more forgiving option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.