--find-links detection for files generated by --download-cache #696

wants to merge 1 commit into


None yet
4 participants


Problem description

--download-cache is nice to avoid re-downloading files.

--find-links does not recognizes files created using --download-cache since the repository name is contained in the file name.

It would be nice if --find-links could also use cached files.

This problem was also raised in #246 but it look like the original reported did not continue to support it.


The regular expression for _egg_info_re was updated to extract the package name from file names created using --download-cache options.

RE named groups are used instead of group numbers.

The tests were done on 'private' method. Not sure if this is the right thing to do, but I saw some other tests for private methods.

I wanted to add the test on a public method, but PackageFinder.find_requirement is doing a lot of things and it would be hard to focus the test.

This is my first contribution to pip, so there is high probability of doing something wrong.
Please do not hesitate to raise any issues, even minor ones.

How to test

Test were added for PackageFinder._egg_info_matches

nosetests tests.test_index



qwcode commented Oct 3, 2012

looks like @pnasrat was looking at the previous attempt at this.
I certainly understand the motivation, but here's my hesitation:

  1. people can currently use "pip install -d" to populate a find-links dir pretty easy.
  2. this seems like a compensation for pip not having a name-based (i.e. not url-based) package cache. I'd rather create that (and plan on doing so time willing, unless someone else does it first). This --package-cache would be new and leave the existing --download-cache as it is.

qwcode commented Oct 3, 2012

@adiroiban btw, if you have any interest in experimenting with a --package-cache feature, let me know.


I am currently using pip install -d, but this is an extra command that needs to be explicitly called.
The goal of this change is to seamlessly use pip install (with many arguments) and have a single command take care of caching and reusing cached files if network is down.

Together with #697 this allows me to use:

pip install --index-url=MY_PYPI --download-cache=CACHE_PATH --find-links=file://CACHE_PATH

if network is down it will use the previously cached files by the same command.

I would like to look at ``--package-cache` feature.
Is there a ticket/issue for that or somewhere describing what it should do?



qwcode commented Oct 3, 2012

@adiroiban let me write up my thoughts on gist about --package-cache. I'll ping you when I post it.


qwcode commented Oct 3, 2012

@adiroiban : thoughts on "--package-cache" https://gist.github.com/3828762


What are the drawbacks of changing the regular expression for --find-link?

I should be able to implement the "--package-cache" option, but I don't know if adding yet another option is the right thing to do.

To me it looks like '--download-cache' is not enabled by default.

I think that beside filenames, this is the only important difference from '--download-cache'

the package-cache (if turned on) would be checked prior to using the PackageFinder
(which searches pypi and findlinks)
if the req was "nose", any nose version in the package-cache would "win", unless --upgrade is specified

This would replace the need of running 2 commands:

    - pip install -d DIR PACKAGES
    - pip install --no-index --find-links=DIR PACKAGES

If you want to have the latest versions, you will have to run pip install -d DIR PACKAGES from time to time.

If a new version of a package is available on the online index, the local cache will no longer be valid... and when using "--package-cache" pip will not check the online index.

Since I am new to pip I can't see the whole picture, so maybe my arguments are not valid



qwcode commented Oct 5, 2012

What are the drawbacks of changing the regular expression for --find-link?

for me, it's not about the drawbacks, it's just not something I would ever do myself, i.e. change pip's find-link support to be able to find stuff in pip's internal download cache. but that's me, maybe other maintainers would want to do this.

about --package-cache, lets just talk on gist about that if you want, and keep this thread about this pull. I would need to drum up conversation on the mailing list at some point to flush out the arguments. I could see the "no more cruft/options" argument winning quickly.


ptone commented Apr 1, 2013

So as the original contributor of this idea in #246 I'll make another attempt at a case for it.

The advantage is when you already have a well seeded download cache, and you are often rebuilding virtualenvs consisting of the same packages.

-d or --package-cache just creates another place where you have a pile of tarballs you might use frequently and seems to be largely redundant when you already have the downloads available in the download-cache. One thing I didn't understand from the gist was how '"--package-cache" could be used with "--download-cache"'. How would one pass the download-cache in to the package-cache option?


qwcode commented Apr 1, 2013

@ptone , I just created a separate issue for the --package-cache idea, since it came up at pycon. we can discuss the independent merits of that over there. #878

--package-cache just creates another place

the idea with --package-cache is that it would be the only cache, not another cache.
--package-cache would become the preferred choice for caching. it's what most people want and expect pip should be doing.


dstufft commented Feb 11, 2014

I'm going to close this. It doesn't merge correctly anymore and I agree with Marcus that we shouldn't be extending find links in this way.

@dstufft dstufft closed this Feb 11, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment