Basic auth credentials in --extra-index-urls are not used when specifying multiple under the same domain

[2rs2ts]

• Pip version: 8.1.2
• Python version: 2.7.9
• Operating System: Debian Jessie (uname -a output: Linux ip-10-254-0-170 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux)

Description:

I am trying to use two --extra-index-urls from packagecloud.io (we have two private indices at our office) but when I specify both I get prompted to log in to packagecloud. When I specify one or the other I do not. I think that pip is having trouble handling the basic auth for each index separately.

What I've run:

This is the problematic command that I wish would work but does not:

$ pip install --user --upgrade \
    --extra-index-url https://<api token>:@packagecloud.io/2rs2ts/oldrepo/pypi/simple \
    --extra-index-url https://<other api token>:@packagecloud.io/2rs2ts/newrepo/pypi/simple \
    mypackage
Collecting mypackage
User for packagecloud.io:

I can work around it by specifying PyPI as an --extra-index-url and one of my private indices as the --index, which makes me think that this is a bug with --extra-index-url:

$ pip install --user --upgrade \
    --index https://<api token>:@packagecloud.io/2rs2ts/oldrepo/pypi/simple \
    --extra-index-url https://<other api token>:@packagecloud.io/2rs2ts/newrepo/pypi/simple \
    --extra-index-url https://pypi.python.org/simple \
    mypackage
Collecting mypackage
  Downloading https://packagecloud.io/2rs2ts/newrepo/pypi/packages/mypackage-1.0.0-py2-none-any.whl (52kB)
etc. etc.

[danriti]

I am running into this same issue with 2 private repositories on packagecloud...

[kahnvex]

Running into this as well.

[kahnvex]

I'm actually unsure if this is an issue with pip or with packagecloud. When pip tries to retrieve a package from packagecloud with the incorrect token, packagecloud should not prompt for a user/pass combo. Im unsure of the specifics, but my guess is that it should signal to pip that the package cannot be found at this url so pip may try another url.

[kahnvex]

Again, I don't know much about pypi/pip, but it seems weird for packagecloud to be doing index auth the way they are. They're creating a new index for each package, but that isn't how pypi works, so this is just going to lead to problems. Maybe there is some undocumented feature to get around this.

[ice799]

Hi @kahnjw, I'm one of the packagecloud.io founders. Not sure I follow what you mean when you say we create "a new index for each package" -- can you elaborate? The way we generate our metadata complies with the Simple Index spec, so this bug is probably not related to the structure of our metadata.

My guess is that based on what @2rs2ts reported that switching the URLs around, it seems the bug is probably in pip.

[kahnvex]

@ice799 I mean that packagecloud requires an indexurl for each package stored on packagecloud.

What I don't understand is whether pip knows which index url (including extra index urls) to use for each package. After eyeballing the structure of the urls, my guess is it does not. If pip naively goes through each index url (including extra index urls) to see which packages it can install, then I think it is possible the issue has nothing to do with pip.

[ice799]

@kahnjw That is not true - we do not require an indexurl for each package stored on packagecloud. Pip uses the URL structure as defined in the spec I linked above to find the package metadata. I think this bug is probably in pip.

[kahnvex]

@ice799 unrelated to this ticket then: can you link me to how to use one index url for all my packagecloud packages? The docs only have examples for one package per index url:

--extra-index-url=https://${token}:@packagecloud.io/user/privateRepo/pypi/simple

[ice799]

@kahnjw The URL you've pasted is for an entire repository -- not a single package. If you upload additional packages to user/privateRepo they will all be accessible at that URL. Please email us at support@packagecloud.io for assistance so we can keep the issues filed against pip relevant to the pip project.

[kahnvex]

Got it repo != package. Thank you.

[ice799]

@2rs2ts @danriti and @kahnjw

I believe the problem is because of how Pip caches the basic auth credentials for index urls: https://github.com/pypa/pip/blob/master/pip/download.py#L160-L162.

It appears that basic auth credentials are cached on the top level domain name of the URL, not the full URL. This means if you have more than 1 repository with the same domain name (in your cases packagecloud.io) pip internally will only be able to store a single basic auth username. Pretty sure this is why you can't authenticate to two different repositories with different basic auth parameters at the same domain name.

I believe a fix to MultiDomainBasicAuth in pip would fix this problem.

[erdnaxeli]

This is extremely annoying. I'm currently trying to do something but it is not really a simple (mostly because once you download the $repo/simple/$package page, the link to the actual tarball does not contains the creds anymore and can be totally different 😱 ).

[pradyunsg]

This issue is labelled as "awaiting PR". If someone wants to file a PR that fixes this, they're welcome to do so. The relevant code is likely in src/pip/_internal/network/auth.py currently.

[SmokinCaterpillar]

Hi, we are facing the same issue, any updates on this issue? Thanks!

[pfmoore]

See @pradyunsg's comment. Someone needs to file a PR for this (which will be linked to this issue, so you'll see the link here when that happens).

In the meantime, while it's OK if people want to note their interest, more reports of people hitting the issue won't help progress much, I'm afraid.