pip trying to reach pypi.org eventhough index-url overriden

[Spriithy]

Environment

• pip version: 19.3.1
• Python version: 3.7.3
• OS: debian Buster

The machine is setup in a secure / isolated network with no internet connection. I have two machines supplying a PyPi server (namely hereafter index0 and index1).

My /etc/pip.conf looks like this :

[global]

index = http://index0:8080
index-url = http://index0:8080/simple
extra-index = http://index1:8080
extra-index-url = http://index1:8080/simple
trusted-host = index0
                        index1
timeout = 100

Description

When installing a package present on index0's PyPi server, everything works fine. But when installing a package that isn't present on index0 but is on index1, pip tries to contact https://pypi.org/simple/<package_name>. This results in pip retrying 5 times to request this URL in vain eventhough I have my pip.conf overriding the index URLs and where to search for packages. This is extremely similar to #6799 but is not fixed by upgrading pip or python or system whatsoever.

Expected behavior

Pip would skip index0 upon failure and directly request index1.

How to Reproduce

/

Output

I am not able to copy / paste the output of the commands directly since it happens on a network-isolated machine. However, output is extremely similar to that of #6799 and contains :

$ pip install --verbose <package>
[creating some temp dirs]
Looking in indexes: http://index0:8080/simple, http://index1:8080/simple
* http://index0:8080/simple/<package>
* http://index1:8080/simple/<package>
Getting page http://index0:8080/simple/<package>
Found index url http://index0:8080/simple
Starting new HTTP connection (1): index0:8080
http://index0:8080 "GET /simple/<package>/ HTTP/1.1" 303 0
Looking up "https://pypi.org/simple/<package>" in the cache

And then a bunch of pointless retries on this request before successfully contacting index1 and installing the package as expected.

This is pretty annoying I must admit. It would be less of a pain would I be able to override the default retry count in the pip configuration but I haven't been able to find such parameter anywhere.

Thanks in advance for the quick reply

[uranusjr]

I don’t see pip accessing pypi.org in the output you provided. The Looking up "https://pypi.org/simple/<package>" line is using that string as a key to search the on-disk cache, not doing actual network access.

[pfmoore]

Also, wasn't the conclusion in #6799 that the issue was related to Debian's debundling of pip? So if this is the same as that issue, the solution would be the same as well - it needs to be taken up with Debian, or you'll need to reproduce the issue with a vanilla copy of pip.

[Spriithy]

@uranusjr Well, I can't tell. Since the error messages doesn't mention towards which host the requests failed. I just know it's the '/simple/<package> URI that is not found and since it was right after the mentionning of pypi.org, I assumed...

@pfmoore I did install pip from a wheel package I retrieved by pip download pip on another system before moving the package and installing it on my other system by hand. I assumed this would fix the "debian debundling of pip".... I don't really know.

[xavfernandez]

http://index0:8080 "GET /simple// HTTP/1.1" 303 0

It looks like index0 is returning a redirect, maybe to pypi.org ? That would explain that pip ends up looking there.

[Spriithy]

I'm serving index0 with pypiserver. Maybe it comes from there ? I will check with them

[xavfernandez]

@Spriithy You might want to try launching your pypiserver instances with --disable-fallback (to Disable redirect to real PyPI index for packages not found in the local index.)

[Spriithy]

Oh nice ! This fixes the issue so I'm closing it. Thanks for your help !