New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip.index.py get_mirrors() is broken. 212,102 mirrors returned #605

Closed
philwhln opened this Issue Jul 18, 2012 · 5 comments

Comments

Projects
None yet
2 participants
@philwhln
Contributor

philwhln commented Jul 18, 2012

To create the mirrors, it uses this....

import socket
socket.gethostbyname_ex("last.pypi.python.org")
('last.pypi.python.org', [], ['129.21.171.98'])

It expects that the returned "last.pypi.python.org" is actually "g.pypi.python.org", which it uses to calculate the other mirror hostnames. This is the seed for the mirrors and because this is wrong, the whole mirroring is broken on a fresh installed VM when pypi.python.org is unreachable.

The list of resulting 4-character mirrors results in 212,102 hostnames which are all invalid mirrors. As you can imagine, pip takes a long time to try all of these.

If I query the CNAME with dig, then it actually comes back ok, so it seems calling gethostbyname_ex is the wrong thing?

;; ANSWER SECTION:
last.pypi.python.org. 86400 IN CNAME g.pypi.python.org.

Are the mirrors cached somewhere? Once pypi.python.org comes up again we never see this issue again and the mirrors are correct.

@philwhln

This comment has been minimized.

Show comment
Hide comment
@philwhln

philwhln Jul 18, 2012

Contributor

I just forked and downloaded onto my Mac, and running "python setup.py test" hangs at "Test installing a package from the PyPI mirrors. ...".

I used dtruss and I can see this is hitting the same issue, so this is not limited to fresh VMs or Ubuntu (the VM I originally found it on).

Contributor

philwhln commented Jul 18, 2012

I just forked and downloaded onto my Mac, and running "python setup.py test" hangs at "Test installing a package from the PyPI mirrors. ...".

I used dtruss and I can see this is hitting the same issue, so this is not limited to fresh VMs or Ubuntu (the VM I originally found it on).

@philwhln

This comment has been minimized.

Show comment
Hide comment
@philwhln

philwhln Jul 18, 2012

Contributor

This is what I see from Vancouver, Canada...

>>> import socket
>>> socket.gethostbyname_ex("last.pypi.python.org")
('g.pypi.python.org', ['last.pypi.python.org'], ['129.21.171.98'])

This is what I see from Japan...

>>> import socket
>>> socket.gethostbyname_ex("last.pypi.python.org")
('last.pypi.python.org', [], ['129.21.171.98'])

When I'm on a machine in Vancouver or on VPN, all is good, but from here, Japan, it's returns the wrong thing and sends my CPU into a spin at hits 212k invalid mirror hosts.

Contributor

philwhln commented Jul 18, 2012

This is what I see from Vancouver, Canada...

>>> import socket
>>> socket.gethostbyname_ex("last.pypi.python.org")
('g.pypi.python.org', ['last.pypi.python.org'], ['129.21.171.98'])

This is what I see from Japan...

>>> import socket
>>> socket.gethostbyname_ex("last.pypi.python.org")
('last.pypi.python.org', [], ['129.21.171.98'])

When I'm on a machine in Vancouver or on VPN, all is good, but from here, Japan, it's returns the wrong thing and sends my CPU into a spin at hits 212k invalid mirror hosts.

@philwhln

This comment has been minimized.

Show comment
Hide comment
@philwhln

philwhln Jul 18, 2012

Contributor

This is solved by #606

Contributor

philwhln commented Jul 18, 2012

This is solved by #606

@pnasrat

This comment has been minimized.

Show comment
Hide comment
@pnasrat

pnasrat Jul 18, 2012

Contributor

I raised the fragility of this a while back on catalog-sig http://mail.python.org/pipermail/catalog-sig/2010-July/003150.html

Can you give more details on the OS and python versions for the boxes above

Contributor

pnasrat commented Jul 18, 2012

I raised the fragility of this a while back on catalog-sig http://mail.python.org/pipermail/catalog-sig/2010-July/003150.html

Can you give more details on the OS and python versions for the boxes above

@philwhln

This comment has been minimized.

Show comment
Hide comment
@philwhln

philwhln Jul 18, 2012

Contributor

On 2012-07-18, at 7:45 PM, Paul Nasrat wrote:

I raised the fragility of this a while back on catalog-sig http://mail.python.org/pipermail/catalog-sig/2010-July/003150.html

Interesting read, thanks.

Can you give more details on the OS and python versions for the boxes above

These are the versions, but it seems network related rather than OS. If I connect to the VPN, works on both. I have no other network issues that I'm aware of.

MacAir Lion 10.7.4

Ubuntu 12
Linux stackato-2ysd 3.2.0-26-virtual #41-Ubuntu SMP Thu Jun 14 18:08:54 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Contributor

philwhln commented Jul 18, 2012

On 2012-07-18, at 7:45 PM, Paul Nasrat wrote:

I raised the fragility of this a while back on catalog-sig http://mail.python.org/pipermail/catalog-sig/2010-July/003150.html

Interesting read, thanks.

Can you give more details on the OS and python versions for the boxes above

These are the versions, but it seems network related rather than OS. If I connect to the VPN, works on both. I have no other network issues that I'm aware of.

MacAir Lion 10.7.4

Ubuntu 12
Linux stackato-2ysd 3.2.0-26-virtual #41-Ubuntu SMP Thu Jun 14 18:08:54 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

@pnasrat pnasrat closed this Jul 29, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment