PyPI download url is not working, upload to PyPI instead?

[jtriley]

Hi Robey,

Your download url to lag.net is not currently working. Would you mind uploading the 1.7.6 zip file to paramiko's pypi page instead of using the download url that points to lag.net?

~Justin

[goosemo]

Seconded. It's been almost 3 days that the download link from your site has been down. It'd be much more reliable to push the hosting of that off to pypi.

[elbeanio]

Thirded. Please upload to pypi.

[dmoisset]

I'd also be happy with this... alternatively, could you at least add a tag for the 1.7.6 release, so I can pull that with pip? Currently the only tag I see is for 1.7.5

[goosemo]

@dmoisset I've actually done that in my fork of this repo, if you needed a shortcut.

[robey]

i'm at the mercy of my VPS, which has been down for four days. :(

[jtriley]

@robey no problem. I don't mean to nag but would you mind uploading the 1.7.6 release to paramiko's pypi page? that should fix the issue all together and stop downloading from lag.net when installed via easy_install. my app depends on paramiko and as a consequence is hanging when easy_install-ing any time lag.net goes down. imo it's safer to upload and distribute your releases on pypi given that pypi.python.org has to be up for easy_install anyway (afaik).

[robey]

i think i just uploaded the zipfile to pypi. did that work?

[jtriley]

@robey thanks a lot for doing this. unfortunately both pip and easy_install are still trying to pull from lag.net for some reason:

(deleteme)user@localhost $ easy_install paramiko
Searching for paramiko
Reading http://pypi.python.org/simple/paramiko/
Reading http://www.lag.net/~robey/paramiko/
Download error: [Errno 8] nodename nor servname provided, or not known -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: [Errno 8] nodename nor servname provided, or not known -- Some packages may not be found!
Best match: paramiko 1.7.6
Downloading http://www.lag.net/paramiko/download/paramiko-1.7.6.zip
error: Download error for http://www.lag.net/paramiko/download/paramiko-1.7.6.zip: [Errno 8] nodename nor servname provided, or not known

I wonder what happens if you try removing the download_url field for the 1.7.6 release? Should be able to do this logged into the pypi site.

[olt]

@robey: removing the download_url and python setup.py register should be enough.

[chrisglass]

I made a pull request for just that (removing download_url), if that's necessary (guess not, it's basically deleting one line).

Registering with pypi you'll have to do yourself however, unless you allow me to, I'm chrisglass there as well.

Cheers

[robey]

ok, how's that?

[jtriley]

hmmm ok it seems a little better...it's still hanging on lag.net but eventually downloads from pypi.

it appears that setuptools is attempting to go through each version of paramiko you've posted. If you look at http://pypi.python.org/simple/paramiko you'll see links for a bunch of old versions. Do you by chance have these versions as 'releases' in your pypi page? I can't seem to access any old versions of paramiko on pypi such as: http://pypi.python.org/pypi/paramiko-1.7.5 so I'm wondering why these links are on the http://pypi.python.org/simple/paramiko page.

Would you mind checking if there are old releases/downloads/urls on paramiko's pypi page?

[jtriley]

i'm now looking into where these links on the http://pypi.python.org/simple/paramiko page come from and how to change them. this should solve the problem I think. obviously pip/easy_install are still trying to parse them:

(deleteme)OEIT-5:site-packages OEIT$ pip install -v paramiko
Downloading/unpacking paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/ (from http://pypi.python.org/simple/paramiko/): <urlopen error [Errno 8] nodename nor servname provided, or not known>
Will skip URL http://www.lag.net/%7erobey/paramiko/ when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.1-charmander.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.1-charmander.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-fearow.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-fearow.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-ivysaur.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-ivysaur.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/secsh-0.1-bulbasaur.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/secsh-0.1-bulbasaur.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-doduo.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-doduo.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/ (from http://pypi.python.org/simple/paramiko/): <urlopen error [Errno 8] nodename nor servname provided, or not known>
Will skip URL http://www.lag.net/paramiko/ when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.4.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.4.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.5.4.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.5.4.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.6.3.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.6.3.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-eevee.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-eevee.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-horsea.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-horsea.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-1.0.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-1.0.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.7.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.7.zip when looking for download links for paramiko
^C Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.7.5.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.7.5.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.7.2.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.7.2.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.3.1.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.3.1.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.5.2.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.5.2.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-gyarados.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/%7erobey/paramiko/paramiko-0.9-gyarados.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.3.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.3.zip when looking for download links for paramiko
Could not fetch URL http://www.lag.net/paramiko/download/paramiko-1.2.zip (from http://pypi.python.org/simple/paramiko/): [Errno 8] nodename nor servname provided, or not known
Will skip URL http://www.lag.net/paramiko/download/paramiko-1.2.zip when looking for download links for paramiko

[goosemo]

I was able to install w/o issue, but you'll need to shorten the timeout to 10 or so, because of those links. I figured they were cached.

[jtriley]

yea i'm wondering if we just have to wait for the simple index page to be regenerated...may take some time

[jtriley]

OK so still not working. I'm fairly certain that this same procedure needs to be done for each previous (hidden) version of paramiko in order for easy_install/pip to stop reading lag.net links.

Robey, I know this is a huge pain but would you be willing to upload the source and remove the download_url field from each previous version's pypi page? You should be able to remove the download_url fields from each previous version just by logging into pypi. I think this will fix things once and for all.

[chrisglass]

Yeah I concur with jtriley - please fix this once and for all :(

[jtriley]

sorry to nag but pretty please?

[jtriley]

Robey, would you please try uploading the source to pypi and remove the download_url field from EACH PREVIOUS VERION's pypi page? You should be able to remove the download_url fields from each previous version just by logging into pypi. I'm almost 100% this will fix the issue. Again, sorry to nag but this affects a lot of software (including mine ;)

[robey]

can you just use the latest version? or fix pypi?

[chrisglass]

Just give me credentials and I'll be glad to do it for you (chrisglass on pypi as well).
That is, go here http://pypi.python.org/pypi?:action=role_form&package_name=paramiko
and type chrisglass in the username box. I guess maintainer should give me enough liberty to fix this once and for all.
Please revoke theses credentials as you see fit (just leave me a day to actually do it)

[jtriley]

Robey,

The previous versions of paramiko on pypi are causing lots of timeouts when installing the latest version (see my comment above Oct 27), this is why I'm suggesting uploading the src and removing the download_url for each previous paramiko version on pypi if you intend to keep them around.

I'm also willing to fix this for you (jtriley on pypi).

~Justin

[ojii]

Is there any chance this will ever be fixed? For now I just put my own package ("paramiko-on-pypi") onto pypi to fix this really annoying issue.

[goosemo]

@ojii you can also just have setup.py or pip point to a github repo. Works well enough when pypi is down.

[ojii]

@goosemo, how is that a valid fix? Also I'm not talking about pypi going down but about lag.net going down

[goosemo]

It's only this line that is making it point to lag.net, in the setup.py. Remove it, or point it explicitly at pypi, as the first was suggested by @chrisglass who has a pull request in:
https://github.com/robey/paramiko/blob/master/setup.py#L57

If you're already making you own paramiko on pypi, you're doing more work than this simple change on your end would be. Beyond that, we just have to wait for this issue to be closed by robey.

[ojii]

@goosemo: My own package on there is just so we can install fabric without having to pray for lag.net to be up. (see fabric-with-working-dependencies). So yes, I want this to be fixed in this paramiko and I want it fixed ASAP. What's the point of releasing a package when it's unavailable all the time?

[goosemo]

Let me be more clear. Pypi has been fixed already. It works. You are being melodramatic, and really not keeping up. If you are having pip issues, you should post the stacktrace of that, and give actual details. But pip install paramiko works w/o issue, downloading from pypi work w/o issue. So really, where is your issue?

[jtriley]

@ojii: I completely agree and this is why I opened this issue in the first place. I have software (http://github.com/jtriley/StarCluster) that depends on paramiko and having the installation stall because of lag.net downtime is annoying and makes any software depending on it appear broken to users. In this issue @chrisglass and I have offered multiple times to fix this for Robey but no response unfortunately...

@goosemo: I believe the fix is a little more than just changing the download_url in setup.py for the latest release. That url only comes into play after you've run "python setup.py register" IIRC. Also, I believe Robey had already removed the download url for the 1.7.6 release. What's causing the delay is pip/easy_install parsing the download urls for all previous versions of pypi. I spent some time figuring this out last year, see my post above (10/29/2010). In short what needs to happen is for Robey (or chrisglass, or me, or I don't care who) to upload the sources and remove the download url for all previous versions from paramiko's pypi page.

[jtriley]

@goosemo: it works because lag.net is currently up. If it goes down you'll see that pip/easy_install'ing paramiko will stall for a while until it times out on all the old versions. This is why I repeatedly mention needing to update the old versions on pypi. I posted the verbose run of pip when lag.net was down above a while back (10/27/2010). Have a look for yourself.

[goosemo]

@jtriley are you still seeing timeouts? I just did a pip install and everything went through w/o any issues.

$ pip install paramiko
Downloading/unpacking paramiko
  Downloading paramiko-1.7.6.zip (215Kb): 215Kb downloaded
  Running setup.py egg_info for package paramiko
    warning: no files found matching '*' under directory 'docs'
Downloading/unpacking pycrypto>=1.9 (from paramiko)
  Downloading pycrypto-2.3.tar.gz (331Kb): 331Kb downloaded
  Running setup.py egg_info for package pycrypto
...

Works for easy_install too.

$ easy_install paramiko 
Searching for paramiko
Reading http://pypi.python.org/simple/paramiko/
Reading http://www.lag.net/\~robey/paramiko/
Reading http://www.lag.net/paramiko/
Best match: paramiko 1.7.6
Downloading http://pypi.python.org/packages/source/p/paramiko/paramiko-1.7.6.zip\#md5\=b1cfe0cd55772115f808a11c1baba8a0
Processing paramiko-1.7.6.zip
Running paramiko-1.7.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-wck2uV/paramiko-1.7.6/egg-dist-tmp-sAeMt0
warning: no files found matching '*' under directory 'docs'
zip_safe flag not set; analyzing archive contents...
Adding paramiko 1.7.6 to easy-install.pth file

Installed /home/mgoose/.virtualenvs/paramiko/lib/python2.6/site-packages/paramiko-1.7.6-py2.6.egg
Processing dependencies for paramiko
Finished processing dependencies for paramiko

So really what is the problem?

[jtriley]

@goosemo: Yes it works RIGHT NOW, but what about when lag.net goes down in the future? That's my point and that's what needs to be fixed ASAP.

[goosemo]

No it works fine even if lag.net is down:

$ grep lag.net /etc/hosts     
127.0.0.1 lag.net

$ wget http://www.lag.net/~robey/paramiko/
--2011-04-20 10:36:58--  http://www.lag.net/~robey/paramiko/
Resolving www.lag.net... 173.255.213.140
Connecting to www.lag.net|173.255.213.140|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://robey.lag.net/paramiko/ [following]
--2011-04-20 10:36:58--  http://robey.lag.net/paramiko/
Resolving robey.lag.net... 173.255.213.140
Reusing existing connection to www.lag.net:80.
HTTP request sent, awaiting response... 404 Not Found
2011-04-20 10:36:58 ERROR 404: Not Found.

$ easy_install paramiko
Searching for paramiko
Reading http://pypi.python.org/simple/paramiko/
Reading http://www.lag.net/\~robey/paramiko/
Reading http://www.lag.net/paramiko/
Best match: paramiko 1.7.6
Downloading http://pypi.python.org/packages/source/p/paramiko/paramiko-1.7.6.zip\#md5\=b1cfe0cd55772115f808a11c1baba8a0
Processing paramiko-1.7.6.zip
Running paramiko-1.7.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-qmM9y3/paramiko-1.7.6/egg-dist-tmp-VcTKON
warning: no files found matching '*' under directory 'docs'
zip_safe flag not set; analyzing archive contents...
Adding paramiko 1.7.6 to easy-install.pth file

Installed /home/mgoose/.virtualenvs/paramiko/lib/python2.6/site-packages/paramiko-1.7.6-py2.6.egg
Processing dependencies for paramiko
Finished processing dependencies for paramiko

[jtriley]

@goosemo: that test isn't quite right, www.lag.net is still accessible (try pinging with that line in your hosts file). Instead try this:

% grep lag /etc/hosts
129.0.0.1 www.lag.net

% easy_install paramiko
Searching for paramiko
Reading http://pypi.python.org/simple/paramiko/
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/~robey/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: timed out -- Some packages may not be found!
Reading http://www.lag.net/paramiko/

....

And it's still running trying to download from lag.net as I type this message...

[goosemo]

Works fine here:

$ grep lag.net /etc/hosts
129.0.0.1 lag.net
129.0.0.1 www.lag.net

$ easy_install paramiko 
Searching for paramiko
Reading http://pypi.python.org/simple/paramiko/
Reading http://www.lag.net/\~robey/paramiko/
Download error: [Errno 113] No route to host -- Some packages may not be found!
Reading http://www.lag.net/paramiko/
Download error: [Errno 113] No route to host -- Some packages may not be found!
Best match: paramiko 1.7.6
Downloading http://pypi.python.org/packages/source/p/paramiko/paramiko-1.7.6.zip\#md5\=b1cfe0cd55772115f808a11c1baba8a0
Processing paramiko-1.7.6.zip
Running paramiko-1.7.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-pKa4g9/paramiko-1.7.6/egg-dist-tmp-qQiqaf
warning: no files found matching '*' under directory 'docs'
zip_safe flag not set; analyzing archive contents...
Adding paramiko 1.7.6 to easy-install.pth file

Installed /home/mgoose/.virtualenvs/paramiko/lib/python2.6/site-packages/paramiko-1.7.6-py2.6.egg
Processing dependencies for paramiko
Finished processing dependencies for paramiko

Perhaps your easy_install is old:

$ easy_install --version
distribute 0.6.14

[jtriley]

@goosemo: that didn't work 'fine', it just eventually worked after timing out. You still suffered two timeouts. Regardless, expecting that users will have a recent easy_install is not a solution. Users can still encounter just what I did and the point is that removing lag.net completely from paramiko's pypi's page would solve this for ALL versions of easy_install, so why not do that?

[olt]

You both have different network configurations. @goosemo you need to make this test with a host that is actually down (not a host with a server, not an unreachable IP, or a host that rejects the connection). During the long downtime of lag.net in October pip timed out like in jtrileys example and not just two times, but a few dozen times.

[chrisglass]

How about giving any one of us pypi rights on the package, and we'll do it ourselves? I already suggested this, so did @jtriley , it's really damn easy to fix it once and for all for everybody (you, me, and even @goosemo will save his timeouts even if he doesn't seem to get it)

Pretty please?

[goosemo]

@olt so one that responds but doesn't have the right data? My configuration was the same as @jtriley's was after he suggested I use 129.0.0.1.

@jtriley I would like it to be solved as well, but demanding it is not the right way to do that. ASAP for busy developers is more probably going to trigger an ignore reflex.

Since there are so many ways to get around this, I really don't see what is causing people to be heated about it. New easy_install times out fine, you can give it a git repo as a source, for both easy_install and pip. I do admit pip's timeout was arduously long, but I feel that's an issue with pip.

More than anything, it would be better to try and get this pull request reopened and merged in, as that's what this whole thread is about.

[goosemo]

@chrisglass do you even need pypi rights? Isn't removing the download link in setup.py enough for lag.net to be sidestepped?

[olt]

@goosemo: No, a server that does not respond. A server where you never get an answer, neither from the server itself nor from any router. A server where you have to wait till it times out (60 second by default IIRC). You have a different network and some router immediately returns 'No route to host', that's not what happens when a server is down like lag.net was. It's not something easy_install or pip can solve.

[jtriley]

@goosemo: I'm not demanding anything, this is simply an issue that needs to be resolved and yes ASAP given that it affects other projects and also looks bad for paramiko when a user must sit through all those timeouts. lag.net should not be referenced in the pypi page, period for any version. The pull request you mentioned will not fix the issue. It will fix the lag.net download url for future releases but does not fix anything on pypi as it stands now for all the reasons we've mentioned in the past on this issue. Go back and read this thread carefully. The old versions need to be updated as well.

[robey]

wow, this thread is huge. is anyone still having problems getting paramiko?

[jtriley]

@robey: not at the moment but as soon as lag.net goes down this thread will continue due to timeouts. As discussed, all paramiko version entries on pypi.python.org need to have the download url removed and the sources uploaded to pypi to fix this once and for all. I'm willing to help with this and @chrisglass has also volunteered if you're willing to give one of us access to the pypi page temporarily.

[bitprophet]

Caveat: I have only skimmed the thread. It is long! But I think I got the gist: there's no longer a download_url going forwards, so modern uploads of Paramiko aren't affected when lag.net goes down, but the historical entries on PyPI require updating so setuptools doesn't crawl them during install?

I'm now maintaining Paramiko & would like to help get this fixed.

@jtriley and @chrisglass, are either of you still active & willing to help with this? I am willing to give one of you temporary PyPI access to help out, and thanks for the offer.

[jtriley]

@bitprophet You nailed it ;) Finally someone besides @chrisglass and myself understands the issue. I'm definitely still active with paramiko/ssh (core dep for https://github.com/jtriley/StarCluster) and I'm more than happy to help out either by doing this myself or guiding you through it on #IRC. Whatever works best for you. I'll ping you on freenode soon...

[jtriley]

@bitprophet As discussed on IRC I was able to find the original downloads for all paramiko versions on lag.net/paramiko/downloads so I uploaded them instead of removing any versions from PyPI. This issue should now be fixed and you can officially remove me from the maintainer list on PyPI at your convenience. Thanks for giving me temporary access and resolving this issue!

[bitprophet]

Huge props to @jtriley for the above :)