PyPI backend and redirections

[gracinet]

Hi, thanks for pypiserver, simple and useful !

Here's a tricky problem when running pypiserver with fallback to a mirror index, not sure if it should be fixed within pypiserver or not.
See also https://bitbucket.org/loewis/pep381client/issue/22

The main PyPI server does some 301 redirections (to fix some mispellings). That's not something a mirror can easily reproduce (see above cited issue on pep381client).
The combination of pypiserver and a mirror leads to installation errors.
Fixing one's spellings is not practically possible, because they occur a lot through dependencies.

Reproduction

Assume for instance that pypiserver does not hold packages for 'babel'.

Client applications (such as pip) try e.g., '/simple/babel' (proper spelling is 'Babel')

• On the main PyPI, they get a 301 response to '/simple/Babel' -> ok
• On a PyPI mirror, they get a 404, then they fetch the whole '/simple/' page, analyse it, find 'Babel' in it, correct the spelling and retry -> inefficient, but ok
• On pypiserver with fallback on a mirror, they get:
  • a 303 to the mirror ('/simple/babel')
  • then, a 404 from the mirror
  • then they fetch the '/simple' page from pypiserver, which does not list the wished distribution ('Babel') -> ERROR

Workarounds

My workaround for now is to setup 301 redirections for commonly mispelled distributions in the mirror as well (in our case, the mirror is ours), but that's a maintenance burden and does not scale.

Another possibility is to manually add commonly mispelled distributions in pypiserver and use '-U'. That's insatisfactoy for the same reasons.

Conclusion

Mispellings in dependencies are really common, since many distribution authors don't even realize their spelling is wrong. Any idea ?

[gracinet]

Maybe two new options --cache-permanent-redirections and --permanent-redirections-url, the latter defaulting to http://pypi.python.org/simple/, performing a light HEAD request to know about redirections would do the trick in most cases.

But that's not really keeping pypiserver simple.

[schmir]

Thanks for your detailed bug report.

If I had to fix that, I'd probably let pypiserver read a text file containing all the package names served by the mirror with the right case. We could then easily fix misspellings and redirect to the right URL on the mirror.

I don't feel like making pypiserver do HTTP requests, so I reject your proposed solution.

Isn't there an easy way to fix the mirror?

[gracinet]

On 05/01/2013 10:36 PM, Ralf Schmitt wrote:

Thanks for your detailed bug report.

Well, a big part of the motivation was letting people know of that kind
of trap.
Thanks for the reply.

If I had to fix that, I'd probably let pypiserver read a text file
containing all the package names served by the mirror with the right
case. We could then easily fix misspellings and redirect to the right
URL on the mirror.

I don't feel like making pypiserver do HTTP requests, so I reject your
proposed solution.

That's hardly a surprise, but yours above, coupled with

• an optional script to refresh that text file from the fallback server
  (to launch in cron for those that don't mind HTTP requests from their
  server)
• ideally, a signal to refresh the spelling mapping

would give a very good enough practical solution.

Isn't there an easy way to fix the mirror?

The authority on the mirror aside, there's no automatic solution :
pep381client mirrors are merely a bunch of static files, served by a
general purpose web server.

One can manually add redirections in that web server configuration for
the most used distributions (what I did for our internal ones), but that
means that people will get errors each time they try and use a
distribution that requires one with an unhandled spelling mistake.
Because the error is quite obscure, user'll lose lots of time or give up
instead of notifying the mirror admin so that he can add the proper
spelling.

Of course, one could also write a WSGI app to put right in front of the
mirror and dynamically fix spelling mistakes, but that's a major
complication setup, a performance loss (nginx is hard to beat), and
pypiserver users would still have to be warned to use it.

[ankostis]

@gracinet Do you think that a workaround would be to invoke pip with --extra-index-url <pypiserver_url> instead of --index-url <pypiserver_url>?

[gracinet]

@ankostis, nice to see some new activity around pypiserver !
About your suggestion, I'm not sure and I don't have an easy way to test it anymore : we dropped the pep381 mirror instance completely since then, the main PyPI became much more robust shortly afterwards (that's not the only problem we had with pep381)

If that helps you, I can try and reproduce again, then tell about the suggestion.

As a side note, we were affected by this problem with pip, but with zc.buildout. I don't remember so well, but maybe the reading of the whole simple/ path was actually implemented in setuptools. I don't know if that would still be true, with the many changes and sanitization that the pypa's been doing since then.

[ankostis]

@gracinet I wouldn't want to burden you, but i would appreciate it if you remember some details to reproduce the error (even without the the pep381 mirror downstream).

In any case i plan to release the new version v1.1.7 with the new usage-recommendation and then assume this bug to be fixed. In case it is not, please comment on it.

[gracinet]

@ankostis yes, it works with pip --extra-index-url on current pip 6 and setuptools 12 !

The behaviour is otherwise mostly the same than it was with older version with just --index-url, except that pip now issues one deprecation warning.

That being said, we're calling this whole stack mostly through zc.buildout, which does not support multiple indexes (nor does easy_install, as far as I can see). Maybe an extension such as gp.vcsdevelop can provide a workaround, as it relies internally on pip, but I wouldn't bet on that too much, since it's meant to pull from remote VCSes anyway.

Tests done on the same version of pep381 mirror as in 2013 (revived a backup).

[ankostis]

@gracinet When you say "behaviour is otherwise mostly the same", do you mean that you still cannot resolve misspelled dependencies?
Or doy you mean that the pypiserver fails on them, like it used to, but since it is not pip's "main" index, the pip-action completes ok?

[gracinet]

@ankostis yes I mean misspelled dependencies can't be resolved with pip install --index-url PYPISERVER_URL, but they can with --other-index-url (as you suggested).

With --index-url only, the sequence is exactly the same as first described.
With --other-index-url, one can see pip calling both servers from the beginning and finding the misspelled dependency in the main PyPI.

[ankostis]

Thanks a lot, considered resolved with the new --extra-index-url advice.

[ankostis]

#140 implemented redirection when un-normalized packages requested - not misspelled ones, but should probably cover 99% of cases.