New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyPI backend and redirections #38

Closed
gracinet opened this Issue Apr 27, 2013 · 11 comments

Comments

Projects
None yet
3 participants
@gracinet

gracinet commented Apr 27, 2013

Hi, thanks for pypiserver, simple and useful !

Here's a tricky problem when running pypiserver with fallback to a mirror index, not sure if it should be fixed within pypiserver or not.
See also https://bitbucket.org/loewis/pep381client/issue/22

The main PyPI server does some 301 redirections (to fix some mispellings). That's not something a mirror can easily reproduce (see above cited issue on pep381client).
The combination of pypiserver and a mirror leads to installation errors.
Fixing one's spellings is not practically possible, because they occur a lot through dependencies.

Reproduction

Assume for instance that pypiserver does not hold packages for 'babel'.

Client applications (such as pip) try e.g., '/simple/babel' (proper spelling is 'Babel')

  • On the main PyPI, they get a 301 response to '/simple/Babel' -> ok
  • On a PyPI mirror, they get a 404, then they fetch the whole '/simple/' page, analyse it, find 'Babel' in it, correct the spelling and retry -> inefficient, but ok
  • On pypiserver with fallback on a mirror, they get:
    • a 303 to the mirror ('/simple/babel')
    • then, a 404 from the mirror
    • then they fetch the '/simple' page from pypiserver, which does not list the wished distribution ('Babel') -> ERROR

Workarounds

My workaround for now is to setup 301 redirections for commonly mispelled distributions in the mirror as well (in our case, the mirror is ours), but that's a maintenance burden and does not scale.

Another possibility is to manually add commonly mispelled distributions in pypiserver and use '-U'. That's insatisfactoy for the same reasons.

Conclusion

Mispellings in dependencies are really common, since many distribution authors don't even realize their spelling is wrong. Any idea ?

@gracinet

This comment has been minimized.

Show comment
Hide comment
@gracinet

gracinet Apr 27, 2013

Maybe two new options --cache-permanent-redirections and --permanent-redirections-url, the latter defaulting to http://pypi.python.org/simple/, performing a light HEAD request to know about redirections would do the trick in most cases.

But that's not really keeping pypiserver simple.

gracinet commented Apr 27, 2013

Maybe two new options --cache-permanent-redirections and --permanent-redirections-url, the latter defaulting to http://pypi.python.org/simple/, performing a light HEAD request to know about redirections would do the trick in most cases.

But that's not really keeping pypiserver simple.

@schmir

This comment has been minimized.

Show comment
Hide comment
@schmir

schmir May 1, 2013

Contributor

Thanks for your detailed bug report.

If I had to fix that, I'd probably let pypiserver read a text file containing all the package names served by the mirror with the right case. We could then easily fix misspellings and redirect to the right URL on the mirror.

I don't feel like making pypiserver do HTTP requests, so I reject your proposed solution.

Isn't there an easy way to fix the mirror?

Contributor

schmir commented May 1, 2013

Thanks for your detailed bug report.

If I had to fix that, I'd probably let pypiserver read a text file containing all the package names served by the mirror with the right case. We could then easily fix misspellings and redirect to the right URL on the mirror.

I don't feel like making pypiserver do HTTP requests, so I reject your proposed solution.

Isn't there an easy way to fix the mirror?

@gracinet

This comment has been minimized.

Show comment
Hide comment
@gracinet

gracinet May 2, 2013

On 05/01/2013 10:36 PM, Ralf Schmitt wrote:

Thanks for your detailed bug report.

Well, a big part of the motivation was letting people know of that kind
of trap.
Thanks for the reply.

If I had to fix that, I'd probably let pypiserver read a text file
containing all the package names served by the mirror with the right
case. We could then easily fix misspellings and redirect to the right
URL on the mirror.

I don't feel like making pypiserver do HTTP requests, so I reject your
proposed solution.

That's hardly a surprise, but yours above, coupled with

  • an optional script to refresh that text file from the fallback server
    (to launch in cron for those that don't mind HTTP requests from their
    server)
  • ideally, a signal to refresh the spelling mapping

would give a very good enough practical solution.

Isn't there an easy way to fix the mirror?

The authority on the mirror aside, there's no automatic solution :
pep381client mirrors are merely a bunch of static files, served by a
general purpose web server.

One can manually add redirections in that web server configuration for
the most used distributions (what I did for our internal ones), but that
means that people will get errors each time they try and use a
distribution that requires one with an unhandled spelling mistake.
Because the error is quite obscure, user'll lose lots of time or give up
instead of notifying the mirror admin so that he can add the proper
spelling.

Of course, one could also write a WSGI app to put right in front of the
mirror and dynamically fix spelling mistakes, but that's a major
complication setup, a performance loss (nginx is hard to beat), and
pypiserver users would still have to be warned to use it.

gracinet commented May 2, 2013

On 05/01/2013 10:36 PM, Ralf Schmitt wrote:

Thanks for your detailed bug report.

Well, a big part of the motivation was letting people know of that kind
of trap.
Thanks for the reply.

If I had to fix that, I'd probably let pypiserver read a text file
containing all the package names served by the mirror with the right
case. We could then easily fix misspellings and redirect to the right
URL on the mirror.

I don't feel like making pypiserver do HTTP requests, so I reject your
proposed solution.

That's hardly a surprise, but yours above, coupled with

  • an optional script to refresh that text file from the fallback server
    (to launch in cron for those that don't mind HTTP requests from their
    server)
  • ideally, a signal to refresh the spelling mapping

would give a very good enough practical solution.

Isn't there an easy way to fix the mirror?

The authority on the mirror aside, there's no automatic solution :
pep381client mirrors are merely a bunch of static files, served by a
general purpose web server.

One can manually add redirections in that web server configuration for
the most used distributions (what I did for our internal ones), but that
means that people will get errors each time they try and use a
distribution that requires one with an unhandled spelling mistake.
Because the error is quite obscure, user'll lose lots of time or give up
instead of notifying the mirror admin so that he can add the proper
spelling.

Of course, one could also write a WSGI app to put right in front of the
mirror and dynamically fix spelling mistakes, but that's a major
complication setup, a performance loss (nginx is hard to beat), and
pypiserver users would still have to be warned to use it.

@ankostis

This comment has been minimized.

Show comment
Hide comment
@ankostis

ankostis Feb 23, 2015

Member

@gracinet Do you think that a workaround would be to invoke pip with --extra-index-url <pypiserver_url> instead of --index-url <pypiserver_url>?

Member

ankostis commented Feb 23, 2015

@gracinet Do you think that a workaround would be to invoke pip with --extra-index-url <pypiserver_url> instead of --index-url <pypiserver_url>?

@gracinet

This comment has been minimized.

Show comment
Hide comment
@gracinet

gracinet Feb 23, 2015

@ankostis, nice to see some new activity around pypiserver !
About your suggestion, I'm not sure and I don't have an easy way to test it anymore : we dropped the pep381 mirror instance completely since then, the main PyPI became much more robust shortly afterwards (that's not the only problem we had with pep381)

If that helps you, I can try and reproduce again, then tell about the suggestion.

As a side note, we were affected by this problem with pip, but with zc.buildout. I don't remember so well, but maybe the reading of the whole simple/ path was actually implemented in setuptools. I don't know if that would still be true, with the many changes and sanitization that the pypa's been doing since then.

gracinet commented Feb 23, 2015

@ankostis, nice to see some new activity around pypiserver !
About your suggestion, I'm not sure and I don't have an easy way to test it anymore : we dropped the pep381 mirror instance completely since then, the main PyPI became much more robust shortly afterwards (that's not the only problem we had with pep381)

If that helps you, I can try and reproduce again, then tell about the suggestion.

As a side note, we were affected by this problem with pip, but with zc.buildout. I don't remember so well, but maybe the reading of the whole simple/ path was actually implemented in setuptools. I don't know if that would still be true, with the many changes and sanitization that the pypa's been doing since then.

@ankostis

This comment has been minimized.

Show comment
Hide comment
@ankostis

ankostis Feb 23, 2015

Member

@gracinet I wouldn't want to burden you, but i would appreciate it if you remember some details to reproduce the error (even without the the pep381 mirror downstream).

In any case i plan to release the new version v1.1.7 with the new usage-recommendation and then assume this bug to be fixed. In case it is not, please comment on it.

Member

ankostis commented Feb 23, 2015

@gracinet I wouldn't want to burden you, but i would appreciate it if you remember some details to reproduce the error (even without the the pep381 mirror downstream).

In any case i plan to release the new version v1.1.7 with the new usage-recommendation and then assume this bug to be fixed. In case it is not, please comment on it.

@gracinet

This comment has been minimized.

Show comment
Hide comment
@gracinet

gracinet Feb 26, 2015

@ankostis yes, it works with pip --extra-index-url on current pip 6 and setuptools 12 !

The behaviour is otherwise mostly the same than it was with older version with just --index-url, except that pip now issues one deprecation warning.

That being said, we're calling this whole stack mostly through zc.buildout, which does not support multiple indexes (nor does easy_install, as far as I can see). Maybe an extension such as gp.vcsdevelop can provide a workaround, as it relies internally on pip, but I wouldn't bet on that too much, since it's meant to pull from remote VCSes anyway.

Tests done on the same version of pep381 mirror as in 2013 (revived a backup).

gracinet commented Feb 26, 2015

@ankostis yes, it works with pip --extra-index-url on current pip 6 and setuptools 12 !

The behaviour is otherwise mostly the same than it was with older version with just --index-url, except that pip now issues one deprecation warning.

That being said, we're calling this whole stack mostly through zc.buildout, which does not support multiple indexes (nor does easy_install, as far as I can see). Maybe an extension such as gp.vcsdevelop can provide a workaround, as it relies internally on pip, but I wouldn't bet on that too much, since it's meant to pull from remote VCSes anyway.

Tests done on the same version of pep381 mirror as in 2013 (revived a backup).

@ankostis

This comment has been minimized.

Show comment
Hide comment
@ankostis

ankostis Feb 26, 2015

Member

@gracinet When you say "behaviour is otherwise mostly the same", do you mean that you still cannot resolve misspelled dependencies?
Or doy you mean that the pypiserver fails on them, like it used to, but since it is not pip's "main" index, the pip-action completes ok?

Member

ankostis commented Feb 26, 2015

@gracinet When you say "behaviour is otherwise mostly the same", do you mean that you still cannot resolve misspelled dependencies?
Or doy you mean that the pypiserver fails on them, like it used to, but since it is not pip's "main" index, the pip-action completes ok?

@gracinet

This comment has been minimized.

Show comment
Hide comment
@gracinet

gracinet Feb 26, 2015

@ankostis yes I mean misspelled dependencies can't be resolved with pip install --index-url PYPISERVER_URL, but they can with --other-index-url (as you suggested).

With --index-url only, the sequence is exactly the same as first described.
With --other-index-url, one can see pip calling both servers from the beginning and finding the misspelled dependency in the main PyPI.

gracinet commented Feb 26, 2015

@ankostis yes I mean misspelled dependencies can't be resolved with pip install --index-url PYPISERVER_URL, but they can with --other-index-url (as you suggested).

With --index-url only, the sequence is exactly the same as first described.
With --other-index-url, one can see pip calling both servers from the beginning and finding the misspelled dependency in the main PyPI.

@ankostis ankostis added this to the v1.1.7 milestone Feb 27, 2015

ankostis added a commit to ankostis/pypiserver that referenced this issue Feb 28, 2015

docs: Update github-links, suitable python-versions, changes and
rel-date.

- Update maintainers.
- Update python-classifiers.
- Various README edits.
- pypiserver#38: Update suggestion on welcome-file.

ankostis added a commit to ankostis/pypiserver that referenced this issue Feb 28, 2015

docs: Update github-links, suitable python-versions, changes and
rel-date.

- Update maintainers.
- Update python-classifiers.
- Various README edits.
- pypiserver#38: Update suggestion on welcome-file.
- Add ankostis to authors (so Ralf may avoid half harassment for
this release).
@ankostis

This comment has been minimized.

Show comment
Hide comment
@ankostis

ankostis Feb 28, 2015

Member

Thanks a lot, considered resolved with the new --extra-index-url advice.

Member

ankostis commented Feb 28, 2015

Thanks a lot, considered resolved with the new --extra-index-url advice.

@ankostis ankostis modified the milestones: No Action, M1.2.0 Jun 17, 2016

@ankostis ankostis modified the milestones: M1.2.0, No Action Jun 17, 2016

@ankostis

This comment has been minimized.

Show comment
Hide comment
@ankostis

ankostis Jun 17, 2016

Member

#140 implemented redirection when un-normalized packages requested - not misspelled ones, but should probably cover 99% of cases.

Member

ankostis commented Jun 17, 2016

#140 implemented redirection when un-normalized packages requested - not misspelled ones, but should probably cover 99% of cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment