Better support for debian/watch files and uscan

[warsaw]

The recent change to remove index pages breaks debian/watch files used by the uscan script on Debian systems to find the upstream tarball used by package maintainers when updating and building their source packages. After discussion with dstufft, I understand why that change was necessary. We'll have to change lots of packages in the Debian Python repository (some of which are not team maintained). We can communicate this to package maintainers, but it would be best to provide a good alternative.

The https://pypi.python.org/simple/package listings which take their place is not very convenient for two reasons. While we can make it work by using debian/watch file contents such as:

version=3
opts="filenamemangle=s|.*/(.*)\#md5.*$|$1|" \
 https://pypi.python.org/simple/wheel/ .*/wheel-(.*)\.(?:zip|tgz|tbz2|txz|tar\.(?:gz|bz2|xz)).*

you can see that this is not as simple as what we used to have:

version=3
https://pypi.python.org/packages/source/w/wheel/wheel-(.*).tar.gz

The first inconvenience is the #md5=blah fragment that all hrefs in the listing contain. uscan is not able to automatically ignore these, although it does ignore query suffixes (e.g. ?md5=blah). This is why we have to have the filenamemangle option.

The second inconvenience is the fact that the hrefs on the simple page point to tar files at a different url, so we need the two-parameter second line.

Both of these issues means that maintainers will be cargo culting the solution, there will be several variants so folks won't know what the details mean, and it will be easy for difficult-to-debug typos to occur. It will also be difficult to explain best practices and give readable examples.

If the urls must change (as I understand they must since you'll be moving away from a file system implementation), then it would be great if we could get simple urls with redirects back. E.g.:

version=3
https://pypi.python.org/simple/wheel/wheel-(.*).tar.gz

This would mean 1) no #md5=blah suffixes, or that # were changed to ? and 2); the hrefs were resolvable to tarfiles on that same page, via redirects if necessary. (Yes, I know not every upstream uses .tar.gz; that's okay don't worry about that part.)

In IRC, dstufft mentioned that the trade-off for this convenience would be the possibility of some short downtime if the backend went down, since the "debian-simple" (or "uscan-friendly") urls wouldn't be cached by the cdn. Since those down times would be short, I can live with that. This is primarily for the convenience of the package maintainers, and really is only needed when they build a source package locally, either for upgrading the Debian version of the package, or perhaps for building a version for which they only have the source branch (which, under svn and other conditions may not have the pristine-tar file readily available). Unless I'm missing something, the worst effect of backend downtime would mean the unlucky package maintainer would just have to wait a little bit.

[dstufft]

This will probably live at https://pypi.python.org/uscan/, but I have questions:

By different URL I assume you mean, what uscan really wants is something like https://pypi.python.org/uscan/wheel/ being an index page with links pointing to https://pypi.python.org/uscan/wheel/wheel-1.0.tar.gz?

I assume there's no special constraints on the markup itself?

To be clear, these URLs will likely be cached, but since they'll have different cache keys and uscan traffic is a lot less then pip/easy_install traffic it's more likely that if downtime occurs that their won't be cached items to return.

[warsaw]

I think that would work well. No special markup, uscan looks for standard <a href="http://pypi.python.org/uscan/wheel/wheel-1.0.tar.gz"> type HTML.

[dstufft]

Going to close this since Debian is setting up a redirector at pypi.debian.net instead.

[lamby]

It would be great if Debian could move away from using a redirector. Could the "new PyPI" support our uscan tool cleanly? :)

[brainwane]

Hi @lamby - I think we just spoke in #debian-python and I just referred to this issue in #2935 . I'd encourage you to check out our APIs/feeds issues and feel free to comment on those or file a new issue to request that PyPI's APIs support uscan better!

[lamby]

@brainwane Hi, thanks for the offer but as I mentioned on IRC I am really not the one to ask for this - I was just throwing ideas around! Sorry about that..