--find-links should not warn about missing HTML5 doctype

[virtuald]

Description

This is not a duplicate of #10825 -- I've moved a number of my comments from #10825 as this is a separate issue. That issue is about enforcing PEP 503, which states that servers implementing the python simple index protocol should have an HTML5 doctype. This is not about that.

---

When using --find-links, this warning can appear:

× The package index page being used does not have a proper HTML doctype declaration.
╰─> Problematic URL: https://www.tortall.net/~robotpy/wheels/2022/roborio/

note: This is an issue with the page at the URL mentioned above.
hint: You might need to reach out to the owner of that package index, to get this fixed. See https://github.com/pypa/pip/issues/10825 for context.

Currently the --find-links documentation says:

  -f, --find-links <url>      If a URL or path to an html file, then parse for links
                              to archives such as sdist (.tar.gz) or wheel (.whl)
                              files. If a local path or file:// URL that's a
                              directory, then look for archives in the directory
                              listing. Links to VCS project URLs are not supported.

There is no HTML5 doctype requirement mentioned.

---

To me, --find-links serves a very different purpose than a full-up pypi-style index implementation. For environments where a full python index is too much work (or in corporate environments where working with IT is really difficult), it's very convenient to stick a bunch of files on a webserver and be able to point pip at an arbitrary directory listing and install packages from that directory. Unfortunately, the most popular webservers in the world (and even python's default http.server!) do not put an HTML5 doctype by default, because it simply does not matter if all you're doing is trying to show a directory listing so users can download a file.

You might say, that it's currently only a warning, and it'll be a long time until we make it an error! But it's a useless warning, and the only way to fully resolve this is to go to every webserver vendor in the world and tell them that they must use an HTML5 doctype in their directory listings because pip says so. And then those changes need to be backported to 'stable' linux distributions like RHEL.

In many corporate environments, developers don't get a choice of which webserver IT is using, and so this warning is just unnecessary noise and will waste hundreds of hours for developers and ops teams.

Production-quality web servers that don't emit HTML5 doctype by default

• Apache2 (likely most popular webserver in the world) does not use an HTML5 doctype by default
• nginx (likely the second most popular) does not even have configurable autoindexing capabilities

Others that don't

• python's default webserver
• Twisted web framework
• golang's default webserver doesn't even bother adding a doctype of any kind
• WEBrick, a ruby webserver
• busybox httpd directory index

Those that do

• Lighthttpd, since late 2016
• Apache Tomcat, since late 2019

---

I appreciate that html5lib adds a lot of work for pip maintainers. If there's a way to use http.parser and ignore the doctype (which the migration from an error to a warning indicates that it is), it seems like that would save hundreds (thousands?) of person-hours for ops teams all around the world who would need to figure out how to reconfigure their webservers because pip is being unnecessarily picky.

Thanks for your consideration.

Expected behavior

No warning

pip version

22.0.3

Python version

3.10

OS

any

How to Reproduce

N/A

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This is a good point. I agree that --find-links should not enforce the doctype restriction.

A PR to fix this would be welcome.

[pradyunsg]

Honestly, I'd be fine with dropping the doctype check entirely as well.

[ppena-LiveData]

Dropping the doctype check would also fix Issue #10880, so I whole-heartedly agree with that idea.

[pombredanne]

My 2 cents, in full agreement and carrying over my comment from #10825 (comment)

FWIW, a plain http directory listing now returns a warning when used with --find-links https://thirdparty.aboutcode.org/pypi/ as a way to supplement PyPI with extra pre-built wheels missing on PyPI.

Looking in links: https://thirdparty.aboutcode.org/pypi/
warning: bad-index-doctype

This is the same issue as reported several times above
Now there is a perfectly valid doctype there:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

May be not an HTML5 type ... but a valid HTML type. I would really appreciate that this deprecation does not force everyone to do busy work for something which is a non-issue IMHO ;)

In practice a warning on doctype is IMHO counterproductive and should be dropped. In the vast majority of the cases, a user will not be able to do anything about it, so that's like crying wolf 🐺
It is similar to having Firefox always warn you about the quirks of the web. That would not be fun and useful. Stated another way, it NOT a client responsability to fix the upstream server, and definitely not pip's role to police PyPI repositories.

[pombredanne]

@pradyunsg re:

Honestly, I'd be fine with dropping the doctype check entirely as well.

I think this makes the most sense to me.

[fungi]

Repeating my day-of comment from the other issue here since it was really about working around --find-links issues, and because the HTML 3.2 doctype mentioned above looks almost certain to be from Apache mod_autoindex: If it helps anyone in a similar situation, here's what we did in the OpenDev Collaboratory in order to coerce Apache's mod_autoindex into returning file lists for our wheel cache which pip 22.0 would tolerate: https://opendev.org/opendev/system-config/commit/e61f584

In short, configure it not to include its normal preamble, and add a custom header file which has the HTML 5 doctype in it instead.

[pombredanne]

@fungi Thanks! this is very useful, but IMHO we should not require the user of a web site to modify its core config just to serve plain links.

[fungi]

@fungi Thanks! this is very useful, but IMHO we should not require the user of a web site to modify its core config just to serve plain links.

Yes, thanks, I should have been clear that we did it as a (hopefully temporary) workaround so that things wouldn't remain broken for users of latest pip while its maintainers work through this.