Broken-Link-Checker should divide in errors/warnings

[dkehne]

Motivation

To keep it simple for municipalities the results should be divided not only in valid and invalid, but in valid, error and warning.
"Ungültig" should be renamed in "Fehlerhaft".

Proposed Solution

404 and "Ungültige URL" should be visible in error-section
all other error(-code)s should be visible in warning-section

For reference: This topic was debated earlier in #1695

[timobrembeck]

Wouldn't it make more sense to be more honest with the division and name the categories "Not found / Nicht gefunden" and "Invalid / Ungültig"?

To me, "Warning" sounds a bit like "well, not ideal, but doesn't need my direct attention", which is not at all what this category is about. These errors are very real and the links are just not working in most cases. We can still try to minimize the false positives by contributing to the link checker library:

• Make status message translatable DjangoAdminHackers/django-linkcheck#125
• Reschedule checking of invalid links with HTTP status 429/529 DjangoAdminHackers/django-linkcheck#153
• Fake User-Agent if response is 403? DjangoAdminHackers/django-linkcheck#159

[osmers]

How far do you think we can limit false positives? Bcs from the ones I checked, I'd say about 50% worked just fine. The other 50% were faulty. I know that some of the ones that worked are technically not 100% correct, but it's not something the Kommunen have the time to address.

[timobrembeck]

Hard to estimate without trying it, I already implemented a few improvements in the link check library, which should resolve most of the false positive 403 errors... as soon as they release a new version, we can check all links again and test how much it already improved the situation.

[osmers]

Alright, then let's take that step first and then have a look.
What would be even more interesting are the less understandable codes like: Connection Error: HTTPSConnectionPool(host='www.kl- or New Connection Error: Failed to establish a new connection: [Errno -2] Name or service not known (both examples from Kaiserslautern, in case you wanna have a look) bcs both pages work just fine when I click on the link
Those would be pages we'd definitely want to put under warning since they almost always work...

[timobrembeck]

Hmm, those are quite unique problems:

• https://www.baf%C3%B6g.de/de/bafoeg-auch-ohne-deutschen-pass-591.php
  This is an encoding problem of the umlauts in the domain (e.g. https://www.xn--bafg-7qa.de/de/bafoeg-auch-ohne-deutschen-pass-591.php would work), not sure if this is incorrect and just happens to work in the browser because browser often times make incorrect things work, or if this is totally valid and just the library cannot handle it. I will open an issue in the library for this and further investigate.
• https://www.kl-inklusiv.de/
  They use a very strange web server which does not really comply with HTTP standards it seems:

curl -I https://www.kl-inklusiv.de/
HTTP/1.1 404 Not Found
Date: Mon, 06 Mar 2023 11:57:56 GMT
Server: waitress
Content-Length: 26
Content-Type: application/json
Via: waitress
X-Powered-By: Zope (www.zope.org), Python (www.python.org)

[osmers]

Thanks for checking those out - I think it is especially these kinds of problems that Kommunen will come to us about bcs to them there is no apparent error with the page, which is why I would put those under warning (possibly with the explanation of what the warning category means)

[timobrembeck]

Well, I'm still not convinced we really need a new category for these - we already have a category for false positive errors: the "ignore" category.

There will always be certain edge cases where the checker does not behave exactly like a user's browser. There is no need for region managers to contact the service team in such cases - they should just ignore a link after they manually checked it. Maybe we should put this information in the wiki? Or directly on the link checker page in the CMS?