Add warning category to broken link checker

[david-venhoff]

Motivation

For some pages the link checker shows an error, even though the page technically works for users. For this reason, a warning category should be added.

Proposed Solution

Add a warning category which should include the following links:

• Links to pages with status code 3xx
• Links with invalid fragment
• Links to pages with status code 403 because the page might just not be available to the link checker, even though it works for users.

[Gaston69]

I would like to catch this issue.
Can anybody please give examples for the mentioned links:
3xx, 200 with invalid fragments, 403

I would add them to the fixtures.

And where are the links saved (db-table)?

[timobrembeck]

And where are the links saved (db-table)?

We use django-linkcheck for this tasks, this library ships two models: Url which contains the url string, the status and message, and Link, which holds a many to many mapping from translation models to the Url model (to de-duplicate the checking process and check urls which appear in multiple pages only once).

I would add them to the fixtures.

Please do not include the real life example links below into our fixtures, they are just for local testing purposes. Instead, I would recommend to use https://httpstat.us/ for the fixtures... you can e.g. generate links like:

• https://httpstat.us/200#broken-anchor
• https://httpstat.us/301
• https://httpstat.us/302
• https://httpstat.us/403

(Only adding the Link objects leads to an inconsistent test database - ideally you would also create a few new page translations which contain these links... then you could also do this the other way round: add the links via the content editor first, then export the data via the dumpdata command)

Can anybody please give examples for the mentioned links:
3xx, 200 with invalid fragments, 403

Some example links for the warning category:

"Broken external hash anchor"

When the message is "Broken external hash anchor", the link usually works, some examples:

• https://www.augsburg.de/umwelt-soziales/gesundheit/coronavirus/diagnosestelle#c1098258
• https://www.vhs-lernportal.de/wws/9.php#/wws/home.php
• https://www.krankenkassenzentrale.de/wiki/international#

So the intention of this warning category would be to encourage content creators to either fix the anchor or remove everything after the #...

"301 Moved Permanently" and "302 Found"

When the message is "301 Moved Permanently" or "302 Found", the link also works, but is just a redirect...

• https://www.zusammengegencorona.de/impfen/basiswissen-zum-impfen/impfmythen/#faqitem=b3a900e6-8fee-5b1d-828b-430395c12ac2
• https://www.vhs-regensburg-land.de/programm/sprachen/kategorie/Alphabetisierung/137?Fsize=0#inhalt
• http://www.amira-pisakids.de/#page=home
• https://dormagen.de/tourismus-freizeit/vereine/#Schützen
• https://www.integrationsbeauftragte.de/ib-de/amt-und-person/informationen-zum-coronavirus#

So content creators could replace the link with the new target to avoid the redirect (which might e.g. be removed in the future, who knows)...

One thing I didn't get is why most of the times the status is True and why it is False sometimes...
I believe this has something to do with that linkcheck cannot follow the redirect, see:
https://github.com/DjangoAdminHackers/django-linkcheck/blob/dfa053fbebd832e17de477dd9653472b775d0261/linkcheck/models.py#L323-L325

But again I'm not sure why...

"403 Forbidden"

These kind of errors are difficult to assess - sometimes, the error is justified, e.g.:

• http://www2.infodienst.bzga.de/bot_Seite3333.html
• http://wushu-coburg.jimdo.com/
• https://www.land.nrw/corona-multilingual

and sometimes, this is just because the link target recognizes our link checker as bot and denies access, even if the link works for normal users:

• https://home.meinestadt.de/augsburg/gesundheit/notdienst-apotheke
• https://munich.mfa.gov.ua/
• https://padlet.com/bbs_germersheim/regal
• https://www.peopleatventure.de/ebook-fuer-azubis
• https://de.indeed.com/Praktikum-Sch%C3%BCler-Jobs-in-Neuss?vjk=6837c9c452516587

So for the 403 status code, I'm undecided whether we should really put this into the "warning" category or keep the current solution of giving the user the option to re-check or ignore the link at any given time if they notice that they are in fact working...

[timobrembeck]

@Gaston69 It appears that this is a problem of the underlying library and not us.
The links which are indeed valid are not correctly recognized as such.
So probably I will open an upstream PR for linkcheck which means we will no longer have to create a separate warning category.

[timobrembeck]

For reference, see the following issues in the linkcheck library:

• Do not mark link as broken when anchor is not found DjangoAdminHackers/django-linkcheck#98
• Incorrect message when redirect has broken anchor DjangoAdminHackers/django-linkcheck#128

After these two have been resolved, the only case that remains is "403 Forbidden", but as I stated earlier, I think we should still leave these cases for the manual review instead of marking all those URLs as potentially valid.