Don't treat sites with 403 status codes as broken links?

[mre]

In a recent test of the top 1000 websites, I found the following 403 status errors:

✗ [403] https://cell.com/ | Failed: Network error: Forbidden
✗ [403] https://ticketmaster.com/ | Failed: Network error: Forbidden
✗ [403] https://kraken.com/ | Failed: Network error: Forbidden
✗ [403] https://media.giphy.com/ | Failed: Network error: Forbidden
✗ [403] https://yummly.com/ | Failed: Network error: Forbidden
✗ [403] https://bhphotovideo.com/ | Failed: Network error: Forbidden
✗ [403] https://inc.com/ | Failed: Network error: Forbidden
✗ [403] https://science.sciencemag.org/ | Failed: Network error: Forbidden
✗ [403] https://support.cloudflare.com/ | Failed: Network error: Forbidden
✗ [403] https://docs.wixstatic.com/ | Failed: Network error: Forbidden
✗ [403] https://deviantart.com/ | Failed: Network error: Forbidden
✗ [403] https://use.fontawesome.com/ | Failed: Network error: Forbidden
✗ [403] https://yoursite.com/ | Failed: Network error: Forbidden
✗ [403] https://upwork.com/ | Failed: Network error: Forbidden
✗ [403] https://gleam.io/ | Failed: Network error: Forbidden
✗ [403] https://kickstarter.com/ | Failed: Network error: Forbidden
✗ [403] https://static.wixstatic.com/ | Failed: Network error: Forbidden
✗ [403] https://onlinelibrary.wiley.com/ | Failed: Network error: Forbidden
✗ [403] https://ericsson.com/ | Failed: Network error: Forbidden
✗ [403] https://journals.sagepub.com/ | Failed: Network error: Forbidden
✗ [403] https://kstatic.googleusercontent.com/ | Failed: Network error: Forbidden
✗ [403] https://coinbase.com/ | Failed: Network error: Forbidden
✗ [403] https://zazzle.com/ | Failed: Network error: Forbidden
✗ [403] https://thelancet.com/ | Failed: Network error: Forbidden
✗ [403] https://bluehost.com/ | Failed: Network error: Forbidden
✗ [403] https://tandfonline.com/ | Failed: Network error: Forbidden
✗ [403] https://event.on24.com/ | Failed: Network error: Forbidden
✗ [403] https://hostgator.com/ | Failed: Network error: Forbidden
✗ [403] https://pixabay.com/ | Failed: Network error: Forbidden
✗ [403] https://avvo.com/ | Failed: Network error: Forbidden
✗ [403] https://pexels.com/ | Failed: Network error: Forbidden
✗ [403] https://canva.com/ | Failed: Network error: Forbidden
✗ [403] https://sciencemag.org/ | Failed: Network error: Forbidden
✗ [403] https://codepen.io/ | Failed: Network error: Forbidden
✗ [403] https://fastcompany.com/ | Failed: Network error: Forbidden
✗ [403] https://homedepot.com/ | Failed: Network error: Forbidden
✗ [403] https://udemy.com/ | Failed: Network error: Forbidden
✗ [403] https://oecd.org/ | Failed: Network error: Forbidden
✗ [403] https://capterra.com/ | Failed: Network error: Forbidden
✗ [403] https://nejm.org/ | Failed: Network error: Forbidden
✗ [403] https://creativemarket.com/ | Failed: Network error: Forbidden
✗ [403] https://s-media-cache-ak0.pinimg.com/ | Failed: Network error: Forbidden

These are websites, which block lychee for one reason or another. For example, they use browser fingerprinting, bot detection, block unknown user agents etc. The list is long.

Given that most of these links work, I vote for treating websites with 403 status codes as non-failing links to reduce the number of false positives. Any objections?

[Techassi]

In my opinion, this behaviour should be configurable. There already is the option to set accepted status codes with --accept. Maybe this feature can be improved with something like this:

accept = ["100..=103", "200..=299", "403"]

With the default being something like:

accept = ["100..=399"]

[mre]

Why not use

accept = ["100..=103", "200..=299", "403"]

as the default?

[Techassi]

Oh yeah. That could also work. It might even be better than my suggestion! I will implement the range selectors in the accept config field / CLI flag if we agree.

[mre]

Yup, I think that sounds reasonable. Go ahead if you like.

[mre]

This is fixed in a way more general and extensive way by @Techassi in #1167. 🥳
Now we also support ranges of status codes, e.g. --accept 200..=206,403 would accept 200 until (including) 206 and 403. Thanks!

[denis-domanskii]

@mre, I think it was a mistake.

I've been using lychee for years and today, because of this change, I missed many 403 links in production. I am using lychee to check link on our website, and some of the links were internal because of page author mistake, returning 403 to our external users. But lychee said that all links are SUCCESS.

lychee is a link checker, it verifies that links are working and accessible, which is wrong in 403 case, because they are not, and a real user can't access the target resource. Such links must be marked as failed.

We need to exclude 403 from default success codes.

[mre]

For reference, the code is here:

lychee/lychee-lib/src/types/accept/selector.rs

Lines 43 to 51 in 13f4339

	impl Default for AcceptSelector {
	fn default() -> Self {
	Self::new_from(vec![
	AcceptRange::new(100, 103),
	AcceptRange::new(200, 299),
	AcceptRange::new(403, 403),
	])
	}
	}

As a workaround, you can overwrite the status codes, of course:

lychee --accept 200 ...

Seems like a lot of sites fail because of browser fingerprinting (Cloudflare protection), as described here:
raviqqe/muffet#189

Survey

I took a brief look into how other link checkers handle the situation:

• muffet: ❌ error, planning to add override
• broken-link-checker: ❌ error, workaround for override
• linkinator: ❌, no override
• linkchecker: ❌, no override
• fink: ❌, no override
• markdown-link-check: ❌, override (--aliveStatusCodes)

Summary

So I think we should revert the change. I'm willing to accept a pull request.

[denis-domanskii]

Thank you, I'll do the change tonight.