Fix problem with false positives (external URLs)

[sypets]

todos

• in brofix: provide information in documentation, but it is not brofix's job to download intermediate certificates
• make it possible to exclude specific error types (e.g. curl(60), but this usually has problem, that legitimate errors are not displayed as well
• use some heuristic for better detection of probable false positives (difficult, because has several reasons and several error types can be affected)

summary

So far, the following reasons for false positives could be verified:

1. certificate chain issue (this is actually an error on the server side of the webserver which is checked, but it is a minor error and page can be loaded without warning in browser, so this is perceived (!) as not broken by user (this should be distinguished from other TLS security isssues, such as outdated certificate etc.)
   • error is usually curl 60 (can be verified by using curl -I "url" on server

curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

* SSLLabs shows "chain issues: incomplete" and "extra download"
* to fix on server side: put complete certificate chain in certifcate (including intermediate certificates)
* to fix on client server side (where brofix is running): download intermediate certificates

2. cloudflare
   • error 503 is given

problem description

some URLs are reported as errors even though they work (in browser)

Examples:

• linkedin: linkedin often, but not always marked as broken
  • Info:
    • https://stackoverflow.com/questions/46214017/public-linkedin-profile-url-returns-server-status-code-999
    • https://stackoverflow.com/questions/27231113/999-error-code-on-head-request-to-linkedin
    • happens with Wordpress Broken link checker as well:
      • https://wordpress.org/support/topic/all-linkedin-links-marked-as-brokenfor-months/
      • https://wordpress.org/support/topic/false-linkedin-error/#:~:text=The%20999%20HTTP%20error%20is,as%20%E2%80%9CNot%20Broken%E2%80%9D%20link.

The 999 HTTP error is a Linkedin error. It happens when Linkedin blocks the User-Agent that tries to access a link. I’m afraid it is an issue from the Linkedin, is streets your site as fake User-Agent.
Since the link is not broken, please feel free to set it as “Not Broken” link.

---

• twitter URLs

---

other

• ...

---

Apart from this, all 401, 403 (access restricted URLs) will fail. In that case, it is not really an error, but expected. For these cases, they could either be added as exclude link target entry, or we could make external link type errors configurable (e.g. have an exclude list for that as well, where you could exclude for example 401, 403, maybe also "too many redirects").

---

see also: https://notes.typo3.org/linkvalidator_problem_external_urls

Related:

• How to deal with external link error types that may be temporary (e.g. timeout, 500) #34

[sypets]

Analysis of some URLs which are causing problems.

Currently brofix sends the following HTTP headers (see TSconfig):

User-Agent: configurable
Accept: */*
Accept-Language: *
Accept-Encoding: *

It looks like the Accept-Language / Accept-Encoding may be causing problems in some cases.

It is possible to simulate this with curl:

curl -IL -H "Accept-Language: *" -H "Accept-Encoding: *"

---

curl sends these headers (by default):

curl -ILv URL

HEAD /pages/de/news411455 HTTP/2
Host: idw-online.de
user-agent: curl/7.68.0
accept: /

be sure to add the -L to follow redirects ....

[sypets]

Reason: Incomplete certificate chain

• curl: (60) SSL certificate problem: unable to get local issuer certificate
• can be verified with Qualys SSL Labs server test (we see: "Chain issues | Incomplete" and "extra download"
• the server should supply all intermediate certificates (except root certificate), but it does not
• browser fetch the other certs or get them from their cache, so they do not show an error
• solution can be to fetch the intermediate certs and make them available on the server

Example:

curl -I "https://www.ylook.de/search.php?&linklist_idx=11116563"
curl: (60) SSL certificate problem: unable to get local issuer certificate

Solutions

0. implicit exclude (consider URLs as not broken for now) (this is not a good solution, as these servers often have other issues as well)
1. Add intermediate certs on server.

This could be done with an extra tool but should not be implemented in brofix.

2. extend client somehow (brofix:ExternalLinkType)
3. extend guzzle somehow, see

• see Replicating browser behaviour with missing Intermediate Certs guzzle/guzzle#848

That is a good start, but instead of extending the client, I suggest creating an event subscriber that can work for both synchronous and asynchronous requests.

• see Guzzle "event subscriber": https://docs.guzzlephp.org/en/5.3/events.html (this page is in older version - 5.3, wehre is "event" docs in newer docs???)

Use custom CA bundle

• https://docs.guzzlephp.org/en/stable/request-options.html#verify-option

Side note

The same error code (curl(60)) may also be the result for more severe TLS / certificate issues.

Same error code but different error message (in command line curl) !

certificate has expired
self-signed certificate

1. "Certificate name mismatch"). Unfortunately, it is not possible to determine which is the case, just from the error message.

Example:

curl -I "https://klimakongressoldenburg.de"
curl: (60) SSL certificate problem: self-signed certificate

Certificate has "Certificate Name mismatch, see Qualis SSL Labs

2. Certificate expired, e.g.

curl -I https://openjournal.uni-oldenburg.de/
curl: (60) SSL certificate problem: certificate has expired

see resources:

guzzle

• https://docs.guzzlephp.org/en/stable/index.html
• Guzzle support, see README: https://github.com/guzzle/guzzle#help-and-docs

other

• tool: chain resolver: https://github.com/zakjan/cert-chain-resolver
• What's my chain cert: https://whatsmychaincert.com/
• https://medium.com/@superseb/get-your-certificate-chain-right-4b117a9c0fce
• https://security.stackexchange.com/questions/143258/how-to-get-hold-of-intermediate-certs-programatically
• https://serverfault.com/questions/844071/cannot-get-curl-or-wget-to-validate-some-ssl-certificates

[sypets]

Reason: probably cloudflare DDoS protection I'm unter attack

• error 503 is given
• if body of webpage is fetched, we see something like cf-im-under-attack

<tr>
      <td align="center" valign="middle">
          <div class="cf-browser-verification cf-im-under-attack">
  <noscript>
    <h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>