Prevent validator errors/retries on read timeouts

[ml-evs]

The last PR (#1051) introduced explicit read timeouts and reduced the default value to 30 seconds. After some testing with the dashboard, 30 seconds is quite a harsh timeout for some implementations/requests, so this is now increased to 2 minutes by default. Also, as requests.exceptions.ReadTimeout does not subclass requests.exceptions.ConnectionError, these requests were failing with "internal" validator errors, rather than being handled as errors related to the implementation.

This PR fixes that error handling, and additionally prevents read timeouts from triggering retries. A CLI arg was also added for --read-timeout for easier testing.

[codecov]

Codecov Report

Merging #1056 (0653af2) into master (b216947) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1056   +/-   ##
=======================================
  Coverage   92.90%   92.90%           
=======================================
  Files          67       67           
  Lines        3790     3790           
=======================================
  Hits         3521     3521           
  Misses        269      269           

Flag	Coverage Δ
project	92.90% <100.00%> (ø)
validator	92.90% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
optimade/validator/__init__.py	100.00% <100.00%> (ø)
optimade/validator/utils.py	94.63% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b216947...0653af2. Read the comment docs.

[CasperWA]

Cheers. Seems you also upped the default readout time to 60 s - I presume this is intentional and not from some local changes and testing? :)

[ml-evs]

Cheers. Seems you also upped the default readout time to 60 s - I presume this is intentional and not from some local changes and testing? :)

Yeah, its in the description above, it seems that some of the bigger validator queries reliably hit 30 seconds for some implementations, so I've upped the default.