Check pagination links->next with validator

[ml-evs]

Closes #265.

[codecov]

Codecov Report

Merging #266 into master will decrease coverage by 0.16%.
The diff coverage is 70.37%.

@@            Coverage Diff             @@
##           master     #266      +/-   ##
==========================================
- Coverage   89.32%   89.15%   -0.17%     
==========================================
  Files          54       54              
  Lines        2220     2241      +21     
==========================================
+ Hits         1983     1998      +15     
- Misses        237      243       +6     

Flag	Coverage Δ
#unittests	89.15% <70.37%> (-0.17%)	⬇️

Impacted Files	Coverage Δ
optimade/validator/validator.py	68.08% <70.37%> (+0.26%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2eddcb7...3452cb4. Read the comment docs.

[ml-evs]

Having lots of fun getting the docker server under CI to return the right base URL in pagination links...

[ml-evs]

Done now, with a bit of a CI-specific hack to the config file.

[CasperWA]

Great! Thanks @ml-evs !

We're slowly turning our tests over to the validator, as it should be.

I have some comments and suggestions for improvements, hope they help :)

[ml-evs]

Thanks for the review @CasperWA, think I've implemented the spirit of your suggestions if not the exact letter of them :)

[CasperWA]

It seems the validator is not getting the version included in it's calls to the Dockerized server.

[ml-evs]

It seems the validator is not getting the version included in it's calls to the Dockerized server.

Ugh, looks like urllib.parse.urljoin("http://host.com/v0", "/structures?filter=a=1") returns http://host.com/structures?filter=a=1. I'll go back and undo those changes...

[ml-evs]

Think we're finally there, don't really want to make any other tweaks to the validator in this PR as it probably need a full refactoring pass to do it properly.

[CasperWA]

It sure is looking great now, thanks @ml-evs!
I have only a single comment (with included change suggestion) left.
I will approve now, if you don't feel the change is needed, otherwise, please apply and I'll re-approve for you to squash+merge.

[CasperWA]

Do you still want to make sure the scheme is part of ("http", "https") or do we not care? If yes, here's a quick and dirty suggestion:

Suggested change

	if urllib.parse.urlparse(request, allow_fragments=True).scheme:
	request_scheme = urllib.parse.urlparse(request, allow_fragments=True).scheme
	if request_scheme and request_scheme in ("http", "https"):

Otherwise, fine 👍

[ml-evs]

I'm going to leave it as is, on the off-chance that an implementation provides a different scheme (presumably accidentally) then it will be obvious from the error spat out. Your suggested change would lead to the next request being e.g. http://base_url.com/ssh://base_url.com?page_offset=5, so I think it's clearer as is.

[CasperWA]

Sure. Then the docstring should probably have been updated, but fine.
Again, I just want to be really nitpicky with everything in the validator since (in my opinion) it should be usable for every and any implementation. I.e., it should be really specific in what it wants, while at the same time being very broad in scope.. not easy :)