Preliminary support for page_above in base entry collections

[ml-evs]

This PR should make it easier to customize which pagination mechanism each backend supports, by adding a class attribute for it (that follows the names of the query parameters).

All supported pagination parameters are now passed down to the collection (with page_cursor and page_below still left as unimplemented for all collections).

Once this PR is merged, we can work to make page_above the default ES pagination mechanism (i.e., the one that is used in next links) and maybe even consider disabling offset-based pagination for ES. It might be cleaner to move the unsupported params attribute down to the collection itself too.

• Pass page_above through query param handler
• Customisable default pagination mechanism
• Skeleton of ES implementation of page_above that maintains current behaviour for now.
• Update test cases

[codecov]

Codecov Report

Merging #1560 (2700ee7) into master (d779725) will decrease coverage by 0.06%.
The diff coverage is 88.52%.

@@            Coverage Diff             @@
##           master    #1560      +/-   ##
==========================================
- Coverage   91.09%   91.04%   -0.06%     
==========================================
  Files          74       74              
  Lines        4538     4578      +40     
==========================================
+ Hits         4134     4168      +34     
- Misses        404      410       +6     

Flag	Coverage Δ
project	91.04% <88.52%> (-0.06%)	⬇️
validator	90.93% <88.52%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
optimade/server/middleware.py	95.33% <ø> (+0.63%)	⬆️
optimade/server/query_params.py	98.27% <ø> (ø)
optimade/server/entry_collections/mongo.py	95.31% <50.00%> (-1.47%)	⬇️
optimade/server/entry_collections/elasticsearch.py	95.55% <85.71%> (-1.98%)	⬇️
...made/server/entry_collections/entry_collections.py	95.88% <90.00%> (-1.96%)	⬇️
optimade/__init__.py	100.00% <100.00%> (ø)
optimade/server/entry_collections/__init__.py	100.00% <100.00%> (ø)
optimade/server/routers/utils.py	95.93% <100.00%> (-0.04%)	⬇️
optimade/validator/utils.py	94.59% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[markus1978]

More data is only available, if data_returned == len(results). There is the inevitable chance that you will do a request with empty response.

[JPBergsma]

I do not understand your comment.

I think data_returned is the total number of entries that match the filter, while len(results) is the number of returned entries. So if data_returned == len(results) all the matching entries have been returned and no more data is available.

[ml-evs]

Just to clarify here, I was working under a potentially wonky assumption that search_after would mangle the total hits for a filter, but perhaps this is not the case, in which case we can still directly use data_returned to set more_data_available (as suggested by @JPBergsma below).

[JPBergsma]

In the documentation of Elasticsearch, they recommand setting "track_total_hits": false to speed things up.
So I would expect that data_returned would give a sane result as we have set "track_total_hits": true.
We could perhaps include the value of data_returned in the next link, so we can set "track_total_hits": false when retrieving the rest of the data.

[ml-evs]

Hi @markus1978 and @JPBergsma, I think the bones of this are all in now, even if we are not testing page_above for ES.

I think I'd like to merge this PR and then continue working on the actual page_above implementation and tests in another PR.

[JPBergsma]

What I see so far looks good.
Good luck with creating the implementations.