Client: add ability to count the number of matches for a filter using binary search over pagination #1925
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1924.
This PR implements a workaround for the client to count the number of results returned by a filter using binary search over pagination. When a database does not return
meta->data_returned
, for whatever reason, the client will now execute the query with a series of probepage_offset
values to find the number of matching results, for example, starting with a query for page 1,000,000, and finding no results, the client will then try 1,000. If a result is found on that page, the window has been narrowed to 1,000 to 1,000,000. The window will be reduced logarithmically until each end of the window has the same approximate power of 10, at which point the average value will be taken, e.g., if the query returned 1,001 entries, the trial values would be:I think this scheme makes more sense than vanilla binary search or exponential search as OPTIMADE queries of this kind are probably either smallish or largeish, without much middle ground...
Caveats:
page_offset
;page_number
is possible but not yet implementedPR also adds a verbosity flag
-vvv
that enables some debug printing on the client.