API fails to load pages for items above 10k

[rkorytkowski]

I hit this issue trying to paginate via UI: https://app.qa.openconceptlab.org/#/orgs/CIEL/sources/CIEL/concepts?q=&isTable=true&isList=false&isSplit=false&page=2097&exactMatch=off

The API request is https://api.qa.openconceptlab.org/orgs/CIEL/sources/CIEL/concepts/?q=&page=2097&exact_match=off&limit=25&includeRetired=false&verbose=false&sortDesc=_score

The response says:
RequestError at /orgs/CIEL/sources/CIEL/concepts/
RequestError(400, 'search_phase_execution_exception', 'Result window is too large, from + size must be less than or equal to: [10000] but was [52425]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.')

[snyaggarwal]

@rkorytkowski here is an interesting discussion on this --
https://discuss.elastic.co/t/result-window-is-too-large-from-size-must-be-less-than-or-equal-to-10000-but-was-10050/124392

[rkorytkowski]

I see. So it's an actual limitation we need to handle. One thing is that https://api.qa.openconceptlab.org/orgs/CIEL/sources/CIEL/concepts/?q=&page=2097&exact_match=off&limit=25&includeRetired=false&verbose=false&sortDesc=_score should not be hitting the ES. It's a DB pagination. Does it hit ES because of sortDesc=_score or is it by default?
We need to make sure ES is hit only when actually needed that is for full text queries.

We also need to handle it gracefully, if hitting such an error with ES.

In the UI we should not be displaying all pages, but stop at the ones that are supported and indicate there are more, but not available (maybe grayed out number(s)). When trying to open them we could say: "Only x pages are available. Please apply additional filters or fine tune your query to get more accurate results.". API should respond with a similar message and 400. One other option is to show only pages that are available and at the end of that page say "There are x more pages, which satisfy your query that can not be presented. Please apply additional filters or fine tune your query to get more accurate results.".

[rkorytkowski]

@jamlung-ri @paynejd do you have any thoughts on that? Basically we can only paginate up to a certain page, which for 25 items per page is 400. The limit should apply only when you use facets or type in a text query. Otherwise we should be able to iterate though all pages.

[jamlung-ri]

@snyaggarwal Could you give a quick update on the current state of this? Is the 10000 limit still in place? It'd be great if this is fixed, but if not, we will need to have a discussion.

[jamlung-ri]

We discussed this on today's Arch call.

Next steps to be done in follow-up ticket:

• To inform API users, specify in Docs what his Elastic Search and what limits there are
• To inform UI users, add a URL to these docs to inform why the ES queries are failing after 10k concepts
• Add a log if someone encounters the error, along with what they are querying