Result window is too large, from + size must be less than or equal to:

[miskolc]

When you get too many pages as a result of a query( say 60.000),and you access the last one of them you will get a response error like this:

no match of right hand side value:
 {:error, 500, %{"error" => %{"caused_by" => %{
"reason" => "Result window is too large, from + size must be less than or equal to: [10000] but was [302060]. 
See the scroll api for a more efficient way to request large data sets. 
This limit can be set by changing the [index.max_result_window] index level setting.", 
"type" => "query_phase_execution_exception"}

You can change the ES config limit as it says but that is not very efficient and certainly not what you would want to do. If you get too many results most search engines never show you all of them. They will display only a limited number of pages( say 1000 if you are Google). So there should be a mechanism for limiting the number of pages.

The solution I came up with is to change the :total_pages key to a certain limit:
page = page |> Map.put(:total_pages, @scrivener_elastic_total_pages)
and this plays nicely with Scrivener_HTML.

[radar]

I agree. Scrivener shows an example of how to do this here.

Could you please submit a patch to add this?

[miskolc]

To be honest, I can't work on it within the next several days( including during the weekend). Maybe I can tackle it next Tuesday or Wednesday, but certainly not until then.

[miskolc]

Ryan, after reading the implementation and trying some changes myself, I don't see how can this issue be resolved solely by enhancing the functionality of this package. I see that in order to be able to pass the new params to use they must be defined in the defstruct of the Scrivener.Config module and returned in the struct of the new method from this module.
I don't see how this can be done without submitting a pull request to Scrivener itself to add an extra option for limiting the number of pages. I've tried to overwrite the Scrivener.Config module but the overwritten functionality doesn't work when the Scrivener.Config is aliased here