Can't paginate using elasticsearch_dsl

[mamikonyana]

When I want to paginate through the search results, not iterate as the scan does from elasticsearch.helpers.

My current workaround is something like this

search = Search()
...
# construct your search query
...
result = search.params(search_type='scan', scroll='1m').execute()
scroll_id = result.scroll_id
# Now start using scroll_id to do the pagination,
# but I have to use Elasticsearch.scroll which returns dictionaries not a Result object
client = connections.get_connection()
while data_to_paginate:
  result = Response(client.scroll(scroll_id, scroll='1m'))

There probably should be a helper function that should abstract at least the following part

client = connections.get_connection()
result = Response(client.scroll(scroll_id, scroll='1m'))

Maybe even getting the scroll_id from the result. Basically the user probably shouldn't be getting a client and manually constructing a Response object.

@honzakral what do you think? If we agree on the interface I could implement that since I am probably going to do that for my project.

[honzakral]

There is already a support for pagination:

s = Search()
s = s.query().filter().....
s = s[10:20]
s.execute()

will work for pagination. The slicing of the Search object will add from/offset args to the search body which should help. The only question I am unsure of is whether the slicing should also execute the search (currently it doesn't because of aggregations).

using scroll for pagination is not ideal because it has non-trivial overhead on the elasticsearch side. That said, there is a .scan method on the Search object that can actually do that thing already if you need it.

What do you think?

[mamikonyana]

slicing is great, but it uses from/offset and that means that deep paginations are very costly.

I need to paginate through pretty much my entire index, that's why i need to use scan. I also don't care about scoring the date so this is the ideal case for scan.

I think your current design that you only communicate with the server on .execute is great. And I can't figure out a good equivalent for the scan since first time I execute with search_type='scan' it only returns me the scroll_id and I have to have a second call. In order to make it similar I could add .scroll to the Search class so one could construct

s = Search()
s = s.scroll(scroll_id='some_id')
s.execute()

And this would return one page from the scanning session with corresponding scroll_id.

[honzakral]

have you tried just calling the scan method on the Search object? it will return a generator iterating over all the documents matching the query completely hiding the scroll_id mechanic.

[mamikonyana]

That works for iterating over whole data. But I want to have pagination, because I'm exposing my elasticsearch data over a REST API and I want to send data over to my API users chunk by chunk (so they don't have to send a request per object and I can't send them all the data at once). The scan helper isn't suited very well for my use case.

[honzakral]

Well, in that case I am afraid you have to replicate a lot of the functionality and use the scroll api directly using a scroll_id that you store somewhere.

You can always take the result of the scroll and feed the output to Response (response = Response(es.scroll(scroll_id=XYZ, scroll='1m'))). You will then be able to access the new scroll_id as response._scroll_id, all the hits will still be available as usual - as response.hits or you can just iterate over the response.

[mamikonyana]

ok, cool. So you think this is a very weird use case and it shouldn't get into elasticsearch_dsl?

[honzakral]

Let's leave this ticket here and if it attracts other people we will add a helper. So far I feel that the workaround is very simple so there is no need.

I would also don't think this is a very common use case - either people case about score and then need to use normal pagination or they want to just grab all the data so scan is ideal for them.

[mamikonyana]

sounds good

[honzakral]

Thanks, closing. If any more people find this and would find it useful, please leave comment.

[phretor]

@honzakral I find it very useful, but it would be nice to have a mention of the "scroll" attached to it in the documentation.

http://elasticsearch-dsl.readthedocs.org/en/latest/search.html?q=scroll&check_keywords=yes&area=default

[honzakral]

Thanks, makes perfect sense. I added a note in d018548

[brizzbane]

Found this thread after elasticsearch's suggestion to look at scroll.

Question is, how can I scan() over all results in a DocType?

Normally do:

 docs = DocType.search().filter().execute()
 for d in docs:
     username, domain = d.email.split('@')

Now that I'm doing:

 docs = DocType.search().filter().scan()
 for d in docs:
      username, domain = d.email.split('@')

once it processes 4,700 or so docs, the next iteration of the loop says:

    user, domain = email.split('@')
ValueError: need more than 1 value to unpack

Which I assume is because scan() is getting the next chunk of data. So then I thought do a while len(docs) > 0: and pop from list, but there is no len for generator object.

Thanks!

[saurfangg]

+1 for a more default ES-DSL implementation for pagination :)

[honzakral]

I am sorry, I missed this comment. @brizzbane the error you posted is in your data - you have an email field that doesn't contain any @ signs. It has nothing to do with elasticsearch or the library.

@0x11 I will be happy to help, but I am not sure what is needed - you can easily paginate just using standard slicing on a search object, tools like the Django paginator work seamlessly with it too. With the only exception of having to call execute if you also want aggregations (https://github.com/HonzaKral/es-django-example/blob/master/qa/views.py#L29-L36).

Is there anything in particular that you are missing? Thanks!