Using the Elastic DSL iterate with an Index

[AndTheDaysGoBy]

Search.scan was the previous method to utilize the Scroll API via elasticsearch-dsl. However, the Scroll API has, for a lot of functionality, been deprecated in favor of the search_after approach. To facilitate this, elasticsearch-dsl has a Search.iterate method which handles the default pagination for the user, default in that, you can't set the page size.

Now, suppose you have a Search object, you can set the index via Search(index='some-index') or with Search().index('some-index'). Regardless, you have a Search object on an index, for which you can then iterate the documents in said index.

for x in Search(index='some-index').iterate():
     pass # do thing

However, this does not behave as I expect it to. In iterate,

elasticsearch-py/elasticsearch/dsl/_sync/search.py

Line 159 in 44cbf67

def iterate(self, keep_alive: str = "1m") -> Iterator[_R]:

a point-in-time (PIT) is opened up, which makes sense to avoid the data changing under you.

However, my issues lies within the point_in_time method.

elasticsearch-py/elasticsearch/dsl/_sync/search.py

Line 141 in 44cbf67

def point_in_time(self, keep_alive: str = "1m") -> Iterator[Self]:

It opens the point in time with the appropriate index, however, next, notice how it takes the self, i.e. the current Search object and clears the index. It then yields this search object. This might make sense in the situation the doc string describes where you are constructing a point in time for multiple queries, e.g.

with s.point_in_time() as neo_s:
    neo_s.index('a').execute()
    neo_s.index('b').execute()

however, in the context of iterate, this yields issues as, index is never set again. Thus, each /search query done by iterate will be against all indices, which could yield issues if the user doesn't have permissions to read from all indices.

Please correct me if I'm wrong, this is just what seemed to be the issue when I tried to iterate on an index with a user with fixed read permissions.

As an aside, since search_after utilizes the response values of the last hit in a query (per https://www.elastic.co/guide/en/elasticsearch/reference/8.18/paginate-search-results.html#search-after )
I am a bit confused as to why

elasticsearch-py/elasticsearch/dsl/_sync/search.py

Line 175 in 44cbf67

s = s.search_after()

is using the s object from the context manager, as opposed to r from the response. I.e., why it isn't r.search_after() per

elasticsearch-py/elasticsearch/dsl/response/__init__.py

Line 186 in 44cbf67

def search_after(self) -> "SearchBase[_R]":

[AndTheDaysGoBy]

OK, to answer my own question, in part (for the first half regarding indices). It seems the indice information is meant to be encoded with the PIT and then there is no need to actually specify it in the subsequent searches. Although, as I said, I then get indices:data/read/search permissions issues.

I hope it isn't the case that a user requires permission to all indices to be able to use PIT against a specific index. From Google, it appears I could at least limit it to only require permission to have "visibility" to all indices.

[miguelgrinberg]

@AndTheDaysGoBy It would be useful if you provide the exact error(s) you are getting instead of a high-level description of them.

The point in time is created with a cleared index specification. See the example in the docs, which includes the following note:

A search request with the pit parameter must not specify index, routing, or preference as these parameters are copied from the point in time.

So as I said above, it would be useful to have the actual error you are receiving, because this may be a server-side issue and not a problem in this client library.

Regarding your second question, the Search object stores a copy of the last response. So s = s.search_after() is identical to s = r.search_after() in functionality. Is this causing any problems?

[AndTheDaysGoBy]

@miguelgrinberg
Sure, I'll try my best to provide specifics. A DSL Search object is constructed as follows

query = Q('term', system__keyword='my_system') & Q('range', timestamp={'gte': '2025-04-01', 'lt': '2025-05-01'})

# prod is the connection configuration for production
search = Search(
    using='prod', index='my-index'
).query(query).source(
    includes=['message', 'transaction_id']
).sort('timestamp')

for response in search.iterate():
    pass

Within the first call to iterate, at s.execute()

elasticsearch-py/elasticsearch/dsl/_sync/search.py

Line 170 in 44cbf67

r = s.execute()

I receive:
AuthorizationException(403, 'security_exception', 'action' [indices:data/read/search] is unauthorized for user [myuser] with effective roles[<redacted>], this action is granted by the index privileges [read,all])

I know that myuser has read access to my-index, hence my confusion as to what could be the issue, and why I mistakenly assumed there was an issue with the PIT clearing the index. Since that clearing is proper behavior, I'm back at square one of not understanding why iterate requires indices:data/read/search.

For the second question, much appreciated. I only added that part since, I was grasping at straws as to the cause of my issue.
Apologies if this still isn't specific enough.

Edit: I will add, my question is probably more so an elasticsearch API question an an elasticsearch-py question, so, if this issue should be closed, that's fine. However, if someone happened to understand the permissions related to PIT, at least then the answer would be readily available here.

[miguelgrinberg]

Out of curiosity, does your query work if you manually iterate with search_after, without the PIT? For example doing something like this:

query = Q('term', system__keyword='my_system') & Q('range', timestamp={'gte': '2025-04-01', 'lt': '2025-05-01'})

# prod is the connection configuration for production
search = Search(
    using='prod', index='my-index'
).query(query).source(
    includes=['message', 'transaction_id']
).sort('timestamp')

while True:
    r = search.execute()
    for hit in r:
        print(hit)
    if len(r.hits) == 0:
        break
    search = search.search_after()

[AndTheDaysGoBy]

@miguelgrinberg yes, without the PIT, I don't run into issues. (well, beyond not having the refresh-protection PIT provides)

[miguelgrinberg]

@AndTheDaysGoBy what version is your Elasticsearch server at?

I'm using v9.0.2 here and could not reproduce this problem. I created a user miguel with a role only-people that gave read access to my people index. The server also has a test index.

This is my test app:

import os
from elasticsearch.dsl import Search
from elasticsearch import Elasticsearch

client = Elasticsearch(hosts=os.environ['ELASTICSEARCH_URL'])

for index in ["people", "test"]:
    try:
        s = Search(using=client).index(index)
        for hit in s.iterate():
            print(f"{index}: {hit}")
    except Exception as error:
        print(f"{index}: {error}")

When running this app under the elastic admin account everything works:

people: <Hit(people/EZ5RwZcBLKgOQVRW0UCl): {'first_name': 'Jane', 'last_name': 'Doe', 'birth_place': 'S...}>
test: <Hit(test/_J7PiJcBLKgOQVRWlD-k): {'name': 'foo', 'title': 'bar'}>
test: <Hit(test/_Z7QiJcBLKgOQVRWkz9X): {'name': 'foo', 'title': 'bar'}>

When running under my restricted miguel account:

people: <Hit(people/EZ5RwZcBLKgOQVRW0UCl): {'first_name': 'Jane', 'last_name': 'Doe', 'birth_place': 'S...}>
test: AuthorizationException(403, 'security_exception', 'action [indices:data/read/open_point_in_time] is unauthorized for user [miguel] with effective roles [only-people] on indices [test], this action is granted by the index privileges [read,all]')

So unfortunately I cannot verify that the problem that you have applies in general. Is there anything in my test setup that you think I should try differently to match your configuration?

[AndTheDaysGoBy]

@miguelgrinberg The server version is 8.17.4. Consider user miguel and index people. In our setup, miguel had read, manage, and index privileges for people. I'll experiment a bit more and get back to you. That you managed to have miguel perform iterate() with minimal access gives me some hope.

e.g.

GET _security/user/_privileges
{
    "names": [ "myindex*"],
    "privileges": ["index", "manage", "read"],
    "allow_restricted_indices": false
}

[AndTheDaysGoBy]

Will comment again later but, for now, just like you, on Elastic 9+, I managed to have things work without issue. Hopefully it fails on Elastic 8 so I can just say it's a version issue and aim for an upgrade.

[AndTheDaysGoBy]

OK, I think I deduced the issue, now to figure out how to get around it.
I don't know how to necessarily phrase this in the proper Elasticsearch terminology, but we have indices like my-index-, generated on a rollover policy, where these indices are all alias to the same 'my-index'.
When I side-step the alias and just pick the particular index for a given week, it works. If I try to use the alias which encompasses all the weeks, I get the aforementioned error.

[miguelgrinberg]

@AndTheDaysGoBy Maybe your permissions are set on individual indices, but not on the alias?

[AndTheDaysGoBy]

Just a small update. So, implementing permissions using data streams works. However, observing our setup, we are not using data streams. Hopefully we can upgrade in the future, however, in the interim, I'll proceed to testing the "without data streams" approach outlined: https://www.elastic.co/guide/en/elasticsearch/reference/8.18/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams

Hopefully that's the last bit of testing I need before this issue can be closed.

[AndTheDaysGoBy]

Thank goodness, I managed to recreate the error. OK, so, for context, the above link describes how there are two ways to handle data. One is with data streams which, as I've already said, does work properly. You grant your user permissions to read the data stream, and, as a result, it does gain the ability to access the underlying ds data streams generated according to the rollover policy.

The other approach outlined is without data streams. In this situation, you create an index template just as before, but with an additional setting of the rollover alias and not marking it as a data stream. You proceed to make an initial index defined as the writable index, with an alias (the value of the rollover alias) to bootstrap the rollover.

In the latter situation, giving read permissions to the base alias fails, and does not grant suitable permissions. This is confusing as, one would assume permission to an alias would grant you permission to the underlying indices.

Edit: testing, it is rather peculiar. So, the permissions issue is very specific then: granting permission to the alias DOES grant permission to the underlying indices; however, if you attempt to open a PIT, that's where issues arise, as we discussed previously.

@miguelgrinberg yes, without the PIT, I don't run into issues. (well, beyond not having the refresh-protection PIT provides)

I don't know then what the resolution is. I will try to see if there are additional permissions that allow PIT to play well with aliases/indices, but maybe this is just an unsupported case with the only way forward being migrating to data streams.

[AndTheDaysGoBy]

Potential resolution:
I created two roles, one with 'read' on the alias, e.g. 'my-index', and one with 'read' on all the underlying indices, i.e. 'my-index-*'.
Placing both of these roles on my user allowed a PIT to be opened against the alias.
I do not understand why the permissions on the the alias itself were insufficient, but at least there is a resolution.

[pquentin]

Thank you for the details! What version of Elasticsearch are you using?