New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DynamoDB Table Resource should support Pagination for Scan and Query #2039
Comments
@rbu - Thank you for your post. I will mark this as feature request. |
@rbu Thank you much for this, I'm also interested. I'm surprised by there being relatively little activity on this issue. But I suppose if the consumers of DynamoDB that aren't using some sort of ORM already are the main ones affected; I'm not really surprised. They're already dealing a lot with some gnarly JSON blobs. |
The client supports paginators whereas the table supports high-level operations. See <boto/boto3#2039>.
Any update on this feature request - is it planned at all? |
What is holding up this issue? Could I help? |
Thank you all for checking in and apologies for the delay. We created a backlog item to look into this feature request and if anyone wants to share additional use cases or feedback then please feel free to leave a comment. Although we can't give a definite timeline at the moment, we'll make sure to keep it posted as soon as we have an update and please feel free to check back here in the future. |
I use a very simple implementation: from typing import Callable, Dict, Iterator
def dynamodb_paginator(action: Callable, kwargs: Dict) -> Iterator[Dict]:
while True:
response = action(**kwargs)
for item in response["Items"]:
yield item
if "LastEvaluatedKey" in response:
kwargs["ExclusiveStartKey"] = response["LastEvaluatedKey"]
else:
break Usage: session = boto3.Session()
dynamodb = session.resource("dynamodb")
table = dynamodb.Table("my_table")
scan_kwargs = { "ScanFilter": ... } # set here the arguments you would use in the scan() method call
for item in dynamodb_paginator(table.scan, scan_kwargs):
... # do something with the item |
The boto3 "Table" Resource API does not provide easy access to paged results when using Scan or Query actions. That's unlike other high level boto3 Resource APIs like S3 which supports
s3.Bucket('bucket').objects.pages()
.The underlying DynamoDB client supports pagination, which leaves the boto3 user with the choice between nice attribute access + ugly pagination or ugly attribute access + nice pagination, which is unfortunate.
The text was updated successfully, but these errors were encountered: