Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamoDB: Fix ScannedCount calculation for Limit-ed calls #7085

Merged

Conversation

bblommers
Copy link
Collaborator

Fixes #7080

Our scan() implementation would iterate through the entire dataset, and only trim the results at the end.

This approach has multiple downsides:

  • We're needlessly filtering/copying all items in the table
  • The ScannedCount is wrong - because we're scanning the whole table, we're essentially saying that ScannedCount = len(table)

This PR changes the approach so that we only read the necessary items.

  • we only start processing items when we've reached the ExclusiveStartKey (DynamoDB's pagination key)
  • we immediately end processing when the result set has reached the desired size (either in length or in bytesize)

Note that query still follows the old approach, and presumably suffers from the same problems. We should rewrite that operation in the same way at some point.

@bblommers bblommers added this to the 4.2.11 milestone Dec 1, 2023
Copy link

codecov bot commented Dec 1, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (ad316dd) 95.87% compared to head (87f925c) 95.87%.

Files Patch % Lines
moto/dynamodb/models/table.py 97.61% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7085      +/-   ##
==========================================
- Coverage   95.87%   95.87%   -0.01%     
==========================================
  Files         834      834              
  Lines       82019    82036      +17     
==========================================
+ Hits        78638    78654      +16     
- Misses       3381     3382       +1     
Flag Coverage Δ
servertests 35.82% <7.14%> (-0.01%) ⬇️
unittests 95.82% <97.61%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bblommers bblommers merged commit bb23df4 into getmoto:master Dec 1, 2023
32 checks passed
@bblommers bblommers deleted the dynamodb/stop-scan-when-limit-is-reached branch December 1, 2023 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mock_dynamodb: response from boto3 and moto is inconsistent when scanning a table with a limit
1 participant