Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make batch_size in kv.Scan effective for first Scan of iterator #7875

Merged
merged 1 commit into from
Jun 17, 2024

Conversation

itaiad200
Copy link
Contributor

@itaiad200 itaiad200 commented Jun 16, 2024

Closes #7864

Tested manually. Requested the first entry in a prefix with tombstones.

  1. AWS DDB, lakeFS running locally: 4162 leading tombstones, the fix reduced the latency from 1m40s to 2.5s.
  2. Postgres & lakeFS running locally: 40K leading tombstones, the fix reduced the latency from 20s to 452ms.
  3. Azure CosmoDB & lakeFS running locally: 931 leading tombstones, the fix reduced the latency from 43s to 40s (cosmosdb page size are harder to reduce after the first scan).

Copy link

E2E Test Results - DynamoDB Local - Local Block Adapter

13 passed

@itaiad200 itaiad200 added the include-changelog PR description should be included in next release changelog label Jun 17, 2024
@itaiad200 itaiad200 marked this pull request as ready for review June 17, 2024 11:44
Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat! Thanks. And having all the tests makes this really easy to review, so kudos to all KV test writers throughout time!

@itaiad200 itaiad200 merged commit 7559cd1 into master Jun 17, 2024
39 of 43 checks passed
@itaiad200 itaiad200 deleted the 7864-iterator-fetch branch June 17, 2024 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
include-changelog PR description should be included in next release changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: A branch with deleted objects is very ineffective with small 'amount' param
2 participants