Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release/oss-v21.10] Support bloom filters for 400gb+ ptables (DB-195) #3880

Merged
merged 1 commit into from
Jun 21, 2023
Merged

[release/oss-v21.10] Support bloom filters for 400gb+ ptables (DB-195) #3880

merged 1 commit into from
Jun 21, 2023

Conversation

github-actions[bot]
Copy link
Contributor

Cherry picked from #3876

@timothycoleman
Support bloom filters for 400gb+ ptables
a9381b6 
These large ptables used to try to create a bloomfilter that is larger than the maximum allowed size. Now we clamp it to the maximum allowed size

Also filters above int32.max bytes were not openable on linux because we were reading in batches of int32.max bytes but linux reads 4095 bytes less than that
@github-actions github-actions bot mentioned this pull request Jun 20, 2023
Merged
@linear
Copy link

linear bot commented Jun 20, 2023

DB-195 Fix index bloom filters for large ptables

Two problems.

  1. Bloom filters above int.max bytes (~2gb) can't currently be read on linux. Instead an error occurs and the bloom filter is not used
System.Exception: Read fewer bytes (2147479552) from bloom filter (/eventstore/db/index/6b4b2f32-20b0-4395-8e20-a6250f2eae79.bloomfilter) than expected (2147483647)
  1. PTables larger than ~400gb attempt to create a bloom filter larger than the max allowed 4gb, causing the server to exit

@pvanbuijtene pvanbuijtene reopened this Jun 20, 2023
@timothycoleman timothycoleman merged commit b5ac0ef into release/oss-v21.10 Jun 21, 2023
8 of 9 checks passed
@timothycoleman timothycoleman deleted the cherry-pick/3876/timcoleman/large-index-bloomfilters-release/oss-v21.10 branch June 21, 2023 08:31
@mmuller99 mmuller99 mentioned this pull request Nov 22, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timothycoleman timothycoleman timothycoleman approved these changes

@pvanbuijtene pvanbuijtene pvanbuijtene approved these changes

@YoEight YoEight Awaiting requested review from YoEight

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@timothycoleman @pvanbuijtene