Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Cache midpoints in Ptable footer for faster startup times #1491
Large databases with very large index files (10+ GB) can take a lot of time to start up due to the time taken to cache the midpoints into memory. Although there are not many midpoints (up to 2^28, 24 bytes each - around 6GB max), it takes much time to load them from disk since they are not stored sequentially and the OS reads data in large blocks.
This change caches the midpoints at the end of the PTable file for faster loading times. It will not improve speeds for existing large databases, until the next large PTable merges or if the indexes are rebuilt
Add a new version of PTable, V4:
Midpoints will be cached during the next PTable merge (only if no index entries have been removed during the merge, due to missing database entries)
What are your views on handling corrupted index entries (might happen if index verification has been disabled)?
@hayley-jean and I proposed the following:
If any of these assertions fail, add a flag that forces index & chunk file verification next time Eventstore is started so that indexes can be rebuilt if corrupted (md5 hashes don't match).
What do you think?
https://github.com/EventStore/EventStore/blob/release-v4.0.4/src/EventStore.Core/Index/PTable.cs#L440 as example (there are bunches of similar checks)
Startup Performance tests
Tests with a 10GB PTable
Generate a 10GB PTable
PTable Merge Performance tests
Results merging 2 10GB files
Note: since we're using same 2 10GB files, the index entries are the same and will be merged into a ~10GB file.
cache-midpoints-in-ptable branch,--skip-db-verify --skip-index-verify: 362s merge, 0.1s load without verification: Total 362s
cache-midpoints-in-ptable branch, --skip-db-verify : 372s merge, 37s load with verification: Total 409s
oss-v4.0.3 branch, --skip-db-verify : 327s merge, 78s load with verification: Total 405s
Creating an indexmap file
Prevent exceptions during test
Results with different options
cache-midpoints-in-ptable branch, --skip-db-verify --skip-index-verify : 362s merge, 0.1s load
cache-midpoints-in-ptable branch, --skip-db-verify : 372s merge, 37s load
oss-v4.0.3, --skip-db-verify : 327s merge, 78s load