-
Notifications
You must be signed in to change notification settings - Fork 6.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Accumulate blob file additions in VersionEdit during recovery #7903
Conversation
Summary: During recovery, RocksDB performs a kind of dummy flush; namely, entries from the WAL are added to memtables, which then get written to SSTs and blob files (if enabled) just like during a regular flush. Note that multiple memtables might be flushed during recovery for the same column family, for example, if the DB is reopened with a lower write buffer size, and therefore, we need to make sure to collect all SST and blob file additions. The patch fixes a bug in the earlier logic which resulted in later blob file additions overwriting earlier ones. Test Plan: Added a unit test and ran `db_stress`.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
@ltamasi has updated the pull request. You must reimport the pull request before landing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Nice test👍
…ok#7903) Summary: During recovery, RocksDB performs a kind of dummy flush; namely, entries from the WAL are added to memtables, which then get written to SSTs and blob files (if enabled) just like during a regular flush. Note that multiple memtables might be flushed during recovery for the same column family, for example, if the DB is reopened with a lower write buffer size, and therefore, we need to make sure to collect all SST and blob file additions. The patch fixes a bug in the earlier logic which resulted in later blob file additions overwriting earlier ones. Pull Request resolved: facebook#7903 Test Plan: Added a unit test and ran `db_stress`. Reviewed By: jay-zhuang Differential Revision: D26110847 Pulled By: ltamasi fbshipit-source-id: eddb50a608a88f54f3cec3a423de8235aba951fd
Summary:
During recovery, RocksDB performs a kind of dummy flush; namely, entries
from the WAL are added to memtables, which then get written to SSTs and
blob files (if enabled) just like during a regular flush. Note that
multiple memtables might be flushed during recovery for the same column
family, for example, if the DB is reopened with a lower write buffer size,
and therefore, we need to make sure to collect all SST and blob file
additions. The patch fixes a bug in the earlier logic which resulted in
later blob file additions overwriting earlier ones.
Test Plan:
Added a unit test and ran
db_stress
.