Add a rebuild command plugin
#2321
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dominiklohmann
force-pushed
the
topic/hackathon-rebuild-command
branch
from
d46d831 to
069b24a
Compare
The new plugin rebuilds the whole database by running an `identity` transformation on the database, causing all partitions to be rebuilt entirely. This conveniently merges partitions up to the configured `max-partition-size`, upgrades to the most recent table slice encoding, and migrates all data to the currently configured store backend. Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
dominiklohmann
force-pushed
the
topic/hackathon-rebuild-command
branch
2 times, most recently
from
730040f to
99e9b06
Compare
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
dominiklohmann
force-pushed
the
topic/hackathon-rebuild-command
branch
from
4c4bbb0 to
a905a4f
Compare
Co-authored-by: Győző Gáspár <gyozo.gaspar@tenzir.com>
dominiklohmann
force-pushed
the
topic/hackathon-rebuild-command
branch
from
a905a4f to
d37ad0b
Compare
lava
force-pushed
the
story/sc-31754/homogenous-partition-transformer
branch
from
d7bcfa9 to
9acd287
Compare
Base automatically changed from
story/sc-31754/homogenous-partition-transformer
to
master
June 2, 2022 16:08
lava
approved these changes
Jun 3, 2022
Add a Nix package for the indicators library
dominiklohmann
force-pushed
the
topic/hackathon-rebuild-command
branch
from
4534d33 to
2cc7f54
Compare
dominiklohmann
force-pushed
the
topic/hackathon-rebuild-command
branch
from
2cc7f54 to
c851294
Compare
tobim
reviewed
Jun 9, 2022
dominiklohmann
commented
Jun 9, 2022
Otherwise version - 1 overflows, which has undesirable side effects for rebuilding.
tobim
approved these changes
Jun 9, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The new plugin rebuilds the whole database by running an
identitytransformation on the database, causing all partitions to be rebuilt entirely. This conveniently merges partitions up to the configuredmax-partition-size, upgrades to the most recent table slice encoding, and migrates all data to the currently configured store backend.📝 Checklist
🎯 Review Instructions
Test it out locally. The testing of the individual pieces is already covered, and integration testing is impossible until we have a
flushcommand (next Hackathon?).For the code I recommend just reading the whole diff at once.
For stress-testing, I recommend running VAST with a low
vast.active-partition-timeout, ingesting a bunch of data, and then looking at the results of rebuilding.