Use Flatbuffers for Persistent State of Segment Store and Meta Index #972
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The utilities in namespace fbs are meant to facility conversion from flatbuffers to VAST and vice versa.
Since a segment wraps a reference-counted chunk, it's not necessary to add another layer of reference counting on top.
Because there is no PPA for version 1.11 for the Ubuntu version of the github CI.
We added an extra layer of indirection into the segment flatbuffer so that we can efficiently append table slice flatbuffers to the segment. See https://groups.google.com/forum/#!topic/flatbuffers/6KRC02iXDhs for the detailed discussion that motivated this change.
Co-Authored-By: tobim <tobim@fastmail.fm>
Support versioning of persistent data structures
We observed a race condition in unit tests where the terminator could sometimes "miss" a down message if one of the actors it tried to kill was a detached actor. In this case, the timeout would trigger but the terminator was left alive even though all the actors it tries to terminate are gone, leading to an indefinite system hang at shutdown. We explicitly kill the terminator as a workaround, until a more exhaustive shutdown logic using retries is implemented or the caf bug involving detached actors is fixed.
This was changed in master, so we fix the remaining instances in this branch.
Add filesystem actor
|
Can you squash my I'm going to check if the branch works with the pro version locally. |
lava
force-pushed
the
epic/application-state-versioning
branch
from
c5862ae to
e98dc27
Compare
|
Should we rebase this onto current master? |
|
@dominiklohmann The last commit merges current master into the branch. Rebasing is problematic with such a big branch: First of all it is pretty labor-intensive but more importantly it potentially adds new conflict resolutions to a ton of old commits, potentially introducing subtle bugs that cannot be reviewed and invalidating the existing reviews. |
dominiklohmann
requested changes
Jul 13, 2020
tobim
approved these changes
Jul 13, 2020
mavam
requested changes
Jul 13, 2020
lava
force-pushed
the
epic/application-state-versioning
branch
from
ab55b53 to
cc4cd7c
Compare
lava
force-pushed
the
epic/application-state-versioning
branch
from
cc4cd7c to
0d8fb9a
Compare
mavam
approved these changes
Jul 13, 2020
dominiklohmann
approved these changes
Jul 13, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to use a well-defined on-disk format for VAST, to enable stable upgrade paths for VAST users.
This PR adds the first step of this effort by changing the segment store and the meta index to use a flatbuffer layout for its persistent on-disk data.
In addition, some preparatory work for more complete flatbuffer integration in the future is included, such as the
LayoutandTypetype definitions.Finally, a new
filesystemactor is included, which is not strictly related to flatbufferization but will be used in follow-up work to centralize the file reads and writes when loading and storing flatbuffers.NOTE: This PR adds a hard dependency on
flatbuffersto VAST.