Flatbufferize index #863
Merged
Flatbufferize index #863
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mavam
force-pushed
the
epic/application-state-versioning
branch
2 times, most recently
from
a9b010f to
0bf5357
Compare
mavam
force-pushed
the
story/ch16436
branch
5 times, most recently
from
728c76a to
d854e8b
Compare
mavam
force-pushed
the
story/ch16436
branch
4 times, most recently
from
82685dc to
a9a119e
Compare
lava
force-pushed
the
story/ch16436
branch
2 times, most recently
from
ba61d08 to
08454bd
Compare
lava
force-pushed
the
epic/application-state-versioning
branch
2 times, most recently
from
cc4cd7c to
0d8fb9a
Compare
dominiklohmann
force-pushed
the
story/ch16436
branch
from
466a0c8 to
25f19de
Compare
dominiklohmann
force-pushed
the
story/ch16436
branch
2 times, most recently
from
ac0e8e3 to
b9b505a
Compare
lava
force-pushed
the
story/ch16436
branch
3 times, most recently
from
0626270 to
a7f0566
Compare
This is a complete refactoring of the INDEX actor, while still
keeping the same outside api for export and import.
The most notable changes include:
* Use of flatbuffers for the persistent state
* Introducing a new PARTITION actor that is responsible
for managing the partition state.
* Writing one separate file for each partition.
This is necessary to support the `vast import -b` option.
Attempting to use a `caf::binary_deserializer` to deserialize a value index with a non-default `cardinality` value results in a stream error; so dont do that until the bug is fixed.
Due to the way a flatbuffer is laid on on disk [1], it is not possible to access specific fields of a given type without knowing the type. In particular, to access the `version()` field of some type, one would have to know the layout of the type first, which presumably could change in later versions. The only part of a flatbuffer that resides at a fixed offset is the 4-byte file identifier, which can be freely chosen by the user. So we use this for versioning instead. [1] https://github.com/dvidelabs/flatcc/blob/master/doc/binary-format.md
Reset to the default 'print_and_drop' default message handler one we received the deserialized state we were waiting for.
Unify log message structure and verbosity.
dominiklohmann
approved these changes
Sep 17, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the persistent state of the index to be flatbuffer-based.