Introduce a minimum partition version #2778
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dominiklohmann
added
maintenance
dominiklohmann
force-pushed
the
story/sc-36344/bump-minimum-partition-version
branch
from
bf968a9 to
0c2a7a1
Compare
lava
approved these changes
Dec 9, 2022
dominiklohmann
force-pushed
the
story/sc-36344/bump-minimum-partition-version
branch
2 times, most recently
from
a0fd670 to
b022435
Compare
This introduces a minimum partition version for VAST in order to enable future cleanup of things we know only existed in databases where all partitions have a partition version lower than the minimum. This PR also bumps the minimum partition version to 1.
This is superseded by the per-partition version and has been effectively unused since before VAST v1.0; now that the minimum partition version is one that was set in VAST v2.2 we can remove it.
dominiklohmann
force-pushed
the
story/sc-36344/bump-minimum-partition-version
branch
from
b022435 to
075a274
Compare
dominiklohmann
deleted the
story/sc-36344/bump-minimum-partition-version
branch
dominiklohmann
added a commit
that referenced
this pull request
Dec 14, 2022
Following the introduction of a minimum partition version of 1 with #2778, we can now—finally—remove the archive and everything that belongs to it.
Merged
dominiklohmann
added a commit
that referenced
this pull request
Dec 14, 2022
Following the introduction of a minimum partition version of 1 with #2778, we can now—finally—remove the archive and everything that belongs to it.
dominiklohmann
added a commit
that referenced
this pull request
Dec 14, 2022
Following the introduction of a minimum partition version of 1 with #2778, we're no longer able to encounter any table slices that are MessagePack-encoded or use an older version of the mapping of our data model onto Apache Arrow. There's a future refactoring that is now unblocked, which is a total replacement of the legacy baggage that is the `table_slice` API with an easier-to-use wrapper around the same underlying data that makes more assumptions about the underlying column-major memory layout. This is based on #2797 to avoid merge conflicts.
dominiklohmann
added a commit
that referenced
this pull request
Dec 16, 2022
Following the introduction of a minimum partition version of 1 with #2778, we're no longer able to encounter any table slices that are MessagePack-encoded or use an older version of the mapping of our data model onto Apache Arrow. There's a future refactoring that is now unblocked, which is a total replacement of the legacy baggage that is the `table_slice` API with an easier-to-use wrapper around the same underlying data that makes more assumptions about the underlying column-major memory layout. This is based on #2797 to avoid merge conflicts.
dominiklohmann
added a commit
that referenced
this pull request
Dec 19, 2022
Following the introduction of a minimum partition version of 1 with #2778, we can now—finally—remove the archive and everything that belongs to it.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, the augmented partition synopsis is now irrelevant as the schema is always contained in the partition synopsis already. This has no user-facing implications, it's purely an internal cleanup.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, all partitions are now guaranteed to be homogeneous. The rebuilder as such does not need to take care of heterogeneous partitions any longer.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, all partitions are now guaranteed to have an id range specified in their partition synopsis. Rewriting old partition synopses on startup is no longer necessary.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, all partitions are now guaranteed to have an id range specified in their partition synopsis. Rewriting old partition synopses on startup is no longer necessary.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, all partitions are now guaranteed to be homogeneous. The rebuilder as such does not need to take care of heterogeneous partitions any longer.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, all partitions are now guaranteed to have an id range specified in their partition synopsis. Rewriting old partition synopses on startup is no longer necessary.
dominiklohmann
added a commit
that referenced
this pull request
Dec 21, 2022
Following the introduction of a minimum partition version of 1 with #2778, the augmented partition synopsis is now irrelevant as the schema is always contained in the partition synopsis already. This has no user-facing implications, it's purely an internal cleanup.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a minimum partition version for VAST in order to enable future cleanup of things we know only existed in databases where all partitions have a partition version lower than the minimum. This PR also bumps the minimum partition version to 1.