cloud_storage: segment_meta_cstore serde optimizations #10805
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7 tasks
andijcr
changed the title
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
c6f7eb6
to
c1ee47f
Compare
|
https://buildkite.com/redpanda/redpanda/builds/29345#01882bdd-48ca-4296-94c0-c7f1ff0c5281 failure is in debug is #10500 |
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
c1ee47f
to
4685e37
Compare
|
https://buildkite.com/redpanda/redpanda/builds/29377#01882f17-cd34-47bf-a778-f19dc0cd14df interesting failure: edit: should be solved by #10852 the other is #10240 |
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
4685e37
to
9d22464
Compare
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
9d22464
to
2c8bf3c
Compare
|
just a bit of benchmark: with this pr: test size is 10000 segment_meta |
Lazin
previously approved these changes
May 24, 2023
| @@ -733,6 +734,16 @@ class column_store | |||
| member_fields()); | |||
| } | |||
|
|
|||
| auto unsafe_alias() const -> column_store { | |||
|
fixed conflict |
looks like CI fails to rebase it |
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
244c4c0
to
ce91f53
Compare
|
rebase - second attempt |
|
Failure is #10977 |
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
ce91f53
to
f7ad42d
Compare
|
/ci-repeat |
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
2 times, most recently
from
a16be1c
to
c818704
Compare
previously to_iobuf was achieved by constructing a temporary cstore, appending all the segments and serializing it. this is done so that to_iobuf can be const, in turn to be able to use it in base_manifest::serialize. To minimize operations and memory, this commits adds the ability to create a column_store that shares all the iobuf with the *this object, and then serialize it. sharing is done by constructing on top of deltafor_encoder::share() const a series of unsafe_alias() const methods, responsible to construct the subobjects. this new schema saves time by not re-encoding the segment_meta, and saves memory by sharing the underlying iobuf objects, to write them directly in the final serde iobuf
andijcr
force-pushed
the
feat/cstore/serde_optimizations
branch
from
c818704
to
549a837
Compare
|
force push: rebase on dev |
Lazin
approved these changes
Jun 27, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
previously to_iobuf was achieved by constructing a temporary cstore,
appending all the segments and serializing it. this is done so that
to_iobuf can be const, in turn to be able to use it in
base_manifest::serialize.
To minimize operations and memory, this commits adds the ability to
create a column_store that shares all the iobuf with the *this object,
and then serialize it.
sharing is done by constructing on top of deltafor_encoder::share()
const a series of unsafe_alias() const methods, responsible to construct
the subobjects.
this new schema saves time by not re-encoding the segment_meta, and
saves memory by sharing the underlying iobuf objects, to write them
directly in the final serde iobuf
Backports Required
Release Notes