[v23.3.x] config/configuration: set cloud_storage_spillover_manifest_size #16174
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
this enable splitting the partition_manifest into spillovers once it reaches 2*cloud_storage_spillover_manifest_size (cherry picked from commit f4f2581)
|
test failures are probably related to the changed default. investigating... |
fix use of cloud_storage_spillover_manifest_max_segments
explicitly disable cloud_storage_spillover_manifest_size if the tests bootstraps with cloud_storage_spillover_manifest_max_segments. this is done because inside redpanda, size takes precendece over max_segments and the tests will work with wrong assumptions
redpanda ignores cloud_storage_spillover_manifest_max_segments when cloud_storage_spillover_manifest_size is set. explicitly set the latter to None
|
/ci-repeat 1 skip-unit skip-redpanda-build |
Partition shutdown and purger run asynchronously and can race since \redpanda-data#11873. 5ms grace period for tests is too as TS requests for final manifest upload can take many milliseconds, especially with real TS backend. Fixes redpanda-data#13510
|
Also pulled in #15814 since there was a conflict and it's a one-line CI flake fix |
These tests either weren't designed to use spillover, or were manually configuring a max segment count. In either case, the new default value is problematic for them since the size-based config takes precedence. I went through all tests I could find that configured cloud_storage_spillover_manifest_max_segments and unset cloud_storage_spillover_manifest_size as appropriate. Previous attempts to alleviate this attempted to set these at runtime with the incorrect ducktape methods (set_cluster_config() instead of set_cluster_config_to_null()). It's more foolproof to just set them at bootstrap time, as they were set before the default was introduced.
andrwng
force-pushed
the
vbotbuildovich/backport-16172-v23.3.x-282
branch
from
de4830a
to
9290244
Compare
|
/dt |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #16172