CORE-1643 rptest: Increase backoff interval for GCS #17545
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The timeing_stress_test performs a lot of TS operations in short time and in some cases it has to upload manifest frequently (more often than once per second). The manifest uploads are driven by segment uploads and retention. Normally, we will try to upload less often (once per 60s or less) even if we're writing into the partition constantly. But under the local storage pressure the ntp-archiver is forced to upload the manifest mroe frequently. The local storage pressure means that the local storage wants to evict some data but it can't do this unless the manifest is uploaded and the clean offset is moved forward.
The timing stress test introduces local storage pressure and uploads manifests frequently. The GCS may throttle us when we're trying to reupload the manifest faster than once per second. If the initial backoff has default value of 100ms this is exactly what we will try to do once some throttling is applied. Redpanda receives SlowDown response and decides to retry after 100ms, after 200ms, 400ms etc. It never uploads the manifest and the test fails.
This fix increases the initial backoff to 1000ms if the test is running on GCS.
Fixes #15488
Backports Required
Release Notes