fix: UploadPartCopy passthrough + copy pipeline memory cap#115
Merged
ServerSideHannes merged 6 commits intoJul 7, 2026
Conversation
Release-before-upload left ~300MB untracked RSS per pod while the governor counted ~177MB, causing OOM and Scylla dedup failures. Hold reservation through upload_part, default max 2 concurrent copy pipelines per pod, and add Scylla-shaped load tests (13 concurrent copies, subprocess Prometheus). Co-authored-by: Cursor <cursoragent@cursor.com>
Native upload_part_copy per internal part with source DEK adoption avoids download/decrypt/re-encrypt during Scylla Manager dedup, which was OOMing s3proxy under concurrent UploadPartCopy floods. Co-authored-by: Cursor <cursoragent@cursor.com>
ServerSideHannes
changed the title
Passthrough still acquires copy_pipeline_peak reservation and the pipeline semaphore so concurrent copies get SlowDown under a tight budget (memory_copy CI). Clean up ruff SIM108/F401 in passthrough tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Only passthrough UploadPartCopy when the source has multipart metadata (Scylla dedup SSTs); large PutObject blobs keep the streaming re-encrypt path so memory_copy governor/sawtooth tests stay valid. Use lightweight per-segment governor slots for native copies. Align except syntax with ruff format on py314. Co-authored-by: Cursor <cursoragent@cursor.com>
Three concurrent large copies serialize on S3PROXY_MAX_PARALLEL_COPIES=2 instead of getting SlowDown from the memory governor. Assert all finish without OOM rather than expecting a rejection. Co-authored-by: Cursor <cursoragent@cursor.com>
Extract shared verification into passthrough_verify.py with 8 pytest cases (96MB + 1280MB + 10 concurrent copies). Runs in memory_copy shard; make verify-passthrough wraps the same tests. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
upload_part_copyper internal part with source DEK adoption — no download/decrypt/re-encrypt amplification (root cause of s3proxy OOM during Scylla backup dedup)upload_part(release afterdel ciphertext), so tracked memory matches RSS during S3 I/OS3PROXY_MAX_PARALLEL_COPIES=2semaphore: max 2 streaming copy pipelines per podtest_upload_part_copy_passthrough_e2e.py): 8 tests on real s3proxy subprocess + MinIO — zero encrypt, ciphertext byte-identity on MinIO, DEK adoption, 1280MB Scylla scale, 10 concurrent copies under 192MB budgetUploadPartCopy passthrough
CopySourceRange), samekidupload_part_copyVerification
E2E passthrough suite (memory_copy shard)
test_zero_encrypt_and_low_memorytest_ciphertext_byte_identical_on_miniotest_dek_adoption_in_sidecarwrapped_dek+ internal part layout preservedtest_range_copy_reencrypts_with_high_memorytest_scylla_scale_passthroughtest_ten_concurrent_passthrough_copiesmake verify-passthrough # same 8 tests locally make test-integration-shard SHARD=memory_copyUnit
test_upload_part_copy_passthrough.py— mock round-trip + server-side pathtest_copy_pipeline_cap.py— pipeline semaphore + governor boundsTest plan
Failedstops trackingDeduplicated, no s3proxy OOM during dedup