fix: make server-side copies O(1) memory (stop Scylla backup 503 storm)#113
Merged
Merged
Conversation
Root cause of the Scylla backup 503 storm (~50% of requests): a dedup UploadPartCopy of a 4.7GB SSTable (single client part, partNumber=1) sized its internal S3 parts as object_size/MAX_INTERNAL_PARTS_PER_CLIENT (=object/20 = 238MB), so the copy pipeline peaked at ~535MB -- larger than the 192MB governor budget. copy_governor_clamped_reserve then clamped the reservation to the WHOLE budget, making the copy exclusive: it could only be admitted when active_bytes hit exactly 0. Under steady backup traffic (baseline reservations of other in-flight requests keep active_bytes > 0) that never happens, so every large copy backpressured for the full timeout and returned 503 SlowDown. Confirmed in prod: MEMORY_CLAMPED_TO_BUDGET copy=true requested_mb=534, MEMORY_REJECTED at active_mb=0.06-0.31, 296x503 vs 304x200. Writer preference (#112) could not fix this: it holds back *new* small requests but cannot evict baseline reservations already held, and a single copy deadlocks even against its own request-gate reservation. Fix: copies frame at a FIXED internal part size (crypto.copy_internal_part_size, 32MB) instead of object_size/20, so the copy peak is O(1) in object size (~90MB) regardless of how large the object is. A copy now fits the budget outright, never clamps, never runs exclusive, and simply reserves ~90MB like any other request -- multiple copies run concurrently, bounded by the budget (so the multi-copy OOM that motivated exclusivity can't happen either, because each copy is light). A large copy needs many internal parts, so allocate them sequentially (client_part_number=0) rather than from the fixed 20-wide per-client window. Safe: copies are single-part uploads (verified 291/291 partNumber=1 in prod) and reassembly is metadata-driven (get.py sorts by internal_part_number and uses stored ciphertext sizes), so smaller/more parts and sequential numbering need no migration and read back byte-identically. Part size grows only past S3's 10k-part ceiling (>288GB objects, which backup SSTables never hit). Tests: rewrote the copy-governor tests that encoded the old 'monopolize the budget' design to assert the new invariants (peak is O(1) and size-independent, copy never clamps, N copies run concurrently within budget, copy coexists with uploads). Added test_large_copy_admitted_while_baseline_reservations_held which reproduces the exact prod deadlock and is verified to fail against the old O(size) sizing. Full unit suite green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The real fix (root cause, not a bandaid)
Follow-up to #112. That patch (writer preference) treated a symptom; this fixes the cause of the Scylla backup 503 storm.
What's actually happening in prod
Pulled from the live pods (
2026.7.8, 30 replicas, 1Gi limit, governor budget 192MB):296 × 503vs304 × 200in a 4-minute window).MEMORY_CLAMPED_TO_BUDGET copy=true requested_mb=534.83(276×),MEMORY_REJECTEDatactive_mb0.06–0.31.partNumber=1(291/291) — a single-partUploadPartCopy(Scylla Manager dedup copies each SSTable as one object).Why
The copy pipeline sized its internal S3 parts as
object_size / MAX_INTERNAL_PARTS_PER_CLIENT(= object/20 = 238MB for a 4.7GB SSTable), socopy_pipeline_peak≈ 535MB — larger than the 192MB budget.copy_governor_clamped_reservethen clamped the reservation to the whole budget, making the copy exclusive: admission requiresactive_bytes + limit ≤ limit, i.e.active_bytes == 0. Under steady backup traffic the baseline reservations of other in-flight requests keepactive_bytes > 0forever, so every large copy backpressures the full timeout → 503 → retry → same.Writer preference (#112) can't fix this: it holds back new small requests but can't evict baselines already held, and a single copy deadlocks even against its own request-gate reservation.
The fix
Copies frame at a fixed internal part size (
crypto.copy_internal_part_size, 32MB) instead ofobject/20, so the copy peak is O(1) in object size (~90MB) no matter how big the object is. A copy now fits the budget outright — never clamps, never exclusive, just a ~90MB reservation like anything else. Multiple copies run concurrently, bounded by the budget (so the multi-copy OOM that motivated exclusivity in #110 can't happen either — each copy is light).A large copy needs many internal parts, so they're allocated sequentially (
client_part_number=0) rather than from the fixed 20-wide per-client window. Safe because:partNumber=1).get.pysorts byinternal_part_numberand uses stored ciphertext sizes), so smaller/more parts + sequential numbering read back byte-identically — no migration.Tests
Rewrote the copy-governor tests that hard-coded the old "monopolize the budget" design to assert the new invariants:
test_copy_peak_is_o1_across_object_sizes— peak identical from 33MB to 100GB, all < 128MB.test_copy_does_not_monopolize_budget,test_multiple_copies_run_concurrently_bounded_by_budget,test_copy_coexists_with_scylla_upload.test_large_copy_admitted_while_baseline_reservations_held— reproduces the exact prod deadlock (copy + baseline reservations held), verified to fail against the old O(size) sizing and pass with the fix.Full unit suite green (one pre-existing, unrelated dashboard asyncio-loop flake).
Follow-up (not in this PR)
The code fix removes the deadlock. Throughput is still capped by the 192MB budget while the pod has a 1Gi limit (
maxconn 20/pod admits ~20 concurrent, but 192MB only fits ~2 copies + small traffic). Recommend bumpingperformance.memoryLimitMbin the argocd values toward ~640MB to use the available headroom now that copies are light and honestly accounted. That's a config change inargocd-internal-services, separate from this code PR.