fix: per-part copy memory reservation (Scylla backup 503 storm)

[ServerSideHannes]

Summary

• Streaming UploadPartCopy now acquires/releases governor memory per 32MB internal part (~88MB peak) instead of holding one reservation for the entire copy duration
• Fixes sustained active_mb=176 backpressure when two concurrent 1280MB Scylla backup copies ran — memory is free during S3 upload I/O between parts
• Adds unit tests (in-process governor), subprocess Prometheus tests (s3proxy_memory_reserved_bytes sawtooth), and Scylla-shaped load (1280MB × 2 concurrent on 192MB budget)
• Fixes legacy no-kid encrypted COPY passthrough (byte-identical, no failed re-encrypt attempt)

Test plan

• pytest tests/unit/test_copy_per_part_memory.py — 6 passed
• pytest tests/integration/test_copy_per_part_metrics.py -m e2e -n0 — 4 passed (incl. 1280MB Scylla load)
• make verify-copy-memory — pre-merge gate
• 476 unit tests green
• CI memory_copy shard (25min timeout for 1280MB workload)
• Staged prod rollout: watch s3proxy_memory_reserved_bytes sawtooth + 503 rate during Scylla backup window

Path to 100% confidence (post-merge)

1. Canary deploy one pod with this build; compare s3proxy_memory_reserved_bytes min/max during backup vs baseline (expect sawtooth, not flat ~176MB)
2. Alert on flat reservation: min_over_time(s3proxy_memory_reserved_bytes[5m]) > 150MB during copy-heavy periods = regression
3. 503/SlowDown rate on UploadPartCopy should drop during concurrent backup; no OOM restarts
4. Soak: run backup twice back-to-back; confirm no InvalidPart regressions (separate bug, not in scope here)

Made with Cursor