fix: stop memory governor over-count and clamp starvation#109
Merged
ServerSideHannes merged 4 commits intoJul 6, 2026
Conversation
The framed upload path peaks at 2*internal_part + one frame, but streaming_upload_peak used 2*part + 2*frame and scaled with multi-GB Content-Length. When that estimate exceeded the governor budget, clamp reserved the entire budget — starving concurrent ~50MB Scylla parts. - Tighten streaming_upload_peak to measured 4*part (small) / 2*part+frame (large) - Cap governor reservations at the 512MB-workload peak via governor_memory_footprint - Clamp oversized requests to routine peak, not the whole budget - Add regression tests for multi-GB estimates and concurrent Scylla parts Co-authored-by: Cursor <cursoragent@cursor.com>
Add explicit guards for the old 2*part+2*frame formula and full-budget clamp, prod replay (33MB active + multi-GB estimate at 312MB), request- handler reservation spy, deployed budget matrix, mixed Scylla workload, and signed 50MB Scylla part tracemalloc bounds. Co-authored-by: Cursor <cursoragent@cursor.com>
Break a long line and move request-handler spy out of the for-loop to avoid B023 closure-over-loop-variable warnings. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
streaming_upload_peakto match measured framed-upload peaks (4*partfor small parts,2*part + FRAMEfor multi-frame) instead of2*part + 2*framewhich double-counted the frame and scaled with multi-GBContent-Lengthgovernor_memory_footprint()to cap request-gate reservations at the 512MB-workload peak so a 6.6GBContent-Lengthno longer reserves 500MB+Root cause
Prod showed
MEMORY_CLAMPED_TO_BUDGETat ~509MB and perpetualMEMORY_BACKPRESSUREwithrequested_mb == limit_mbwhile a normal ~33MB Scylla part was active. The governor treated clamped requests as owning the full budget slot.Test plan
uv run pytest tests/unit/— 472 passedtests/unit/test_governor_peak_and_clamp.pycovers multi-GB estimate cap, clamp non-monopolization, and concurrent Scylla-sized parts under 312MB budgetMEMORY_BACKPRESSUREatrequested_mb == limit_mbS3PROXY_MEMORY_LIMIT_MB=312pods admit multiple concurrent ~50MB partsMade with Cursor