perf: cut the solver time limit to 10s - #116
Merged
Merged
Conversation
Requests that exhaust the limit walk a cost optimal plateau instead of closing a gap. Replaying 20 of the slow requests collected in production, 17 end on the same objective at 10s as at 20s, two end marginally better and one ends 4.7 percent worse. Halving the limit halves the latency and the core those requests hold on a one vCPU replica. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts the solver time limit from 20 s (Azure) / 25 s (image default) to 10 s.
Why
Requests that exhaust the limit are not requests that are closing a gap — they walk a cost
optimal plateau until the clock runs out. Every second past the last incumbent improvement is
tail latency and a held core on a CPU bound, one vCPU replica.
Measurement
20 requests sampled from the 729 slow requests collected in production (all of them had
exhausted the 20 s limit there), replayed with
OPTIMIZER_NUM_THREADS=1like production, onesolve at a time so no run steals CPU from another. The objective is maximized, so a negative
gap is a worse schedule.
The two "better" cases are search path luck, not a systematic win: +0.24 % and +0.13 %.
A 60 s reference over the same sample puts 20 s within 0.37 % of it, so 20 s is not the optimum
either — both limits stop on the plateau, only at different points.
Sweeping the one regressing case plus the two noisy ones across more limits shows where the
cost sits:
That case finds its better incumbent between 12 s and 15 s. 15 s would be the loss free
setting on this sample; 10 s buys another 5 s of latency for a −4.7 % schedule on roughly one
in twenty slow requests. Slow requests are about 2 % of traffic, so that is on the order of
0.1 % of all requests. Flip the value if that trade is not worth it.
Also
The dump threshold is the time limit itself, so
/tmp/slow-requests.jsonlnow collectseverything above 10 s instead of above 20 s — more lines per replica, still ephemeral.
Reproduce with the collected log: sample requests from
slow-requests.jsonl(seemake slow),POST each to
/optimize/charge-schedulewithOPTIMIZER_TIME_LIMITset per run and compareobjective_value.