fix(warm): warm capacity deferred itself out of existence - #290
Merged
Conversation
`ReconcileWarm` and `AdmitWarm` both refused to build warm capacity whenever `queueIntents.HasActive` was true. That predicate is global: one active intent in any of the ten scale sets blocked warm creation for every pool. The fleet is rarely idle, so the gate was almost always shut. Measured on the services host over six hours: 634 reconciles, 510 deferred with reason `queue-intent`, `ready_before` was 0 in 631 of them and `ready_after` 0 in 630. One warm instance was created in six hours against a target of two. The guard was self-reinforcing -- with no warm capacity every job cold-started, which lengthened the queue, which kept the guard shut. Capacity is still protected, and by the checks that actually measure it: `AdmitWarm` continues into the pool, CPU, allowance, memory and pressure decisions, warm creation is bounded by `max_ready`, and a real job claims a warm instance through `ClaimWarm` rather than waiting behind it. Refusals are still recorded -- `createWarm` writes the real admission decision into `deferral_reason`, so a refusal now names the resource that was short instead of naming the queue. `ReasonQueueIntent` goes with it: nothing emits it any more, and a reason the system cannot reach is a record that misleads whoever reads it next. No dashboard or alert rule referenced it. The test that asserted the old behaviour is replaced by its inverse, and the negative control holds: restoring the gate with the new test unchanged fails on `Deferred`. Claude-Session: https://claude.ai/code/session_017rA31sF7EzvtbRmi18Uzz9
The provider release ceremony is two commits by construction: the source commit, then the derivative that pins it. This is the second half for 9907cba. `source_commit` names that commit, `binary_sha256` is the reproducible double-build of it -- CGO_ENABLED=0, -trimpath, -buildvcs=false, -buildid= -s -w, go1.26.6 -- verified identical across two builds locally before writing it down. Nothing else in the tree states the version: the build stamp, the admission baseline and each host's desired state all derive from this file, which is why it exists. Claude-Session: https://claude.ai/code/session_017rA31sF7EzvtbRmi18Uzz9
The derivative is the authority and every host config is compared against it, so bumping one without the others is exactly the divergence the file was written to prevent. `fleetcontract.ValidateConfig` is what catches it: the control-plane artifact versions must equal the contract's. Claude-Session: https://claude.ai/code/session_017rA31sF7EzvtbRmi18Uzz9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ReconcileWarmandAdmitWarmboth refused to build warm capacity wheneverqueueIntents.HasActivewas true. That predicate is global: one active intent in any of the ten scale sets blocked warm creation for every pool. The fleet is rarely idle, so the gate was almost always shut.Measured on the services host over six hours:
queue-intentready_before == 0ready_after == 0The guard was self-reinforcing: with no warm capacity every job cold-started, which lengthened the queue, which kept the guard shut. Over the same window
queue.queuedspans infleet_tracesshow a mean wait of 115–184s per repository.Capacity is still protected, and by the checks that actually measure it.
AdmitWarmcontinues into the pool, CPU, allowance, memory and pressure decisions; warm creation is bounded bymax_ready; and a real job claims a warm instance throughClaimWarmrather than waiting behind it. Refusals are still recorded —createWarmwrites the real admission decision intodeferral_reason, so a refusal now names the resource that was short instead of naming the queue.ReasonQueueIntentgoes with it: nothing emits it any more, and no dashboard or alert rule referenced it.The test that asserted the old behaviour is replaced by its inverse. Negative control: restoring the gate with the new test unchanged fails on
Deferred.https://claude.ai/code/session_017rA31sF7EzvtbRmi18Uzz9