feat(serving): predictive-scheduling instrument — measured schedulable coverage from the live trace (#281/#282) - #2122
Merged
Conversation
…e coverage from the live trace (#281/#282) Joel's directive: use the ML advantage for predictive scheduling. Before any CUDA copy-stream work, the go/no-go must be MEASURED, not assumed: exposed H2D per token = (1 − schedulable coverage) × the ~11GB expert working set. This slice wires the #276-ported CrossLayerExpertPredictor into the trace tail as a cross-TOKEN transition model and measures, live from the real trace, the two halves of coverage: - repeat_recall: experts already in the previous token's set — what pure recency residency covers for free (her measured 4416-stable warm set says this is high on decode). - predicted_delta_recall: of the NON-repeat delta, how many the predictor called one token ahead — what prefetch adds on top. - schedulable_coverage = both, over all experts: THE number that sizes her DeviceUploadFetcher's win before it's built. Published on the serving.moe_host_cache_lease probe every plan write (repeat/delta/coverage ×100). predicted_next() exposes the live delta prediction — the future plan-file prefetch list (#273's third axis), held back from the wire until the consumer coordinates the extension. All prediction state resets with the stream (truncation/geometry) — stale transitions never leak across serves. Test pins the instrument's honesty: a repeating stream scores 100% recency/zero delta; an alternating DISJOINT stream scores 0% recency but the predictor learns the cycle and delta recall climbs ≥50% — prediction covering exactly what recency cannot. First token scores nothing (honest None, never a fake 100). trace_tail 5/5; serving_daemon suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Joel's directive: use the ML advantage for predictive scheduling. Before any CUDA copy-stream work, the go/no-go is MEASURED, not assumed: exposed H2D per token = (1 − schedulable coverage) × the ~11 GB expert working set.
This wires the #276-ported
CrossLayerExpertPredictorinto the trace tail as a cross-TOKEN transition model and measures, live from the real trace:All three publish on the
serving.moe_host_cache_leaseprobe each plan write.predicted_next()exposes the live delta prediction — the future plan-fileprefetchlist (#273's third axis), held off the wire until the consumer coordinates the extension. Prediction state resets with the stream (truncation/geometry) — stale transitions never leak across serves.The test pins the instrument's honesty: a repeating stream = 100% recency, zero delta; an alternating DISJOINT stream = 0% recency but delta recall climbs ≥50% as the predictor learns the cycle — prediction covering exactly what recency cannot. First token scores None, never a fake 100.
trace_tail 5/5; serving_daemon 26/26.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo