Row: SPEC-DFLASH2
The gap now has two measured denominators and no attribution
.agents/benchmark-record.md records ours 14.951 vs vLLM 16.111 (0.9280x). A same-workload SGLang DFlash2 run on dgx:gpu0 (pin 1cf2b8c54d, identity gate green, WORKLOAD_FINGERPRINT_MATCHES_GATE=True, flashinfer top-k called and agreeing with torch.topk) folds by the gate's own predicate to 16.034, i.e. 0.9325x.
Two independent oracles agree within 0.5% of each other and we are ~7% behind both. That makes the gap an engine property rather than a denominator artefact — but nothing in the record says WHICH property.
The missing datum
The oracle arms record acceptance; our arm does not.
| engine |
tokens per verify step |
accept rate |
source |
| vLLM |
1000/310 + bonus = 4.23 |
1000/2170 = 46.1% |
evidence-d1/vllm-arm.json metrics.vllm:spec_decode_num_* |
| SGLang |
4.67 |
- |
primary-legs.jsonl accept_length, from spec_verify_ct |
| ours |
NOT RECORDED |
NOT RECORDED |
our-arm.json legs carry only tok_s |
our-arm.json's leg keys are completion_tokens, finish_reason, generate_start_unix, generate_end_unix, prompt_tokens, run, secs, tok_s. No acceptance field exists anywhere in the file.
Why this blocks the row
The two oracles reach the same throughput by different balances: SGLang accepts more per step than vLLM and lands at the same tok/s. So throughput alone does not identify the lever, and ours is unmeasured on the axis that separates them.
Choosing between those with no measurement is guessing, and the row has an open history of acceptance moving under configuration: a prior finding recorded acceptance collapsing for part of a concurrent run, and another recorded an MMA kernel arm changing acceptance from 8.7% to 41.6%.
What is asked
Capture acceptance on our arm in tools/bench/dflash2_oracle_capture.py, in the same run and the same units as the oracle arms, so the three are directly comparable. The engine already computes it: examples/bench/bench_core.h:161 and :871 report "Acceptance rate (accepted/proposed)". The gate drives examples/cli, which does not surface it.
Report per leg, not only as a run total, because a mean hides the collapse shape the row has seen before. Fold acceptance with the same warm-leg predicate the throughput axis uses, so no leg population differs between the two axes.
Not in scope
Any change to the decode path, the drafter, or the veto. This issue adds an instrument and changes no product behaviour.
Row:
SPEC-DFLASH2The gap now has two measured denominators and no attribution
.agents/benchmark-record.mdrecords ours 14.951 vs vLLM 16.111 (0.9280x). A same-workload SGLang DFlash2 run ondgx:gpu0(pin1cf2b8c54d, identity gate green,WORKLOAD_FINGERPRINT_MATCHES_GATE=True, flashinfer top-k called and agreeing withtorch.topk) folds by the gate's own predicate to 16.034, i.e. 0.9325x.Two independent oracles agree within 0.5% of each other and we are ~7% behind both. That makes the gap an engine property rather than a denominator artefact — but nothing in the record says WHICH property.
The missing datum
The oracle arms record acceptance; our arm does not.
evidence-d1/vllm-arm.jsonmetrics.vllm:spec_decode_num_*primary-legs.jsonlaccept_length, fromspec_verify_ctour-arm.jsonlegs carry onlytok_sour-arm.json's leg keys arecompletion_tokens, finish_reason, generate_start_unix, generate_end_unix, prompt_tokens, run, secs, tok_s. No acceptance field exists anywhere in the file.Why this blocks the row
The two oracles reach the same throughput by different balances: SGLang accepts more per step than vLLM and lands at the same tok/s. So throughput alone does not identify the lever, and ours is unmeasured on the axis that separates them.
A2async chain (SPEC-DFLASH2 A2-2: device-resident verify, the wave that closes reason B #2802 and its successors) is the right ladder.A2chain is the wrong ladder for closing 7%.Choosing between those with no measurement is guessing, and the row has an open history of acceptance moving under configuration: a prior finding recorded acceptance collapsing for part of a concurrent run, and another recorded an MMA kernel arm changing acceptance from 8.7% to 41.6%.
What is asked
Capture acceptance on our arm in
tools/bench/dflash2_oracle_capture.py, in the same run and the same units as the oracle arms, so the three are directly comparable. The engine already computes it:examples/bench/bench_core.h:161and:871report "Acceptance rate (accepted/proposed)". The gate drivesexamples/cli, which does not surface it.Report per leg, not only as a run total, because a mean hides the collapse shape the row has seen before. Fold acceptance with the same warm-leg predicate the throughput axis uses, so no leg population differs between the two axes.
Not in scope
Any change to the decode path, the drafter, or the veto. This issue adds an instrument and changes no product behaviour.