Skip to content

The DFlash2 speed gate records vLLM's acceptance but not ours, so the 0.928x gap cannot be attributed #2832

Description

@localai-org-maint-bot

Row: SPEC-DFLASH2

The gap now has two measured denominators and no attribution

.agents/benchmark-record.md records ours 14.951 vs vLLM 16.111 (0.9280x). A same-workload SGLang DFlash2 run on dgx:gpu0 (pin 1cf2b8c54d, identity gate green, WORKLOAD_FINGERPRINT_MATCHES_GATE=True, flashinfer top-k called and agreeing with torch.topk) folds by the gate's own predicate to 16.034, i.e. 0.9325x.

Two independent oracles agree within 0.5% of each other and we are ~7% behind both. That makes the gap an engine property rather than a denominator artefact — but nothing in the record says WHICH property.

The missing datum

The oracle arms record acceptance; our arm does not.

engine tokens per verify step accept rate source
vLLM 1000/310 + bonus = 4.23 1000/2170 = 46.1% evidence-d1/vllm-arm.json metrics.vllm:spec_decode_num_*
SGLang 4.67 - primary-legs.jsonl accept_length, from spec_verify_ct
ours NOT RECORDED NOT RECORDED our-arm.json legs carry only tok_s

our-arm.json's leg keys are completion_tokens, finish_reason, generate_start_unix, generate_end_unix, prompt_tokens, run, secs, tok_s. No acceptance field exists anywhere in the file.

Why this blocks the row

The two oracles reach the same throughput by different balances: SGLang accepts more per step than vLLM and lands at the same tok/s. So throughput alone does not identify the lever, and ours is unmeasured on the axis that separates them.

Choosing between those with no measurement is guessing, and the row has an open history of acceptance moving under configuration: a prior finding recorded acceptance collapsing for part of a concurrent run, and another recorded an MMA kernel arm changing acceptance from 8.7% to 41.6%.

What is asked

Capture acceptance on our arm in tools/bench/dflash2_oracle_capture.py, in the same run and the same units as the oracle arms, so the three are directly comparable. The engine already computes it: examples/bench/bench_core.h:161 and :871 report "Acceptance rate (accepted/proposed)". The gate drives examples/cli, which does not surface it.

Report per leg, not only as a run total, because a mean hides the collapse shape the row has seen before. Fold acceptance with the same warm-leg predicate the throughput axis uses, so no leg population differs between the two axes.

Not in scope

Any change to the decode path, the drafter, or the veto. This issue adds an instrument and changes no product behaviour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions