I normalized 638 measurements from my 4x3090 rig

[alesha-pro]

After the work in #773, I went back through the rest of my benchmark history and cleaned it up.

I have 638 actual measurements from 20 benchmark campaigns on the same 4x RTX 3090 rig, across 14 model families. Most are vLLM runs. There is a smaller llama.cpp set, plus 38 exllamav3 records from a short experiment. I would not read much into that exllamav3 slice by itself, but I can include it if anyone is interested.

I normalized model and checkpoint names, engine versions, TP/PP/DP and replicated layouts, context, concurrency, KV dtype, quantization, spec decode, and power limits. I kept AutoRound INT4 separate from AWQ, generic or official INT4, and W4A16. Out of the 638 measurements, 637 still have the exact launch argv or command.

The catch is that this is not one giant comparison matrix. My harness and the thing I was testing changed between campaigns, so some rows should not be compared. I kept the original labels and source references, and marked fields inferred during cleanup.

Rather than drop a 638-row JSONL here, I can pull a matched slice, put it into the club-3090 Results Card or BENCHMARKS format, and rerun an important pair against a current recipe if needed.

The best-covered parts are quantization comparisons on Ampere and KV-cache behavior as context grows. I also have topology and engine-version runs, although coverage varies. Which missing comparison would be useful to tackle first?

[noonghunna]

637 of 638 with the exact argv, non-comparables kept labeled instead of merged, inferred fields marked — that's a corpus, not a pile of numbers. Direct answers to "which comparison first," in priority order, and then a suggestion for what this could become beyond doc rows.

1. First ask — and it needs zero new runs: do the KV backend tiers generalize across model families?

Everything we know about the backend-tier depth story (fp8→FLASHINFER flat, int8-PTH→TRITON cratering, crossover ~7–10K — your #773 data, @Whamp's TP=4 series, our #594 originals) is Qwen-family only. You have 14 families. The question your corpus may already answer: does the tier structure hold where the candidate lists differ? Gemma-4-class models force TRITON_ATTN outright (heterogeneous head dims — we track this upstream), so their "tiers" should look completely different; nemotron-class and the rest are unmeasured. A per-family slice of KV dtype × context depth → decode, even sparse, either confirms the tiers as an Ampere-general mechanism or finds the family where the story breaks. That's the highest-value query I can think of against data you already hold.

2. The quant gap we've had on the books: fp8-static vs AutoRound INT4, matched

Our fast/max tier row carries a literal "fp8-vs-autoround unbenched" debt on our side. Your OP already showed both at matched 220 W (no-spec arms); if the corpus has matched pairs at other contexts/engine versions — and especially your kept-separate INT4 lineages (AutoRound vs AWQ vs generic INT4 vs W4A16) under one power cap — that's a BENCHMARKS-ready Ampere quant matrix filling a named hole. This is the slice I'd take in Results Card form first.

3. Replication-vs-TP as a curve, not two points

MULTI_CARD's new replication-vs-TP section exists because of your Finding 4, and it currently rests on peak points (3.4× / 2.07–2.40×). If the corpus holds the intermediate layouts (1×TP4 / 2×TP2 / 4×TP1 across concurrency ladders), the section upgrades from "directional rule" to an actual frontier plot.

4. The strategic one: map a sample onto our measurement-record schema

This is the suggestion that outlives any single table. The repo ships a frozen measurement-record schema (scripts/lib/profiles/measurement_record.py) — every first-party bench emits one structured record, and that corpus is the designed seed for the compose-optimizer work (the thing that will eventually recommend configs from measured data instead of tables). Your normalized 638 — with argv, engine versions, layouts, KV dtype, power caps — is precisely the community-corpus case that schema was built to eventually ingest. Concrete ask: gist 5–10 rows mapped onto the schema fields (don't transcribe all 638); we diff your normalization against the frozen fields and see what's lossy. If it fits, your history becomes optimizer fuel, which is worth more than any number of BENCHMARKS rows.

Mechanics, so the good discipline survives contact with our table

Per-comparison Results Cards in this thread rather than the full JSONL; keep your inferred-field markers (they're the audit trail); anything destined for BENCHMARKS carries the Rig-cell convention fields — power cap, interconnect state, resolved custom-AR state — which your rows mostly already have by construction. Rows you marked non-comparable stay out of cross-comparisons no matter how tempting; your own labeling instinct is the policy.

On the slices we'd park: the exllamav3 set (your own caveat is right — 38 records from a short experiment; if we ever build out the engines survey it gets a caveated mention, not rows), and pure engine-version deltas — though as context: the dual-fast v0.25.1 first-party re-run answering your #773 version question is literally executing while this posts. Fresh numbers land in that thread.

So: start with §1 — it's a query, not a campaign — and the §4 schema sample whenever convenient. §2 is the first slice we'd fold into the table. And thank you for asking "which comparison is useful" instead of posting all 638 — that question is why this corpus will actually get used.

[noonghunna]

lol my claude is in space! :)

[alesha-pro]

yeah I see))