make bench is the command the README points at for every published performance claim. make help still describes it as something else.
Where
Makefile, the help target — the "Bench" block:
make bench | bench-mm raw probes | multimodal encoders
"raw probes" was true when bench filtered the bench_* test binaries. That is make bench-smoke today. make bench now runs tools/bench_reproduce.py: it downloads the BitNet GGUF, runs a frozen protocol, and measures a llama.cpp/bitnet.cpp baseline in the same run if it finds one.
make fetch-bench-model is not listed at all, though make bench depends on it and a user may well want the model without the benchmark.
Reproduce
make help | grep -A6 Bench
Compare against what the targets actually do (Makefile, the # ---- make bench block) and against the README's "Don't trust the numbers, run them" section.
Done when
make help describes bench as the reproducible cross-engine benchmark, lists bench-smoke separately as the raw probes, and mentions fetch-bench-model. No behaviour change — this is the help text only.
make benchis the command the README points at for every published performance claim.make helpstill describes it as something else.Where
Makefile, thehelptarget — the "Bench" block:"raw probes" was true when
benchfiltered thebench_*test binaries. That ismake bench-smoketoday.make benchnow runstools/bench_reproduce.py: it downloads the BitNet GGUF, runs a frozen protocol, and measures a llama.cpp/bitnet.cpp baseline in the same run if it finds one.make fetch-bench-modelis not listed at all, thoughmake benchdepends on it and a user may well want the model without the benchmark.Reproduce
Compare against what the targets actually do (
Makefile, the# ---- make benchblock) and against the README's "Don't trust the numbers, run them" section.Done when
make helpdescribesbenchas the reproducible cross-engine benchmark, listsbench-smokeseparately as the raw probes, and mentionsfetch-bench-model. No behaviour change — this is the help text only.