RFC: llmfit bench --share — contribute your benchmark results as a pull request

[AlexsJones]

We just shipped measured ✓ tok/s provenance in 1.0 (#708) — when llmfit has real benchmark data for your hardware, you see measurements instead of estimates. The obvious next step: let llmfit bench contribute your results back, so coverage grows beyond a handful of popular GPUs to the long tail where estimates are weakest.

We're proposing a repo-native approach: your benchmark run becomes a pull request. No third-party service, no telemetry endpoint. Seeking input before we build.

The idea

llmfit bench --all --share
  → runs a standardized sweep (fixed prompt, batch 1, warmup discarded, median of 3)
  → prints the results table AND the exact JSON that would be contributed
  → "Contribute 4 results as a PR to AlexsJones/llmfit-benchmarks? [y/N]"
  → forks, commits your results file, opens the PR as *you*

Every data point in the corpus is then a reviewed PR with full provenance: who submitted it, on what hardware, with what engine version — auditable by anyone, forever. CI validates the schema and sanity-checks the numbers (reject tok/s wildly above the hardware's physical ceiling); in-envelope submissions can auto-merge. Merged data ships to everyone in the next release via the same embedded-cache pipeline that powers the ✓ badges today.

Why PRs instead of an upload API

• Transparency is the product. After the estimate-accuracy journey (tok/s overestimation (~3×) for Qwen3 30B vs. LM Studio benchmark #449, For the given recommendation model, can you provide the corresponding llama.cpp runtime parameters? #292), "every number is a public commit you can audit" beats "trust our database."
• Attribution and incentive — your GitHub handle is on every contribution; the contributors graph is the leaderboard.
• Moderation for free — spam/garbage data has to survive code review + CI validation, and git revert is the correction mechanism.
• No server to run, secure by construction — nothing to breach, no auth system to build; the PR flow uses your existing GitHub identity via gh.
• Friction is a feature — one deliberate confirm beats silent telemetry. Nothing is ever sent without you seeing the exact payload first (--dry-run prints it and exits).

Hard requirements (non-negotiable)

• Opt-in per invocation (--share), no config default, no auto-upload — v1 asks every time.
• No identifiers beyond what you choose to publish via the PR: hardware model, RAM/VRAM, OS family, engine + version, quant, measured tok/s + TTFT. No hostnames, no serials.
• Only measured timings qualify — e.g. Ollama's eval_duration is real measurement; providers that report estimated timing are excluded from contribution.

Open questions — we want your input

1. Separate llmfit-benchmarks repo vs. in-tree data dir? Separate keeps the main repo lean and lets the corpus grow freely; in-tree keeps everything in one place. (Currently leaning: separate repo, aggregated + embedded into llmfit at release time.)
2. No gh CLI installed — fallback options: print the JSON + a prefilled "create PR" URL? GitHub device-flow auth built into llmfit? Just require gh?
3. File layout for mergeability — one file per (hardware, engine, model) so parallel contributions never conflict? Monthly aggregation to keep file count sane?
4. Auto-merge policy — should schema-valid, in-envelope submissions merge without human review, or does every data point deserve eyes on it (at least initially)?
5. What metadata matters to you when deciding whether to trust a community number — engine build flags? Power/thermal state? Driver version?

This supersedes the upload-API direction from #81/#112 discussions. If the shape lands well, milestone 1 is: standardized sweep + --share PR flow for a single model, with --dry-run from day one.