community: Submit your orchestration benchmarks to bernstein.run

[chernistry]

The benchmark page is live under docs/benchmarks/ (BENCHMARKS.md plus leaderboard.html) and links back to this thread for submissions. This thread is the ongoing place to submit a run from your own setup, so the page carries a reproducible cost and performance reference across hardware and model mixes. Your handle is credited on each row.

What to submit

Two kinds of numbers are useful.

Real-run cost and performance (per session):

bernstein cost --json

Capture:

• agents / tasks completed
• models used (Opus, Sonnet, Haiku, GPT, Gemini, and so on)
• total cost and cost-per-task
• wall-clock time
• hardware (for example, M2 MacBook, Linux CI runner)
• project size (LOC or repo link)

Component benchmarks (hardware capacity numbers):

uv run python benchmarks/bench_orchestrator.py
uv run python benchmarks/bench_task_store.py
uv run python benchmarks/bench_quality_gates.py
uv run python benchmarks/bench_startup.py

Include your hardware, OS, Bernstein version, and Python version.

How to submit

1. Comment on this thread with the JSON output or the component-benchmark table.
2. Or open a PR adding a row under "Community-submitted benchmarks" in docs/benchmarks/BENCHMARKS.md.

What already exists

• docs/benchmarks/BENCHMARKS.md - the published page: a simulated DAG-scheduling section, a SWE-Bench Lite section, and component benchmarks.
• docs/benchmarks/leaderboard.html - the rendered page.
• bernstein cost --json - cost and duration schema for a real run.
• Component benchmark scripts under benchmarks/.
• YAML eval harness (feat(eval): YAML eval harness #1565) - operator-runnable spec format with judge and golden diff (src/bernstein/eval/harness.py, docs/eval/yaml-harness.md).
• Flake quarantine for reproducible runs (feat(ci): add ctrf test reporter + pytest-xflaky flake quarantine #1502) - .github/workflows/flake-quarantine.yml.
• SWE-Bench harness under benchmarks/swe_bench/ for verified eval runs.

Notes

• Community rows are grouped under "Community-submitted benchmarks" in BENCHMARKS.md. One row has already landed (component benchmarks on a low-end laptop).
• The simulated DAG numbers on the page are a scheduling estimate, not measured agent runs. Real submissions fill in the measured picture.
• Only artifacts marked verified from the SWE-Bench harness are used for public benchmark claims.

[snakefood3232]

I can set up the benchmarks page with a submission pipeline. I’d handle this by creating a structured markdown table in docs/benchmarks.md and a script to validate and format incoming data from bernstein cost outputs. I’ve built similar public benchmark pages before and can have a PR up in a day.

[chernistry]

@snakefood3232 sorry for the slow reply, and thanks for the offer. Since April the page itself has landed: docs/benchmarks/ now carries BENCHMARKS.md plus a rendered leaderboard.html, so the table structure is covered.

The validation piece is still open and still wanted: a small script that takes bernstein cost --json output, checks required fields, and emits a formatted leaderboard row would make submissions much less error-prone. If you are still up for it, a PR adding something like scripts/validate_benchmark_submission.py is very welcome.

Housekeeping note: this is a standing submission channel rather than a work item, so it is moving to Discussions. The URL will redirect and submissions continue there.