v2.0.0

PinchBench 2.0.0 🦀

A major release with significant expansion of the benchmark suite and infrastructure improvements.

Highlights

• 148 tasks (up from ~25 in v1.x) — nearly 6x more comprehensive benchmark coverage
• Parallel judge execution — overlaps grading with task execution for faster benchmarks
• Haiku judge by default — faster grading without sacrificing accuracy
• Thinking-level support — benchmark models at different reasoning intensities
• Axiom observability — detailed logging for benchmark runs
• Multi-turn session isolation — new_session:true support for multi-turn tasks

New Task Categories

Data Analysis (CSV)

• US cities, iris flowers, temperature, life expectancy, weather stations, GDP, pension funds, Apple stock analysis

Meeting & Document Analysis

• Government meetings, advisory boards, city council, tech meetings
• Executive summaries, sentiment analysis, action items, Q&A extraction

Log Analysis

• Nginx, Apache, SSH, HDFS, MapReduce, and Syslog analysis tasks (24 new tasks)

Development & DevOps

• CI/CD debugging, Kubernetes issues, Dockerfile optimization
• Multi-file refactoring, test generation, commit message writing
• Git rescue, shell generation tasks

Image & PDF

• Image identification tasks
• PDF to calendar import

Research & Writing

• Market research, email drafting, Todoist cleanup
• Contract/legal analysis

Infrastructure Improvements

• --core flag for running core task subset
• --trend and --trend-window flags for post-run analysis
• Judge result caching to avoid redundant grading
• Manifest-based task ordering (replaced numbered IDs)
• CI workflow for manifest linting
• Better OpenClaw transcript compatibility

Breaking Changes

• Task IDs now use manifest-based ordering instead of numbered prefixes
• Default judge backend changed from openclaw to api

Full Changelog

111 commits since v1.2.1. See compare view for complete details.

---

🦞 Snip snip — benchmarking one claw at a time