Skip to content

v0.2.0 — Industry-standard benchmark, bilingual docs, docs site

Choose a tag to compare

View all tags
@Madapexai Madapexai released this 18 Jul 22:32
v0.2.0
647d544

Highlights

Quark Agent v0.2.0 ships the first industry-standard evaluation, a fully rewritten bilingual README, an MkDocs Material docs site, and the contributor infrastructure (CONTRIBUTING, Issue/Discussion templates) for community growth.

New Features

  • 200-task benchmark suite on AgentBench (THUDM) + GAIA (Meta/HF) — 178/200 pass, 89.0% overall
    • GAIA: 93.0% (L1 97.5%, L2 94.3%, L3 84.0%)
    • AgentBench: 85.0% (DB-Bench 76.0%, OS-Interaction 93.3%, Knowledge-Graph 95.0%)
  • Interactive HTML report with task-level filtering, tool-call expansion, and latency stats
  • Bilingual README (English + Simplified Chinese) with codex/aider-style hero, badges, ToC
  • MkDocs Material docs site at madapexai.github.io/quark-agent (auto-deploys on push to master)
  • Issue / Discussion templates (YAML form style) for bug reports, feature requests, ideas, Q&A, show & tell
  • CONTRIBUTING.md with the 60-second defineAction path and PR checklist

Bug Fixes

  • Set tool_choice: "auto" so providers no longer skip tool calls
  • Skip the duplicate text event when content was already streamed via text_delta
  • Added a "Tool Use Policy" block to the system prompt so the LLM reliably calls read_file / list_dir / run_code / web_fetch for real-time data instead of answering from memory

Breaking Changes

None.

Reproduce

PORT=3465 node e2e/full-eval.mjs --bench all   # ~75 min, ~200 LLM calls
node e2e/gen-html-report.mjs                     # regenerate the HTML report

Benchmark Snapshot

Benchmark Total Pass Fail Error Pass Rate
GAIA (Meta/HF) 100 93 7 0 93.0%
AgentBench (THUDM) 100 85 9 6 85.0%
Overall 200 178 16 6 89.0%

GAIA by difficulty — L1: 97.5% · L2: 94.3% · L3: 84.0%
AgentBench by category — DB-Bench: 76.0% · OS-Interaction: 93.3% · Knowledge-Graph: 95.0%

Full trace-level reports attached as release assets:

  • full-eval-results.json — raw trace for all 200 tasks
  • FULL-EVAL-REPORT.html — interactive report, open in a browser
  • FULL-EVAL-REPORT.md — full markdown report
Assets 5
Loading