Aeval v0.1.0 — initial release / 首个公开版本

Aeval v0.1.0 — initial release / 首个公开版本

Aeval is an open-source agent evaluation framework driven by OpenTelemetry traces: define suites in YAML, run your agent repeatedly, grade each trial, aggregate into pass@k / pass^k / consistency / saturation.

Naming: repo is Aeval; PyPI distribution is aeval-framework (pip install aeval-framework), Python module is agent_eval, CLI is eval-suite — same convention as pip install pillow → import PIL. (PyPI agent-eval belongs to an unrelated project.)

Features / 功能

• Suite as YAML — declarative suites with strict validation (semver, unique task ids) / 用 YAML 声明评测套件，严格校验
• 9 built-in graders — code checks, LLM-as-judge, environment state, tool-call validation, transcript analysis, artifact checks, human-in-the-loop, step-level, metric dispatch / 9 个内置评分器
• Statistically sound aggregation — pass@k (capability) & pass^k (reliability) with binomial extrapolation, consistency & saturation detection / pass@k 与 pass^k 统计聚合
• Pluggable everything — AgentRunner / TraceProvider / Storage / Environment / Grader are small protocols / 全组件可插拔
• REST API & SSE — mount in any FastAPI app or serve standalone (/v1), live run event streaming
• CLI — eval-suite run / validate / list / show / compare / serve
• Datasets & LLM metrics — build datasets from traces, RAG-style quality metrics (answer relevancy, faithfulness, context recall/precision) / 数据集构建与 RAG 质量指标
• Dashboard — Next.js app (overview, suite management, live run reports, trial drill-down, A/B comparison) / 可视化控制台

Install / 安装

pip install aeval-framework            # core (orchestration, graders, storage)
pip install "aeval-framework[api]"     # + REST API service
pip install "aeval-framework[cli]"     # + eval-suite CLI

Offline quickstart: eval-suite run examples/minimal/suite.yaml / 离线快速开始

Known Limitations / 已知限制

• Transcript process metrics lean optimistic in AChat — n_turns / token metrics depend on OTel span coverage that is incomplete in AChat sessions (toolcalls dimension fully covered) / transcript 过程指标在 AChat 上偏乐观（toolcalls 维度完整）
• RAG & orchestration scenarios in live calibration — mechanics shipped and tested (env.agent_id + rag_search, dispatch_mode + achat_dispatch), not yet production-calibrated / 机制就绪并通过测试，实战校准中
• PostgreSQL storage & A/B significance testing are Phase 3 / PostgreSQL 存储与 A/B 显著性检验为 Phase 3
• No human review UI yet — human grader works as pending state + API score callback / human grader 为 pending 语义 + API 回传，评审 UI 未含

---