[RFC] Agent Engineering Benchmark / Agent 工程评估体系

[CodeCasterX]

Status / 状态

Discovery / RFC — defining the evaluation contract and the minimum viable benchmark before implementation.

探索与方案阶段——先明确评估契约和最小 Benchmark，再进入实现。

Vision / 愿景

FIT Lab - AI is building an Agent Engineering Benchmark to measure whether AI software-engineering workflows produce correct, stable, efficient, and auditable outcomes—not merely whether they follow a prescribed process.

FIT Lab - AI 正在建设 Agent Engineering Benchmark，用于衡量 AI 软件工程流程是否真正交付正确、稳定、高效且可审计的结果，而不只是验证流程是否遵守既定步骤。

The benchmark is intended to evaluate reusable skills, stage handoffs, complete workflows, different AI tools, and future third-party agent infrastructure under a consistent protocol.

该体系将以统一协议评估单个 Skill、阶段交接、完整工作流、不同 AI 工具，以及未来的第三方 Agent 基础设施。

Proposed Architecture / 总体架构

The system separates the public evaluation framework from private holdout data:

评估系统将公开框架与私有保留测试集分离：

agent-infra (public / 公开)
├── benchmark runner and subject adapters
├── case and grader contracts
├── isolated execution and reporting
└── non-sensitive example cases

private benchmark dataset (planned / 规划中)
├── active case definitions
├── mutation locations and seeds
├── hidden tests and gold patches
└── answer-sensitive traces and grading details

agent-infra defines how to evaluate. The private dataset defines what to evaluate and what counts as correct.

agent-infra 负责定义“如何评估”；私有数据集负责保存“评估什么，以及什么结果才算正确”。

During a run, the Agent may access only a disposable challenge workspace. It must not receive the private dataset, repository credentials, hidden tests, gold patches, answer-bearing Git history, or unrestricted network access.

运行时，Agent 只能访问一次性的挑战工作区，不得访问私有数据集、仓库凭证、隐藏测试、Gold patch、包含答案的 Git 历史或不受限网络。

MVP / 最小可行版本

The MVP deliberately starts with three manually qualified private cases. Random generation, multilingual coverage, LLM judges, dashboards, and public leaderboards are deferred until the basic evaluation loop proves reliable.

MVP 刻意从 3 道经过人工验题的私有 Case 开始。随机生成、多语言、LLM Judge、Dashboard 和公开排行榜均推迟到基础闭环被证明可靠之后。

MVP success criteria:

MVP 成功标准：

• Define the benchmark objective, threat model, case contract, and grader contract.

• Prepare three private cases whose healthy versions pass, injected versions fail, and valid repairs pass hidden evaluation.

• Generate disposable challenge repositories without answer-bearing history or remotes.

• Run hidden tests, regression tests, and build checks outside the Agent-visible environment.

• Compare direct repair with the complete agent-infra workflow using the same model, tools, permissions, and budget.

• Repeat each case three times per subject and publish a sanitized A/B report.

• 明确 Benchmark 目标、威胁模型、Case 契约和 Grader 契约。

• 准备 3 道私有题：健康版本通过、注入版本失败、有效修复通过隐藏评估。

• 生成不包含答案历史和 remote 的一次性挑战仓库。

• 在 Agent 不可见的环境中执行隐藏测试、回归测试和构建检查。

• 在相同模型、工具、权限和预算下，对比直接修复与 agent-infra 完整流程。

• 每个 Subject 每题重复 3 次，并发布脱敏后的 A/B 报告。

Workstreams / 工作流

Public framework / 公开框架

Tracked in agent-infra implementation initiative #741. Its sub-issues cover the public protocol, runner, isolation, grading, reproducibility metadata, and sanitized reporting.

由 agent-infra 实施跟踪 #741 承载，其 sub-issues 覆盖公开协议、Runner、隔离、评分、可复现元数据和脱敏报告。

Private holdout dataset / 私有保留测试集

Private repository created; access restricted. Its tracking space holds case content, hidden oracles, qualification evidence, and dataset lifecycle details. The repository URL is intentionally not published here. Only dataset version, case count, qualification status, and sanitized aggregate results will be exposed publicly.

私有仓库已创建，访问受限。 其跟踪空间保存题目内容、隐藏 Oracle、验题证据和数据集生命周期细节；本 Discussion 不公开仓库入口。公开侧只披露数据集版本、题目数量、qualification 状态和脱敏汇总结果。

Current Decisions / 当前决策

• The organization Discussion is the canonical initiative-level roadmap and decision record.

• The organization Profile README links here instead of duplicating a mutable plan.

• Repository Issues track implementation only; they are not the source of initiative-level design decisions.

• Active benchmark cases remain private and are versioned independently from the public framework.

• The first three cases are manually curated; automated seeded mutation generation is a later phase.

• Public software-engineering benchmarks may be used for runner compatibility, but not as the sole private capability signal.

• 组织 Discussion 是 Initiative 级路线图与决策记录的唯一权威入口。

• 组织 Profile README 只链接本 Discussion，不复制易过期的详细计划。

• 仓库 Issue 只跟踪具体实施，不承担 Initiative 级设计决策。

• 活跃 Benchmark Case 保持私有，并与公开框架独立版本化。

• 首批 3 道题人工构建；自动化 seeded mutation generation 属于后续阶段。

• 公开 SWE Benchmark 可用于 Runner 兼容性验证，但不作为唯一的私有能力信号。

Open Questions / 开放问题

• Which minimal external fixture best represents the first three cases?

• What exact resource budget and repetition policy should define the first A/B run?

• Which execution boundary is sufficient for the MVP, and which guarantees require container isolation?

• When should retired private cases be published, if ever?

• 哪个最小外部 fixture 最适合承载首批 3 道题？

• 首次 A/B 运行应固定怎样的资源预算与重复策略？

• MVP 需要怎样的执行边界，哪些安全保证必须依赖容器隔离？

• 退休的私有 Case 是否以及何时公开？

This Discussion will be updated when initiative-level decisions change. Concrete implementation progress belongs to the linked repository tracking Issues.

当 Initiative 级决策发生变化时更新本 Discussion；具体实施进度记录在关联仓库的跟踪 Issue 中。