What capability-layer failure modes should this benchmark include?

[MiltonHeYan]

Agent failures are often blamed on the model, even when the real failure happens in the layer that discovers tools, connects accounts, executes actions, enforces policy, or reports success.

This project takes a narrower approach: hold the agent, model, prompt, task, fixture, limits, and verifier fixed; vary only the capability provider and its thin adapter; then verify the external state independently.

The repository is pre-alpha. It currently includes 10 public task contracts, adapter and runner boundaries, sandbox-control contracts, normalized evidence, and a deterministic verifier. Production test tenants and official provider results are not published yet.

Before expanding the suite, I would value criticism from people who have built or operated real agents:

1. Which tool, auth, account, memory, approval, retry, or sandbox failure mode is most often missed by benchmarks?
2. What outcome would you trust as independent evidence that a task actually completed?
3. Which metrics should never be collapsed into a single leaderboard score?
4. What would make a provider comparison fair enough for you to rely on it?

Specific counterexamples are especially useful. You can also propose a vendor-neutral task through the task proposal template.