What should TraceArena measure in your agent world? | 你的 Agent 世界最需要验证什么？

[tonyhyworld]

TraceArena is now public. The first no-key Market Replay is deliberately small: it lets you inspect the runtime path from evidence and structured actions to world events, settlement, and replay without an API key or brokerage connection.

We are using this thread to shape the next public scenarios and evaluation adapters.

What would you most like to test?

• a workflow where a wrong action has a measurable business cost;
• a tool or provider adapter;
• deterministic replay and regression across model versions;
• failure recovery, permissions, evidence, or authoritative settlement;
• a non-financial scenario pack that others can run locally.

If you have an existing Agent, please include:

1. the decision or outcome that must be trusted;
2. the tools and external systems involved;
3. what counts as pass/fail;
4. whether a synthetic fixture is enough for a first public reproduction.

Links:

• Live demo: https://tonyworld888-tracearena-demo.static.hf.space/index.html
• Quickstart: https://github.com/tonyhyworld/TraceArena/tree/main/examples/market_replay
• Scenario guide: https://github.com/tonyhyworld/TraceArena/blob/main/docs/scenario-pack-guide.md
• Evaluation pilot form: https://github.com/tonyhyworld/TraceArena/blob/main/.github/ISSUE_TEMPLATE/agent-evaluation-pilot.yml

Please share constraints and failures as well as successful demos. We will summarize recurring requests into public issues and scenario-pack priorities.

[tonyhyworld]

v0.1.6 adds one concrete lesson from the first replay report: a run needs two different hashes. canonical_replay_sha256 remains the full-file integrity digest, while deterministic_replay_sha256 removes per-run identity and wall-clock fields so the same fixture can be compared across runs.

That distinction makes the evaluation question more precise: should a scenario optimize for semantic determinism, recovery quality, evidence completeness, or all three? If you have an agent workflow, please share the outcome that must be trusted and the failure that would be most expensive to miss.

Release: https://github.com/tonyhyworld/TraceArena/releases/tag/v0.1.6
Quickstart: https://github.com/tonyhyworld/TraceArena/blob/main/docs/quickstart.md