feat: save agent traces to disk in OTLP-JSON for benchmark evaluation

[ShuxinLin]

Adds an evaluation-records persistence layer for AssetOpsBench: every agent run emits two artifacts joined by run_id — an OpenTelemetry root span (metadata + aggregate metrics) and a per-run trajectory JSON (per-turn content) — both written directly by the agent process. No Docker, no Collector, no live backend required.

Closes #270.

Motivation

Per-run token usage and tool-call detail used to live only in the in-process AgentResult.trajectory object and vanish when the command exited. For a benchmark we need durable records that survive runs, are keyed to a scenario, and can be analysed offline or replayed into an observability backend later. Rather than invent a custom JSON schema, this PR uses OTLP — the format every OTEL backend already speaks — for the trace half, and a small companion JSON file for the per-turn content.

Two-artifact model

Spans and trajectories carry disjoint data:

• Trace (OTEL_TRACES_FILE=…/traces.jsonl) — root span per run with metadata + aggregate metrics: runner, model, IDs, span duration, token totals, turn / tool-call counts, plus auto-instrumented HTTPX child spans per outbound LiteLLM request.
• Trajectory (AGENT_TRAJECTORY_DIR=…) — {run_id}.json per run with per-turn content: turn text, tool call inputs / outputs, per-turn token usage and timing.

Aggregate numbers live on the span; per-turn numbers live in the trajectory. Nothing is repeated.

What lands

Instrumentation (src/observability/, src/agent/*/runner.py)

• init_tracing(service_name) wires a global TracerProvider plus HTTPXClientInstrumentor (auto-propagates traceparent to the LiteLLM proxy). When OTEL_TRACES_FILE is set, attaches the in-process OTLPJsonFileExporter (canonical OTLP-JSON, replayable via the OTel Collector's otlpjsonfile receiver). When OTEL_EXPORTER_OTLP_ENDPOINT is set, attaches the OTLP/HTTP exporter. Either / both / neither — all valid.
• agent_run_span(...) wraps every runner's run() with a root span carrying GenAI semconv attributes (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) plus agent.runner / agent.turns / agent.tool_calls / agent.question.length / agent.answer.length / agent.duration_ms / agent.run_id / agent.scenario_id. plan-execute additionally records agent.plan.steps / agent.planning_time_ms / agent.summarization_time_ms / agent.llm_time_ms.
• set_run_context(run_id=..., scenario_id=...) seeds contextvars so the runner signature stays clean.
• persist_trajectory(...) writes {AGENT_TRAJECTORY_DIR}/{run_id}.json per run with the runner's full Trajectory (SDK runners) or list[StepResult] (plan-execute).
• atexit.register(provider.shutdown) ensures the BatchSpanProcessor flushes the final root span on CLI exit.

Token usage on plan-execute — LLMBackend.generate_with_usage() returns an LLMResult (text + input/output tokens). LiteLLMBackend populates from response.usage; mocks default to zero. A _TokenMeter wrapper inside the plan-execute runner accumulates across planning, per-step arg-resolution, and summarisation calls so plan-execute spans now report gen_ai.usage.* alongside the SDK runners.

Timing metrics

• Span: agent.duration_ms (all), agent.planning_time_ms / agent.summarization_time_ms / agent.llm_time_ms (plan-execute).
• Trajectory: Trajectory.started_at (SDK runners), TurnRecord.duration_ms (claude-agent), StepResult.duration_ms (plan-execute). Per-tool timing intentionally not captured — adding the PreToolUse hook to claude-agent broke compatibility with several @anthropic-ai/claude-code CLI versions; openai-agent / deep-agent SDKs don't expose clean per-tool callback surfaces either.

CLI flags — --run-id (auto-UUID4 if omitted) and --scenario-id on every entry point, seeded into the run context before dispatch.

Docs (docs/observability.md) — full enable → persist → query → replay workflow, span attribute table with per-runner coverage, trajectory layout for both shapes, jq recipes, log rotation guidance, optional Jaeger / Collector replay sections.

Bundled refactor (merged in from the now-closed #273 / #274 / additional commits on this branch) — consolidates what were previously copies across runners:

• src/agent/models.py — canonical ToolCall / TurnRecord / Trajectory.
• src/agent/_litellm.py — LITELLM_PREFIX + resolve_model().
• src/agent/_prompts.py — shared AGENT_SYSTEM_PROMPT.
• src/agent/_cli_common.py — setup_logging / add_common_args / print_result / run_sdk_cli; each SDK CLI's main() is one line.
• src/agent/runner.py — DEFAULT_SERVER_PATHS lives on the base; AgentRunner.__init__ resolves server_paths into a concrete dict once.
• openai_agent — custom _managed_servers replaced with stdlib contextlib.AsyncExitStack.
• deep_agent — _chat_model is now a cached_property so the LangChain client is built once per runner instance.
• LiteLLMBackend._WATSONX_PREFIX constant.
• Six duplicated _resolve_model tests collapsed into one parametrized suite.

Dependencies

New optional group [dependency-groups.otel] keeps the base install lean:

opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp-proto-http
opentelemetry-exporter-otlp-proto-common
opentelemetry-instrumentation-httpx

Enable with uv sync --group otel. Trajectories need no extra deps. Both env vars are optional — runs work normally with zero persistence overhead when neither is set.

Test plan

• uv run pytest src/ -k "not integration" — 270 pass, 50 deselected, 0 failures.
• New tests cover: file-exporter encoding / append semantics / directory creation, trajectory persistence (disabled-without-env no-op, happy path, list shape, missing run_id warn-and-skip, nested dir creation), run_id / scenario_id kwargs and contextvar precedence, nullable timing defaults, plan-execute token accumulation across planner + arg-resolution + summarise calls.
• uv run {plan-execute,claude-agent,openai-agent,deep-agent} --help all render --run-id / --scenario-id.
• End-to-end smoke: OTEL_TRACES_FILE=./traces/traces.jsonl AGENT_TRAJECTORY_DIR=./traces/trajectories uv run deep-agent --run-id bench-001 "..." produces both artifacts; jq recipes from the docs return the expected metadata / per-turn content.
• Optional Jaeger replay path verified end-to-end with jaegertracing/all-in-one.

Not included / out of scope

• OpenTelemetry metrics API — traces carry enough for aggregation (token totals, latency via span duration, counts). If dashboards later need proper Counter / Histogram instruments, add them as a follow-up.
• Per-turn child spans — only the root span is emitted per run. Per-turn detail lives in the trajectory file.
• Raw prompt / response text on spans — deliberately omitted (PII risk, attribute truncation). Trajectories carry the text on disk; if you want it in traces, add it explicitly.
• Per-tool timing across SDK runners — see the PreToolUse compatibility note above; revisit when the SDKs grow stable callback surfaces.
• src/tmp/ cleanup — owner handling separately.