ATIF trajectory format used by Harbor

[jeremyeder]

The Harbor benchmark captures trajectories in ATIF (Agent Trajectory Interchange Format).

This issue is to gather feedback and track interest in the ATIF format among OTEL users.

We (and others [1]) convert ATIF trajectories to [OpenTelemetry (https://opentelemetry.io/) spans for visualization.

While this works, the conversion is bespoke and incomplete until the conventions exist to cover the full surface area of what is valuable and unique in both test and production GenAI environments.

Several of the LLM-specific fields in ATIF have been requested. The "why" column is mine.

Priority	#	Type	Topic	Dashboard Status	Why
P1 — Must	#287	Issue	Cost conventions	Not tracked	Without cost attributes, every consumer builds bespoke token-to-price tables
P1 — Must	#197	PR	Cache/phase token breakdowns in metrics	Waiting on authors (has conflicts)	Cache tokens are 88% of our input volume; metrics without this dimension are misleading
P1 — Must	#96	PR	Cache & reasoning token metric attributes	Draft	Prerequisite for cache-aware cost dashboards
P1 — Must	#309	Issue	Causal span linking for tool execution	Not tracked	Parallel tool calls are invisible without causal links to the triggering LLM step
P2 — Want	#203	PR	gen_ai.agent.steps metric	Waiting on authors (has conflicts)	Step count variance (7–48×) is the primary driver of cost variance
P2 — Want	#336	PR	Inference/tool calls per invocation	Waiting on reviewers	Needed to correlate agent behavior with cost at the metric level
P2 — Want	#79	Issue	Evaluation experiments & test cases	Not tracked	Experiment metadata (task, wave, rep, reward) currently lives in custom attributes
P2 — Want	#185	PR	Evaluation operation & span	Waiting on authors (CI failing)	Standardizes how eval scores attach to traces
P3 — Nice	#76	Issue	Detailed token usage metrics	Not tracked	Covers reasoning tokens; less urgent than cache for our use case
P3 — Nice	#320	Issue	Agent harness hook conventions	Not tracked	Useful for cross-harness policy correlation but not blocking. Will increase in priority as time passes

[1] https://github.com/Arize-ai/phoenix/tree/main/packages/phoenix-client/src/phoenix/client/helpers/atif

cc @lmolkova @trask - filed after the June 23 call.

[HarperZ9]

One interop test I would want for ATIF -> OpenTelemetry is not only a field mapping, but an explicit loss/coverage report for the conversion.

A small conformance fixture could carry three artifacts side by side:

• source ATIF trajectory: one model step with parallel tool calls, one retry/attempt, one skipped or not-run eval case, selected/suppressed context inputs, and one scorer/verdict;
• expected OTLP-style trace: stable span names plus join fields that link model step -> tool calls -> eval/scorer result;
• conversion coverage report: preserved, represented_by_ref_or_hash, intentionally_not_exported, unsupported, or unverifiable for each ATIF field family.

That would make the hard cases testable instead of prose-level:

• Did a tool result disappear, or was it intentionally represented by a redacted ref/hash?
• Does a skipped/not-run eval case become a real per-case status, or only an aggregate delta?
• Do parallel tool spans still join to the triggering model step without encoding semantics in span names?
• Can context selection/load be represented without claiming later relevance/usefulness?
• Is private prompt/output material absent because of a privacy boundary, or because the converter lost it?

I am working through similar trace/receipt questions in Project Telos and in a narrow Pydantic AI fixture PR around stable span names plus turn-index attributes: pydantic/pydantic-ai#6135

If Harbor publishes a tiny ATIF fixture for one run with parallel tools + retry + eval outcome, I can compare it against the Telos receipt expectations and report concrete gaps rather than general mapping feedback.