Usage semantics: should traces count operations or attempts? (retry double-counting)

[roy-tong]

Describe the feature or potential improvement

Context: I am building AgentMeasure, an open measurement model for how agents consume software capabilities (https://github.com/roy-tong/AgentMeasure). Langfuse is where most production agent traces end up, so this boundary question matters here directly.

The edge case:

An agent retries the same provider call once, then succeeds. Two telemetry records exist. What should usage metrics say?

A. 1 operation, 2 attempts (operation success 100%, attempt success 50%)
B. 2 operations with 50% success

Both telemetry systems can be factually correct. The disagreement is not in the data - it is in measurement semantics: which records get grouped into one countable unit.

Questions for Langfuse:

1. In production traces today, does a runtime-level retry of one logical intent show up as one trace/observation or several?
2. When Langfuse reports cost and success rate, which grain does it aggregate over - calls (attempts) or logical operations?
3. Same question for cache attribution: run A writes the prompt cache, run B reads it. Do cache-read tokens get attributed to B (the consuming run), and is that distinguishable across providers?

My leaning: attempts are execution facts (append-only), operations are views over them, and any inferred grouping must carry an evidence grade so consumers can tell declared boundaries from inferred ones.

Would love to hear how the Langfuse team draws this line internally - happy to contribute a writeup of the model if useful.

Additional information

No response

[jaden3824]

I would keep attempts append-only, then expose operation and task aggregates as separate views. The key safety rule is that grouping may deduplicate a success denominator, but it must never deduplicate actually billed cost.

A minimal event row could be:

task_id, operation_id, attempt_id, retry_of_attempt_id, kind, cache_role, provider_usage, billed_cost, outcome, fallback_of, boundary_source

where provider_usage keeps input, output, cache-read, cache-write, reasoning, and total independently nullable, and boundary_source is runtime-declared | SDK-inferred | trace-inferred.

I would report all three levels:

1. attempt: exact execution fact and provider-reported usage;
2. operation: one logical intent, success if its terminal attempt succeeds, plus attempt count;
3. task: inclusive sum of every operation, retry, repair, and fallback needed for the outcome.

Cache attribution should stay with the attempt whose provider receipt reports the cache write/read. A separate cross-run reuse view can relate producer and consumer, but should not move billed tokens between them. Missing usage stays unknown rather than zero.

A small conformance fixture would be useful: cache write → failed attempt → retry cache read → fallback → success. The expected output should show 4 attempts, the declared operation count, one task outcome, and the unreduced sum of all known costs.

We reached this boundary while building a public agent-communication ledger. Our uncomfortable general unfamiliar-agent result is still 0%, largely because setup, retry, and fallback cannot be omitted from the task total: https://github.com/jaden3824/urusilla/tree/79310ca/competitive_eval

One concrete Langfuse question: is there currently a stable field that correlates automatic SDK retries before export, or must the application declare operation_id and retry_of_attempt_id itself?

Disclosure: I maintain Urusilla; Codex helped prepare and post this comment under owner authorization.

[roy-tong]

Your three-level split (attempt / operation / task) matches what we shipped in AgentMeasure Draft 0.4 almost exactly, and I want to second two of your rules explicitly:

• billed cost never deduplicates - our cost accounting is attempt-scoped for the same reason; grouping is for success denominators only
• boundary_source (runtime-declared | SDK-inferred | trace-inferred) - we call this Measurement Labels; an observation without its label is where most cross-tool disagreements come from

Your proposed conformance fixture (cache write -> failed attempt -> retry cache read -> fallback -> success) is close to what our experiment engine runs as its parsing self-test. One more data point from practice: we just completed the first preregistered controlled A/B on a real agent harness (codex / gpt-5.6-sol, v0.2.1) - a description rewrite moved selection 0/4 to 1/4, every call token-metered at 188K-215K per operation, and the engine verdicted it an honest null at n=4 rather than a win. Attempt-level facts are exactly what made that null trustworthy.

On your question to Langfuse about correlating automatic SDK retries: from the spec side we treat runtime-declared retry chains as first-class, and the honest answer for trace-inferred boundaries today is heuristic clustering - which is why we push providers to declare operation_id themselves.

Disclosure: I maintain AgentMeasure; the live A/B run and its report are in the v0.2.1 release notes.

[roy-tong]

Follow-up offer, since your event row and our open formats are nearly isomorphic: FMT-001/002/003 (lab/schemas) carry the same split - operation_id, attempt_id, retry_of, outcome, usage-with-absent-allowed. If you map your Urusilla event row onto them, any divergences are exactly the spec bugs I want to find.

Two things you might find useful from our side:

1. The live-validated preregistration discipline: your competitive_eval discloses an uncomfortable 0% on unfamiliar agents - that honesty is rare, and it is the same posture our engine enforces mechanically (verdicts like honest-null-at-n, next-round n sizing). The v0.2.1 release has the first real-harness run (codex/gpt-5.6-sol) with the full report.

2. If you ever re-run that eval under a preregistered plan with token metering, the report writes itself - and a cross-check between your ledger's semantics and ours would strengthen both.

No pressure either way; the schemas and the runner are open.

[jaden3824]

Thanks, Roy — I took up the mapping offer and pinned AgentMeasure v0.2.1 at commit 20807ad39483a6b78d2db056d78af6e4c7196bf4.

The full project-authored mapping and exact source identities are here: https://github.com/jaden3824/urusilla/blob/474dd89f23149008af3bb9ba73d686a257221491/outputs/AGENTMEASURE_URUSILLA_MAPPING_2026-08-23.md

The useful positive result is that this eight-event synthetic fixture passes the bundled v0.2.1 validator, and the official aggregator preserves all four attempt costs: 4 attempts/operation and 25 unreduced synthetic cost units.

• events: https://github.com/jaden3824/urusilla/blob/474dd89f23149008af3bb9ba73d686a257221491/outputs/agentmeasure_urusilla_fixture_001.events.jsonl
• expected core metrics: https://github.com/jaden3824/urusilla/blob/474dd89f23149008af3bb9ba73d686a257221491/outputs/agentmeasure_urusilla_fixture_001.expected.json

The current released FMT-002 projection is intentionally lossy: task_id and outcome map directly, stable operation/attempt identities collapse to indexes, and I could not find released core fields for retry_of, fallback/cache edges, provider-usage components, billed-cost units, boundary provenance, or a task terminal.

Two small conformance questions also reproduced: the bundled validator returns after oneOf and can accept records missing root sibling constraints; separately, the aggregator trusts declared operation attempt count/outcome without reconciling the attempt rows. I also could not locate the live v0.2.1 preregistration/events/run/report bundle in the tagged tree or release assets, so I treat its A/B numbers as maintainer-reported rather than independently recomputed.

No live model run, external reproduction, Langfuse integration/adoption, or paid call occurred in this mapping. Does your comment describe an upcoming FMT revision or intended extension vocabulary? If these boundaries match your intent, I would be glad to turn the fixture and the three mutations into a small attributed upstream conformance contribution.

[roy-tong]

Jaden — thank you. This is the first external conformance pass against AgentMeasure, and it's genuinely useful. Taking your points in three groups:

Confirmed bugs (will be filed as issues):

• Validator early-return after oneOf accepting records that miss root sibling constraints — reproduced from your fixture description; fix lands before the next tag.
• Aggregator trusting declared operation attempt count/outcome without reconciling against attempt rows — agreed this is a correctness bug, not a limitation. A reconciliation check is going in.

Known limitations (intentional, documented as such):
The released FMT-002 projection is deliberately lossy today: task_id and outcome map, but stable operation/attempt identities collapse to indexes, and retry_of, fallback/cache edges, provider-usage components, billed-cost units, boundary provenance, and a task terminal are not in the released core fields.

To answer your question directly: yes, the earlier follow-up describes the upcoming FMT-0.5 revision, and the fields you listed are exactly its scope — stable operation/attempt identities, retry_of, fallback/cache edges, usage components with absent-allowed, billed-cost units, boundary provenance, and a task terminal. Your mapping document is the cleanest external statement of that delta I've seen; I'll cite it in the revision notes.

On the preregistration/events/run/report bundle: fair call. The tagged v0.2.1 tree doesn't carry it, so "maintainer-reported" is the correct reading today. The next release will attach the full bundle as release assets so the A/B numbers become independently recomputable.

Contribution — yes, please. The fixture plus the three mutations as an attributed upstream conformance contribution is exactly what the repo needs: synthetic, machine-checkable, and pinned to a schema version. Open the PR against conformance/ and I'll review.

Last, your Langfuse-side question from earlier in this thread (stable field correlating automatic SDK retries before export): nothing stable exists today — which is precisely why boundary_source declares who grouped, and why FMT-0.5 makes operation identity explicit rather than inherited from span topology.

[roy-tong]

Accepted — and thank you: this is exactly what an external conformance pass is for.

Both findings are fixed, before the next tag as committed, in v0.2.2 (issues #8 / #9 closed with reproductions):

• The validator now treats oneOf/anyOf as compositional keywords, not terminators — root sibling constraints always run after a branch matches, and zero-match errors carry the first branch error. Your eight events still validate; every root-required-field-stripped mutation is now rejected. That is a permanent guard (conformance/runners/run_external_fixture.py, guard 2), not a one-off test.
• The aggregator no longer trusts declared summaries. Each operation_result is reconciled against its attempt rows — declared count vs rows, declared outcome vs the rows' derived outcome (rule op-success-any/1, last-attempt outcome disclosed). Mismatches surface as an explicit operation_reconciliation: failed block, and counts/outcomes always use measured rows. Your expected metrics (4 attempts/operation, 25 unreduced units) reproduce exactly; tampered declarations (attempts=9, wrong outcome, zero rows) fail closed (guards 3–4).

Your fixture is now the first external conformance vector — conformance/vectors/external/urusilla-001/, events + expected + your mapping doc verbatim, with the runner guarding schema / #8 mutations / metric reproduction / #9 tamper in four classes. Your claim boundary is preserved as written: project-authored synthetic evidence — no endorsement, adoption, reproduction, or live run implied.

Two honest concessions on your points:

1. You are right about the v0.2.1 bundle: the preregistration/events/run/report artifacts were workspace-local, not in the tagged tree or release assets — maintainer-reported is the correct reading. v0.2.2 starts the practice of shipping reproducible bundles with the release (its notes include a "Conformance bundle" section pinning every artifact needed to re-run your pass); the lab A/B bundle ships the same way with the next lab release.
2. On your question — no FMT revision is scheduled yet, deliberately: the revision should follow real fixtures, not priors. Your x_urusilla sidecar is the sanctioned interim for exactly this, and the lossy areas you listed (retry_of/fallback edges, cache roles, usage components, boundary provenance, task terminals) are queued as the revision's input list.

One ask, since you offered mutations as well: we built the mutation guards from vector #1 ourselves, so more mutations of the same shape are worth less to us than a second vector in a different shape — your mapping's lossiest areas (retry/fallback edges as first-class, cache-role attribution, task terminals) are where a second vector would bite hardest. Same three-file format, guarded the same way, attributed the same way. No pressure either way — the offer stands either way.

[jaden3824]

@roy-tong Thank you for carrying both findings through to v0.2.2 and for preserving the evidence boundary in the upstream vector. I have verified the release notes and conformance/vectors/external/urusilla-001; the validator and reconciliation fixes are a concrete, attributable interoperability outcome.

The proposed second vector is the right next test because it exercises a different loss boundary rather than repeating the first mutations. If I prepare it, I will keep it pinned and synthetic and focus on first-class retry/fallback edges, cache-role attribution, a task terminal, and an unknown-schema fail-closed case. I am treating that as a candidate next contribution, not as completed work or an adoption/reproduction claim.

Thank you again for turning an external pass into durable guards.

Disclosure: I maintain Urusilla; Codex helped prepare and post this reply under owner authorization.

[jaden3824]

@roy-tong I completed the second vector in the different shape you requested.

Pinned report: https://github.com/jaden3824/urusilla/blob/cfb4ef352980501fe28eeec6032c84cea4fabf89/outputs/AGENTMEASURE_URUSILLA_VECTOR_002_2026-08-28.md
Events: https://github.com/jaden3824/urusilla/blob/cfb4ef352980501fe28eeec6032c84cea4fabf89/outputs/agentmeasure_urusilla_fixture_002.events.jsonl
Expected result: https://github.com/jaden3824/urusilla/blob/cfb4ef352980501fe28eeec6032c84cea4fabf89/outputs/agentmeasure_urusilla_fixture_002.expected.json
Validator: https://github.com/jaden3824/urusilla/blob/cfb4ef352980501fe28eeec6032c84cea4fabf89/tools/validate_agentmeasure_urusilla_vector_002.py

It has two operations, four attempts, and ten FMT-002 rows: an explicit failed cache-write attempt → cache-read retry, then a local required-schema rejection → explicit JSON fallback. The task terminal remains incomplete and non-effect-authorizing because no scorer receipt proves task success. AgentMeasure v0.2.2 validates all rows, reconciles both operation declarations, and preserves all 13 synthetic cost units.

It also gives a smaller AM-U-007 candidate: per-operation attempt-step totals are [3, 2], so their semantic median is 2.5, while v0.2.2 reports 5.0 with denominator 1 after assignment-level aggregation. Would you treat one-operation-per-assignment as a required invariant, or should this metric grain/label change?

Claim boundary: project-authored synthetic offline evidence only; no AgentMeasure endorsement, upstream acceptance, external reproduction, Urusilla adoption, live model run, or real provider-cost observation is implied.

Disclosure: I maintain Urusilla; Codex assisted with the fixture, validation, and this owner-authorized reply.

[vgmartinez]

Useful boundary question — we hit the same split when a gateway sits in front of providers.

In that topology, retries (and cache hits) often happen at the gateway, not in the app SDK. Downstream traces then see either:

• one app span wrapping N gateway→provider attempts, or
• N sibling spans if each attempt is exported separately.

Our leaning matches yours: attempts are facts; operations are a view. Practically we keep attempt-level token/cost on each provider call, and only roll up to an “operation” when something explicit declares the boundary (shared parent / correlation id / gateway request id). Otherwise cost and success rates double-count under load with retries.

Cache attribution is the same problem: a gateway semantic cache can serve run B without a provider call — if only B’s consuming run is scored, you need a distinct signal (e.g. cache-hit attribute) so Langfuse doesn’t look like “zero tokens, success” with no explanation.

Happy to share how we wire OTLP export from a governance gateway if useful. Building TrustGate at NeuralTrust (CTO) — complementary to Langfuse (gateway for policy/routing; Langfuse for traces/evals): https://github.com/NeuralTrust/TrustGate

[roy-tong]

Victor — I'd like to take you up on the OTLP offer, with one concrete shape rather than a general description.

Could you share a minimal real topology from TrustGate as you run it today — five records: the app span, the gateway request, provider attempt 1, provider attempt 2 (retry), and a cache hit? For each, what actually exists today: span name/type, parent (or span link) relationships, and any correlation ID that ties them together.

The reason this specific shape matters: it's the minimal case where "attempts are facts, operations are a view" has to survive without an app-declared operation_id. If the gateway request ID is the only stable correlator, that's a trace-inferred operation boundary — and I'd rather encode that reality than assume SDK-declared grouping exists.

Strong +1 on the cache-hit signal needing to be distinct: "served from gateway cache, zero provider call" and "provider called, zero tokens" must not collapse to the same record. That's exactly why AgentMeasure keeps cache_role on the attempt whose receipt reports it, with a separate cross-run reuse view relating producer and consumer without moving billed tokens between them.

[roy-tong]

Victor — this topology writeup is genuinely useful, and it lands close enough to our current semantics that the deltas are the interesting part:

• "Attempts are facts; operations are a view" is exactly the Draft 0.4 boundary. Operation grouping requires boundary evidence; when it is absent, attempts stay ungrouped and the report discloses the coverage share rather than guessing a grouping — for precisely the double-counting-under-retries reason you name.
• Roll-up only on explicit declaration: our derivation is fail-closed the same way — structural guesses do not silently become operations.
• Gateway cache serving run B: our attribution rule is that the reading run gets the cache-read attribution, never the creating run. Your "zero tokens, success, no explanation" case is what that rule exists to prevent — though in a gateway topology the rule needs the cache-hit signal to actually cross the export boundary, which is a real gap worth naming.

The questions your topology raises that we have not answered:

1. One app span wrapping N gateway→provider attempts vs N sibling spans — which does TrustGate export today? The answer changes whether operation boundary evidence is present at the app layer or only reconstructable at the gateway.
2. Does a shared gateway request id count as correlation-grade boundary evidence, or should it (it is minted by the party that owns the boundary) count as declaration-grade? We currently treat gateway-minted ids as weaker than an explicit terminal declaration — pushback welcome.
3. Policy-gateway denials that trigger retries: are those operation-visible from the app side at all, or a gateway-internal attempt class? Our three-tier model (operation/attempt/charge) would put them as attempts with a distinct failure reason, but we have no real data to check that against.

Hence the ask: if you can share one real or anonymized OTLP trace set containing a retry and a cache hit (any format you already export — we will write the one-off adapter, it does not need to be clean), we will run it through the canonical pipeline and return a coverage-first report: where grouping evidence actually stands, what is safely reportable, and the share that cannot be safely reported. That "cannot be safely reported" share is our product thesis, and gateway topologies are where we most expect it to be non-trivial.

Either way, the writeup above already sharpened questions 1–3 for us — thank you.

[roy-tong]

@jaden3824 Completed the second boundary vector on our side, and AM-U-007 is confirmed.

Your validator passes end to end against the v0.2.2 tag on a clean checkout: 10/10 events schema-valid, both operation declarations reconciled, 4 attempts and 13 cost units preserved, unknown schema rejected fail-closed.

On your question: one-operation-per-assignment should not become a required invariant. Assignments legitimately contain multiple operations, and the metric name already fixes the grain. The defect is grain mixing in analysis.py: it sums steps per assignment and takes the median across assignments, so one assignment with per-operation totals [3, 2] reports 5.0 with denominator 1. The correct computation sums steps per resolved operation (unresolved attempts contribute nothing, fail-closed), takes the median across those operations, and reports the denominator as the number of contributing operations: 2.5 over 2 in your vector.

We will handle it the same way as #8 and #9: an issue first, then a fix with a regression vector, credited to this fixture. And we are ingesting urusilla-002 as the second external conformance vector, with your claim boundary stated verbatim.

Thank you. The two-operation shape is exactly the boundary vector 001 could not stress, and finding a third defect candidate in it is the conformance loop working as designed.