Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract

[RhysSullivan]

Why

Expected tool failures (ToolResult.fail) resolve through the Effect success channel by design, which meant the tracer recorded healthy spans while callers were hitting upstream error walls. Auditing a week of traces showed three concrete gaps:

• plugin.openapi.invoke spans never carried http.status_code — the annotation ran inside the inner OpenApi.invoke span (created by Effect.fn), not the wrapper span queries target. 0 of ~19.5k spans had it.
• The majority of executor.tool.execute error spans had an empty status.message: most SDK tagged errors define no message field, and TaggedErrorClass instances default to "".
• Some error spans rendered [object Object] as their only label (non-Error causes stringified with String(...)).

The absence of error data is indistinguishable from health, so none of this was noticed until incidents were debugged by hand.

What

• Outcome attributes on tool spans. New annotateToolResultOutcome (exported from the SDK) stamps executor.tool.outcome (ok/fail), executor.tool.error_code, and executor.tool.error_status onto executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult. Codes and statuses are enumerable identifiers, never user content.
• Tenant/subject attribution on executor.tool.execute, so error queries no longer need a trace-id join against the outer request span.
• http.status_code on the plugin.openapi.invoke wrapper span (in addition to the inner span).
• Derived message getters for the message-less SDK tagged errors (ToolNotFoundError, ToolBlockedError, PluginNotLoadedError, NoHandlerError, ConnectionNotFoundError, ElicitationDeclinedError, integration lifecycle errors), and formatInvocationCauseMessage now prefers the cause's _tag / structural stringify over [object Object].
• An end-to-end telemetry contract scenario (e2e/cloud/telemetry-contract.test.ts): drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real local upstream returning 502 → span batch → OTLP export — and asserts the spans as the suite's trace store actually received them (failure outcome attributes + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, success distinguishable from failure). The prod bug class this guards against (an attribute stamped on a span the exporter never carries, a failure riding the success channel) only manifests at the exported layer, so that's where the contract is pinned. Adds a Telemetry e2e service that queries the suite's OTLP store with arrival polling.

Verification

• The e2e scenario passes against the full cloud dev stack (real workerd topology, suite-owned OTLP store) and fails if any stamped attribute, span name, or outcome distinction regresses at the exported layer.
• packages/core/sdk, packages/core/execution, packages/plugins/openapi suites green; lint, format:check, typecheck clean.

Stack

1. Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract #992 👈 current

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	executor-marketing	e9dda28	Commit Preview URL Branch Preview URL	Jun 15 2026, 02:34 AM

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	executor-cloud	e9dda28	Jun 15 2026, 02:34 AM

[github-actions]

Cloudflare preview

Torn down — the PR is closed.

[pkg-pr-new]

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@992

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@992

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@992

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@992

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@992

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@992

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@992

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@992

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@992

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@992

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@992

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@992

executor

npm i https://pkg.pr.new/executor@992

commit: e9dda28

[greptile-apps]

Greptile Summary

This PR makes expected tool failures (ToolResult.fail) visible in telemetry by annotating outcome attributes onto spans, fixing three production blind spots audited on 2026-06-12: missing http.status_code on plugin.openapi.invoke wrapper spans, empty status.message on tagged SDK errors, and [object Object] labels from non-Error causes. An end-to-end telemetry contract test pins these fixes at the OTLP-export layer where the silent-failure regression class actually manifests.

• Span outcome attributes (executor.tool.outcome, executor.tool.error_code, executor.tool.error_status) are stamped on both executor.tool.execute and mcp.tool.dispatch via a new annotateToolResultOutcome helper; executor.tenant/executor.subject are added to executor.tool.execute so error attribution no longer requires a trace-join.
• http.status_code is now annotated on the plugin.openapi.invoke wrapper span (in addition to the inner OpenApi.invoke span) by placing Effect.tap inside Effect.withSpan.
• Derived message getters are added to all message-less Schema.TaggedErrorClass errors, and formatInvocationCauseMessage now prefers _tag over String(plainObject) to eliminate [object Object] labels in telemetry.

Confidence Score: 4/5

Safe to merge; all three production gaps are correctly addressed and the e2e contract pins them at the export layer.

The span annotation ordering, Effect tap/withSpan nesting, and skip mechanism are all correct. The one issue is in the new telemetry surface: Effect.promise rather than Effect.tryPromise means a transient network error while polling the motel store becomes an unretryable defect rather than a retried failure — harmless for a local CI motel but worth hardening before the pattern is copied elsewhere.

e2e/src/surfaces/telemetry.ts — the Effect.promise calls for fetch and response.json() should be Effect.tryPromise to keep network errors inside the retry loop.

Important Files Changed

Filename	Overview
packages/core/sdk/src/tool-result.ts	Adds annotateToolResultOutcome — stamps outcome/error_code/error_status on the current span; correctly handles ok, fail, and non-ToolResult values; annotation always succeeds (no-ops without a span context).
packages/core/sdk/src/errors.ts	Adds message getter overrides to all message-less tagged errors; getters are prototype-level so serialization/encoding is unaffected; coverage is complete across all tagged error classes in the file.
packages/core/sdk/src/executor.ts	Places annotateToolResultOutcome via Effect.tap inside the executor.tool.execute span (correct ordering); adds executor.tenant/executor.subject span attributes; improves formatInvocationCauseMessage to prefer _tag over [object Object].
packages/plugins/openapi/src/sdk/invoke.ts	Stamps http.status_code on the plugin.openapi.invoke wrapper span via Effect.tap before withSpan, so the annotation runs inside the correct span context.
e2e/cloud/telemetry-contract.test.ts	New e2e telemetry contract test; drives the full production topology to an OTLP store; uses unique random slugs for isolation; upstream HTTP server is properly scoped with acquireRelease; assertions cover both outcome classes and tenant attribution.
e2e/src/surfaces/telemetry.ts	New telemetry surface for querying the motel OTLP store; expectSpan polls correctly via Schedule.both; uses Effect.promise instead of Effect.tryPromise, which means network errors become defects that bypass the retry loop.
packages/core/execution/src/tool-invoker.ts	Correctly calls annotateToolResultOutcome before the isToolResult guard so both ToolResult and raw-value outcomes are stamped on the mcp.tool.dispatch span.
e2e/src/scenario.ts	Wires Telemetry into the test context when E2E_MOTEL_URL is set; the existing missingServices skip mechanism correctly converts an absent Telemetry service into a vitest skip rather than a failure.
e2e/src/services.ts	Adds Telemetry Context.Service with key e2e/telemetry; pattern matches existing optional services; jsdoc clearly states the availability condition.
packages/core/sdk/src/elicitation.ts	Adds message getter to ElicitationDeclinedError with clear cancel/decline distinction; consistent with the pattern applied across errors.ts.
packages/core/sdk/src/index.ts	Exports annotateToolResultOutcome from the SDK; grouped correctly with the existing tool-result exports.
e2e/setup/cloud.globalsetup.ts	Guards E2E_MOTEL_URL behind if (motel) so unset env propagation never overwrites a missing motel with an empty string.

Sequence Diagram

sequenceDiagram
    participant C as Caller
    participant E as executor.tool.execute span
    participant D as mcp.tool.dispatch span
    participant O as plugin.openapi.invoke span
    participant U as Upstream HTTP

    C->>E: execute(address, args)
    activate E
    Note over E: annotates tenant/subject at span open
    E->>D: dispatch to sandbox
    activate D
    D->>O: OpenAPI invoke
    activate O
    O->>U: HTTP GET /fail
    U-->>O: 502 Bad Gateway
    Note over O: annotates http.status_code=502
    O-->>D: ToolResult.fail
    deactivate O
    Note over D: annotateToolResultOutcome()
    D-->>E: ToolResult.fail (success channel)
    deactivate D
    Note over E: Effect.tap(annotateToolResultOutcome)
    E-->>C: ToolResult.fail
    deactivate E
    Note over E,O: All three spans carry failure attributes

Loading

Comments Outside Diff (1)

1. e2e/src/surfaces/telemetry.ts, line 476-484 (link)

   Effect.promise bypasses retry on network errors

   Effect.promise converts rejected promises into Effect defects (the Die channel), which Effect.retry in expectSpan does not catch — only expected failures are retried. If the motel HTTP endpoint is momentarily unreachable (e.g., a transient ECONNREFUSED or a slow response.json() parse), the defect propagates directly through the retry schedule and kills the poll loop immediately. Effect.tryPromise would convert the rejection into a retryable failure instead, giving the full ~20 s window to recover.

_{Reviews (1): Last reviewed commit: "Document the telemetry surfaces for futu..." | Re-trigger Greptile}