Repository navigation
Replies: 3 comments 4 replies
|
Thanks again, @smota, for the clearly scoped proposal. As you put it, the question is whether an optional, backend-neutral OpenTelemetry integration fits OpenRig's direction, building on the existing finite telemetry contract, so operators can see where work waits, why it retries and what it costs in their own observability stack. Your first outcomes are queue age and time between transitions; runtime durations, retries and approval waits only when a producer explicitly reports them; token usage and cost with their provenance kept distinct; and correlation from a queue item through a seat's tenure to runtime operations, with incomplete evidence left visible. You'd start outside the daemon, with a small optional collector reading the documented API. It's received, and whether and how it fits is still to be decided; nobody here has started on it. Would you plan to build it yourself? |
Proposed architecture and implementation scopeFollowing up on my offer to build this: here is a concrete proposal, grounded in the existing implementation at Reason and operator valueNative coding clients already expose telemetry about model requests, tools and usage. The missing view is the coordination around them:
This integration would let operators build seat-state timelines, compare recorded queue-state durations and inspect coordination spans alongside native client telemetry in their existing observability stack. These are coordination observations, not individual-productivity measurements. Scope and reuseThe first delivery would provide:
The design reuses the finite telemetry contract, existing node inventory, activity oracle, taxonomy and immutable identities. It adds no daemon instrumentation, database migration, activity classifier, topology registry, npm workspace or runtime execution mode. Native clients remain the owners of model/tool/token/cost telemetry. The exporter would not proxy, ingest or reproduce their signals, change their telemetry configuration or alter their launch environment. ArchitectureOne exporter process would observe one configured daemon source. Queue history comes from The inventory response is broader than the finite metadata contract. The exporter would copy only explicitly approved fields into outgoing telemetry. It would not read private SQLite tables. Identity and topology changesSeat metric identity would be Proposed default: charts follow the logical seat across occupant changes, with observed tenure changes recorded as annotations.
A stable, optionally pseudonymous seat identifier is necessary for per-seat charts. Queue IDs and native-session IDs would not be metric dimensions. Mutable names, topology placement and tenure IDs would not be dimensions on every metric; any exported metadata must preserve what was observed at the time. Disappearance is reconciled only after successful, complete reads of the configured scope. Archived, filtered, stopped, deleted and unavailable are different conditions. Departed seats stop receiving activity samples; freshness/staleness handling must prevent the last MetricsNames below are proposed custom OpenRig semantics, not claims of an existing standard OpenRig convention.
The exporter would use the existing activity display derivation, not create a second classifier: Owning an Seat metrics are sampled observations: short transitions between polls can be missed. They support timelines and estimated state proportions, not exact historical seat-state durations. The live activity stream is a change notification requiring rehydration, not a replayable state-history feed. Transition counts describe observations rather than authoritative lifetime totals. Historical records read today must not be represented as transitions occurring today. Replay and ambiguous export acknowledgement can produce overlap. Queue-state spansThe first span family would be
Rules:
Logs and privacyStructured events:
A valid source timestamp maps to event time; collection time is separate. Unknown source time stays unset. An observed tenure change does not imply that every intermediate handover was captured. Exclude prompts, message/queue bodies and summaries, transcripts, tool arguments, commands, paths, environment values and credentials. Actor and native-session identities are omitted by default. Attribute values, metric dimensions, checkpoint state and outgoing batches remain bounded. OpenTelemetry compatibility and connected client tracesUse standard OTLP/HTTP JSON messages for metrics, logs and traces, explicit resource identity and instrumentation scope, typed attributes, correct timestamps/integer encoding, documented metric temporality/reset behavior and protocol-correct rejection/retry handling. Queue intervals would not be mislabeled as Sending OpenRig and native-client telemetry to one Collector does not connect their traces. There are three distinct levels:
The desired future connection is: That is explicitly a follow-up. Current tmux delivery has no verified per-message trace-context channel; one launch-time context cannot represent successive queue items in a persistent seat. The external exporter cannot retrospectively propagate context into work that already happened. Claude documents inbound context for Agent SDK and non-interactive References: OTLP, metric conventions, logs data model, context propagation, Claude tracing. Bounded collection, checkpoints and recoveryCheckpoint state contains source configuration identity, API cursor, bounded open intervals/pending export state and previous seat identity observations. It is not a second historical database. Metrics/logs/traces need separate delivery accounting: acceptance of one signal does not establish acceptance of the others. Partial acceptance is not a whole-batch retry. Exactly-once delivery is not promised. During destination failure, pause ingestion rather than grow an unbounded queue. Lag and resulting retention gaps remain visible, including local diagnostics when export itself is unavailable. Proposed minimum restart policy: record boot/watermark discontinuity, invalidate open intervals and adopt a fresh checkpoint. Retained-history recovery and cross-boot deduplication are outside this delivery. Preserve decimal IDs, finite windows and the API's Implementation and test locations
This follows existing executable-reference examples. Reference assets already ship in the assembled package, but startup materializes Markdown only; document the script's actual packaged/checkout invocation rather than imply it appears in No daemon behavior change is proposed. If an existing projection cannot honestly support a promised signal, return for a scope decision instead of automatically expanding the API. Extend tests for:
Reuse existing activity/identity fixtures where practical rather than modifying their implementations for exporter tests. Add a real OpenTelemetry Collector interoperability exercise: a mock receiver proves requests, not Collector acceptance. Regression checks must fail at the intended assertion with the corresponding defect introduced. Run the repository's normal build/typecheck/test checks. Possible follow-ups, after proving the implementationThese are not part of the first delivery:
Would this scope fit upstream, and would a runnable reference exporter under |
|
Implemented and merged: #2589 External package at https://github.com/smota/openrig-telemetry |
Uh oh!
There was an error while loading. Please reload this page.
The ask
Would an optional, backend-neutral OpenTelemetry integration fit OpenRig's direction? I would like to connect operational evidence across queues, seat tenures and agent runtimes so operators can understand where work waits, why it retries, and what it costs, using their existing observability stack.
This builds on the existing finite telemetry contract, which already exposes bounded event, queue-transition and tenure reads. It complements the usage/learning discussion #1099 and provider-neutral harness plugins #43; it does not assume the proposed plugin protocol is implemented.
Useful first outcomes
These are proposed measurements, not claims that every runtime or today's telemetry API supplies them.
Suggested boundary
Start outside the daemon. A small optional collector could consume the documented finite API, persist its checkpoints, and export standard OTLP to an operator-managed local OpenTelemetry Collector. It should not read private SQLite tables or require a particular dashboard/backend. The external collector owns backend authentication, routing, retention and deployment; OpenRig needs no backend credentials. A local 14-day retention policy, for example, would be an operator choice, not an upstream requirement.
Define a producer contract before expanding signals. Version the allowlisted fields, units, timestamps, provenance and capability/unknown states. Explicitly exclude prompts, message bodies, queue summaries, transcripts, tool arguments, commands, paths, environment values and credentials. Existing operational IDs and actor addresses can identify people: give operators control over exporting or pseudonymizing them. Stable logical seat/node and tenure identities belong in controlled correlation attributes; use low-cardinality metric dimensions such as runtime, operation and outcome, never queue IDs, native session IDs or arbitrary text as metric labels.
Preserve the API's evidence limits. Keep the existing event/transition/tenure streams and decimal-string identifiers. Streams do not establish cross-stream causal ordering. Record boot epochs, cursor failures, coverage gaps and overlap; do not promise exact continuity or global deduplication while
sequenceSpaceIdis null. After a daemon restart, record the gap and explicitly choose retained-history recovery with possible overlap, or a fresh checkpoint that skips the missed interval. Deduplication must be scoped to known source identity and stream, with documented limits across restores/restarts.Keep export off the execution path. Use bounded buffers, bounded batches/timeouts and retry backoff, with visible drops/export failures. Telemetry must fail open: a disconnected collector must not block tools, queue operations or approval decisions. For live runtime instrumentation, define optional W3C
traceparentpropagation across supported process/API boundaries with version/capability compatibility, validation and documented trust rules. Do not infer parentage from timing, and do not make tracing a prerequisite for work.Small phased MVP
Acceptance tests should cover pagination (including empty filtered pages), duplicate overlap, daemon restart/watermark regression, missing history, unknown timestamps, privacy allowlists, bounded cardinality, and an unavailable or slow OTLP receiver without blocking normal work. A synthetic runtime should verify supported/unsupported signals and trace-context compatibility.
Questions for maintainers
All reactions