Scope executor spans to operations and link handed-off work

[kmatasfp]

Summary

• Scope executor OTel spans to the operation they measure, instead of a worker's or background loop's lifetime.
• Relate handed-off work to its originator by an OpenTelemetry link rather than by nesting.
• Set otel.kind on entry-point, producer and consumer spans.
• Make the OTLP layer's filter a plain RUST_LOG-style filter under GOLEM_OTLP_FILTER, with stable target names for the two high-volume sources.
• Record how long each worker admission phase blocks, as worker_admission_wait_seconds.
• Backport main's scheduler storage retry and ack-failure propagation.

Trace 126019611fb73a83e301cd611cb13505 showed get_or_create_worker running for 3m26s inside a 1.76s gRPC call. Spans modelled worker and loop lifetime rather than operations, so they stayed open for as long as the worker was resident, reported durations unrelated to any request, and accumulated an OpenTelemetry event for every tracing event recorded inside them. tracing-opentelemetry holds those events until the span closes, and the SDK applies max_events_per_span only at close, keeping the first N — so a span that never closed retained everything and exported none of it. Executor RSS reached 11 GB, and the resulting OOMs failed worker admission.

Behavior

Every span now covers one bounded operation. Background loops — the scheduler, quota renewal, resource-limit batching, agent status flushing, oplog transfer and forwarding — span each tick rather than the loop, so a span always closes and its events are bounded by one iteration. Worker residency and the invocation loop have no span at all; the phases inside them span themselves. A worker's admission waits are spanned one at a time, and each records its duration to worker_admission_wait_seconds, labelled by phase.

Work that is handed off is related to its originator by a link, never by parent-child nesting. This includes the synchronous invoke-and-await path: the caller enqueues an invocation and waits on a published event rather than a return, and the work continues if the caller goes away, so the caller does not contain it in time. Nesting there reported children outliving their parents. The caller keeps its own span for the time it spends waiting, and the execution is the root of its own trace, linked back to the producer span. otel.kind is server on gRPC and HTTP entry points, and producer/consumer on the enqueue and pickup pair, following the OpenTelemetry messaging conventions. A request abandoned by its caller is marked cancelled rather than failed.

Origins are carried as captured span contexts rather than tracing::Span handles, so remembering where an invocation came from no longer holds a span open until it completes.

GOLEM_OTLP_FILTER sets the OTLP layer's filter in RUST_LOG syntax, and is used exactly as written — nothing is merged into it, so info means info everywhere including third-party crates, and quietening a crate is the deployment's judgement to make. Unset means off: exporting already requires GOLEM__TRACING__OTLP__ENABLED, __HOST and __PORT, and this is the remaining switch. Levels below info export nothing, because the spans bounding an operation are info and an error reaches a trace as an event on a span rather than as a span of its own. GOLEM_OTLP_LOG is still honoured, with a warning naming the replacement. The filter in force, and the variable it came from, are logged at startup.

Two sources whose volume is decided outside the executor have stable target names, so they can be tuned without naming a Rust module path: golem::plugin_log for log output from oplog-processor plugin agents, and golem::agent_rdbms for SQL an agent runs against its own database. Everything else keeps its module's target, so ordinary RUST_LOG conventions apply.

Scheduler storage operations retry transient failures according to scheduler.storage_retry, and a failed acknowledgement aborts the remaining claimed actions rather than being ignored.

Notes

• A trace no longer contains both services for any invocation path. Handed-off work is a linked root by design, which is the representation the OpenTelemetry messaging conventions prescribe for a producer and consumer; the link is what ties the two traces together.
• Flat RSS over a long run and the absence of OOM-induced admission failures are runtime properties and are not asserted in CI.
• Benchmark trace volume is not comparable across this change: the benchmark runner previously exported at debug with a set of directives merged in, and now exports at info.