Problem Statement
Two observability streams need to travel from the sandbox supervisor to the gateway: supervisor-emitted infrastructure spans (#2508) and agent-emitted traces relayed through the supervisor (#2641). Both streams share the same transport path (supervisor -> session protocol -> gateway -> external collector) and face the same production challenges. These include backpressure when the gateway or external collector is slow, event dropping under sustained load, and the question of whether telemetry should share the control channel or use a dedicated one.
The session protocol was designed for control plane traffic. Using it for telemetry changes the traffic profile substantially. At 100 concurrent sandboxes with active agents and supervisor instrumentation, the aggregate telemetry bandwidth could reach 1-10 MB/second sustained. Without explicit design for backpressure, buffering, and drop semantics, telemetry load could interfere with sandbox lifecycle operations on the same channel.
Persona Workflows
Platform operator: sandbox lifecycle stalls during telemetry spike
An operator deploys a batch of 50 agent sandboxes for a parallel evaluation run. Each agent is OTel-instrumented and generates 100 spans/second. The supervisor relays these over the session protocol alongside control messages. After a few minutes, the operator notices that sandbox create and sandbox exec commands are timing out intermittently. The telemetry volume on the session protocol is causing head-of-line blocking, and control messages are queued behind span batches.
How this issue enables the workflow: With a dedicated telemetry channel (or priority framing on the shared channel), control messages flow independently of telemetry volume. The operator's sandbox lifecycle operations complete at normal latency regardless of how many spans the agents generate. When the external collector slows down, the supervisor drops telemetry (with counters) rather than stalling control operations.
Security auditor: verifying no OCSF events were silently dropped
A security auditor reviews the quarterly compliance report and needs assurance that all OCSF deny events from sandboxes reached the centralized log aggregator. Without explicit drop accounting, the auditor cannot distinguish "no deny events happened" from "deny events happened but were lost in transit."
How this issue enables the workflow: The relay infrastructure maintains per-sandbox drop counters for both traces and OCSF events. The auditor checks the gateway's metrics endpoint and sees ocsf_events_dropped_total{sandbox="sb-abc123"} = 0 for all sandboxes in the reporting period, confirming complete event delivery. When drops do occur (rate limits, buffer overflow), the counter quantifies the gap.
Proposed Design
Transport channel design
Evaluate whether telemetry (traces + Open Cybersecurity Schema Framework (OCSF) logs) should flow through the existing session protocol or use a dedicated channel:
Option A: Shared channel with priority framing - Telemetry uses the existing session protocol but with lower priority than control messages. The gateway processes control messages first and telemetry in available capacity. Simpler to implement, but telemetry backpressure could still affect control message latency under load.
Option B: Separate telemetry channel - A dedicated gRPC stream (or multiplexed sub-channel) for telemetry alongside the control channel. Complete isolation between control plane and telemetry. More complex, but guarantees control message latency regardless of telemetry volume.
The spike should test both approaches under load to inform the choice.
Backpressure and drop semantics
When the gateway is unreachable or the external collector is slow, the supervisor must make explicit decisions about buffered data:
- Bounded buffer: Fixed-size in-memory buffer per stream (traces, logs). When the buffer fills, the oldest entries are dropped.
- Drop counting: Every dropped span or log event increments a counter metric. Silent drops are not acceptable for audit compliance.
- Backpressure signaling: The gateway signals the supervisor when it cannot accept telemetry (collector slow, buffer full). The supervisor reduces its send rate or switches to sampling mode.
- Graceful degradation: Under sustained backpressure, the supervisor applies head sampling (accept 1-in-N spans) rather than dropping entire traces. Partial traces are better than no traces, but mid-trace drops corrupt the parent-child structure.
Dropped spans are worse than dropped logs because they corrupt trace structure. The relay needs documented, tested drop behavior for each failure mode (gateway unreachable, collector slow, buffer full, supervisor overloaded).
OCSF log handling
OCSF events serve compliance and audit. Dropping them silently is not acceptable. Rate limiting (cap events/second per sandbox) is preferred over probabilistic sampling. When rate limits kick in, the supervisor should log a local warning and increment a drop counter.
Scale estimates
| Pillar |
Per-sandbox rate |
Per-record size |
Per-sandbox bandwidth |
| Agent traces |
10-100 spans/sec |
200-500 bytes |
2-50 KB/sec |
| Supervisor traces |
Variable |
200-500 bytes |
1-10 KB/sec |
| OCSF logs |
10-300 events/sec |
1-5 KB |
10 KB - 1.5 MB/sec |
| Supervisor tracing events |
Variable |
100-500 bytes |
1-10 KB/sec |
At 100 concurrent sandboxes, aggregate bandwidth: 1-10 MB/second sustained. OCSF logs may dominate the stream.
Flush-on-shutdown
Both supervisor spans and agent traces need flush-on-shutdown semantics. When a sandbox tears down:
- The supervisor drains any buffered agent spans (the supervisor outlives the agent process)
- The supervisor flushes its own final spans (sandbox teardown events)
- The supervisor signals the gateway that the session is closing and waits for acknowledgment or timeout
This matters especially for short-lived sandboxes (CI tasks, one-shot scripts) where the entire lifecycle may be 10-30 seconds.
Mitigations for scale
- Separate transport channel for telemetry (no backpressure on control plane)
- Head sampling at the supervisor (configurable per-sandbox or globally)
- Per-sandbox rate limits with counter metrics for drops
- Async batch forwarding at the gateway
- Backpressure signaling from gateway to supervisor
- Bypass option: set
OTEL_EXPORTER_OTLP_ENDPOINT directly for users who want to skip the relay
Alternatives Considered
No shared infrastructure, each stream solves its own transport: Rejected. Both streams face identical production challenges, and solving them independently leads to duplicated buffer management, duplicated backpressure logic, and inconsistent drop behavior between traces and logs.
Use the existing PushSandboxLogs RPC for everything: The log push RPC exists and works for tracing events. Extending it to carry OpenTelemetry Protocol (OTLP) spans would require protocol changes (spans are not log lines) and would conflate two different data formats in one stream. Better to have a clean transport abstraction that handles both.
External collector sidecar instead of supervisor relay: This would add a separate process (OpenTelemetry (OTel) Collector) to every sandbox. The supervisor already sits in the right position and adding a collector sidecar increases resource usage, complexity, and the attack surface of the sandbox.
Agent Investigation
PushSandboxLogs client-streaming RPC exists in crates/openshell-supervisor-process/src/log_push.rs- The session protocol is established during sandbox creation and maintained for the sandbox lifetime
- The proxy already manages bounded buffers for intercepted traffic; the same patterns (bounded channels, drop counters) apply
- #2508 sub-issue 1 ("Span transport: supervisor-to-gateway channel") describes the transport need from the supervisor span perspective
Related: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission)
Problem Statement
Two observability streams need to travel from the sandbox supervisor to the gateway: supervisor-emitted infrastructure spans (#2508) and agent-emitted traces relayed through the supervisor (#2641). Both streams share the same transport path (supervisor -> session protocol -> gateway -> external collector) and face the same production challenges. These include backpressure when the gateway or external collector is slow, event dropping under sustained load, and the question of whether telemetry should share the control channel or use a dedicated one.
The session protocol was designed for control plane traffic. Using it for telemetry changes the traffic profile substantially. At 100 concurrent sandboxes with active agents and supervisor instrumentation, the aggregate telemetry bandwidth could reach 1-10 MB/second sustained. Without explicit design for backpressure, buffering, and drop semantics, telemetry load could interfere with sandbox lifecycle operations on the same channel.
Persona Workflows
Platform operator: sandbox lifecycle stalls during telemetry spike
An operator deploys a batch of 50 agent sandboxes for a parallel evaluation run. Each agent is OTel-instrumented and generates 100 spans/second. The supervisor relays these over the session protocol alongside control messages. After a few minutes, the operator notices that
sandbox createandsandbox execcommands are timing out intermittently. The telemetry volume on the session protocol is causing head-of-line blocking, and control messages are queued behind span batches.How this issue enables the workflow: With a dedicated telemetry channel (or priority framing on the shared channel), control messages flow independently of telemetry volume. The operator's sandbox lifecycle operations complete at normal latency regardless of how many spans the agents generate. When the external collector slows down, the supervisor drops telemetry (with counters) rather than stalling control operations.
Security auditor: verifying no OCSF events were silently dropped
A security auditor reviews the quarterly compliance report and needs assurance that all OCSF deny events from sandboxes reached the centralized log aggregator. Without explicit drop accounting, the auditor cannot distinguish "no deny events happened" from "deny events happened but were lost in transit."
How this issue enables the workflow: The relay infrastructure maintains per-sandbox drop counters for both traces and OCSF events. The auditor checks the gateway's metrics endpoint and sees
ocsf_events_dropped_total{sandbox="sb-abc123"} = 0for all sandboxes in the reporting period, confirming complete event delivery. When drops do occur (rate limits, buffer overflow), the counter quantifies the gap.Proposed Design
Transport channel design
Evaluate whether telemetry (traces + Open Cybersecurity Schema Framework (OCSF) logs) should flow through the existing session protocol or use a dedicated channel:
Option A: Shared channel with priority framing - Telemetry uses the existing session protocol but with lower priority than control messages. The gateway processes control messages first and telemetry in available capacity. Simpler to implement, but telemetry backpressure could still affect control message latency under load.
Option B: Separate telemetry channel - A dedicated gRPC stream (or multiplexed sub-channel) for telemetry alongside the control channel. Complete isolation between control plane and telemetry. More complex, but guarantees control message latency regardless of telemetry volume.
The spike should test both approaches under load to inform the choice.
Backpressure and drop semantics
When the gateway is unreachable or the external collector is slow, the supervisor must make explicit decisions about buffered data:
Dropped spans are worse than dropped logs because they corrupt trace structure. The relay needs documented, tested drop behavior for each failure mode (gateway unreachable, collector slow, buffer full, supervisor overloaded).
OCSF log handling
OCSF events serve compliance and audit. Dropping them silently is not acceptable. Rate limiting (cap events/second per sandbox) is preferred over probabilistic sampling. When rate limits kick in, the supervisor should log a local warning and increment a drop counter.
Scale estimates
At 100 concurrent sandboxes, aggregate bandwidth: 1-10 MB/second sustained. OCSF logs may dominate the stream.
Flush-on-shutdown
Both supervisor spans and agent traces need flush-on-shutdown semantics. When a sandbox tears down:
This matters especially for short-lived sandboxes (CI tasks, one-shot scripts) where the entire lifecycle may be 10-30 seconds.
Mitigations for scale
OTEL_EXPORTER_OTLP_ENDPOINTdirectly for users who want to skip the relayAlternatives Considered
No shared infrastructure, each stream solves its own transport: Rejected. Both streams face identical production challenges, and solving them independently leads to duplicated buffer management, duplicated backpressure logic, and inconsistent drop behavior between traces and logs.
Use the existing
PushSandboxLogsRPC for everything: The log push RPC exists and works for tracing events. Extending it to carry OpenTelemetry Protocol (OTLP) spans would require protocol changes (spans are not log lines) and would conflate two different data formats in one stream. Better to have a clean transport abstraction that handles both.External collector sidecar instead of supervisor relay: This would add a separate process (OpenTelemetry (OTel) Collector) to every sandbox. The supervisor already sits in the right position and adding a collector sidecar increases resource usage, complexity, and the attack surface of the sandbox.
Agent Investigation
PushSandboxLogsclient-streaming RPC exists incrates/openshell-supervisor-process/src/log_push.rsRelated: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission)