perf: trace and reduce remote first-turn latency

[apanasenko-oai]

Summary

This change makes remote first-turn and command latency attributable end to end, and removes several avoidable waits found while profiling that path.

• propagate W3C trace context across Core, exec-server RPC, and the encrypted Noise relay
• add stage-level spans for tool dispatch, RPC queues, relay framing, local process lifecycle, terminal delivery, and thread metadata work
• consume pushed terminal events directly instead of polling when the transport supports them
• keep thread metadata persistence off the synchronous event-delivery path while preserving explicit flush barriers
• make turn-diff repository discovery lazy
• reuse the AGENTS project root once during adjacent startup skill discovery, with a pipelined fallback for later independent loads
• preserve client/server timing correlation for Responses WebSocket requests

Why

The previous telemetry exposed large envelopes such as exec_command and session_init, but could not distinguish queue residence, transport transit, child-process work, terminal notification delivery, or persistence. Profiling also showed that skill discovery repeated the same remote ancestor-marker walk that AGENTS discovery had just completed.

The one-shot root handoff removes that duplicate startup walk without caching filesystem topology across turns. Later loads rediscover the nearest marker, so creating or removing a closer repository boundary remains observable.

Validation

• cargo fmt --all -- --check
• cargo check -p codex-core
• cargo test -p codex-core-skills repo_skill_roots_use_one_shot_hint_then_observe_new_nearest_marker -- --nocapture
• cargo test -p codex-exec-server process_lifecycle_trace_separates_output_readers_and_exit_watcher -- --nocapture
• cargo test -p codex-exec-server --test exec_process
• cargo test -p codex-thread-store
• just bazel-lock-check

The new trace boundaries were also exercised with a real remote command flow. Local process handling remained small; the added spans separated it from the dominant transport legs and exposed the repeated startup metadata RPCs.

This is an internally requested performance and observability enhancement; there is no public issue.

[apanasenko-oai]

Superseded by focused draft PRs:

• telemetry(model): correlate WebSocket stage timing #30667 model WebSocket timing
• perf(rollout): reduce and trace writer flushes #30668 rollout writer flushes
• perf(thread-store): project append metadata asynchronously #30669 asynchronous thread metadata projection
• perf(core): avoid duplicate first-turn filesystem discovery #30670 first-turn filesystem discovery
• telemetry(core): trace unified exec process lifecycle #30671 unified-exec process lifecycle
• telemetry(core): trace session startup stages #30672 Core session startup stages
• telemetry(core): trace tool dispatch stages #30673 Core tool dispatch stages
• telemetry(core): trace terminal event delivery #30674 terminal event delivery
• telemetry(exec-server): trace RPC transport #30675 exec-server RPC transport foundation
• telemetry(exec-server): trace Noise virtual streams #30676 Noise virtual streams, stacked on telemetry(exec-server): trace RPC transport #30675
• telemetry(exec-server): trace Noise harness relay #30677 Noise harness relay, stacked on telemetry(exec-server): trace Noise virtual streams #30676
• telemetry(exec-server): trace remote client lifecycle #30678 remote client lifecycle, stacked on telemetry(exec-server): trace RPC transport #30675
• telemetry(exec-server): trace local process lifecycle #30679 local process lifecycle, stacked on telemetry(exec-server): trace RPC transport #30675

The original branch is preserved, but this umbrella is closed to avoid duplicate review and CI.