feat(tracing): correlate business spans with obs via dedicated wrapper span

[NiteshDhanpal]

What & why

Model the business ↔ observability correlation edge on the emit side, so a persisted business span can pivot to its Tempo/Datadog trace and back. No schema migration — the ids ride the existing operation_metadata JSONB (→ ClickHouse metadata_raw).

Two worlds today: business spans (trace_id = task id, stored in application_trace_span + CH) and obs traces (OTel/ddtrace in Tempo/Datadog) share no id. This adds the link.

Changes

• obs_ids.py — correlation keys standardized to obs_trace_id/obs_span_id (underscored → Postgres JSON-path addressable); removed the non-working dual mode (it needed an in-process ddtrace↔OTel bridge that can't exist; degrades to dd_only); obs_correlation() hardened to never raise.
• obs_span.py (new) — dedicated per-business-span wrapper obs span:
  • Opens a real span named for the step and makes it active, so obs_span_id is stable/meaningful (not an arbitrary innermost httpx span); nested instrumentation parents under it.
  • Backends: OTel (lgtm) / ddtrace (dd_only, only when a request trace is active — avoids orphan roots).
  • Reverse tag: stamps agentex.business_span_id / agentex.business_trace_id on the obs span → bidirectional pivot.
  • Error status propagated to the obs span (failed step is not a false green).
  • child_of nesting (ddtrace): ddtrace's start_span does not auto-parent — passes child_of=current_trace_context() so a turn's spans roll up into ONE trace instead of N roots.
• trace.py — wraps start_span/end_span (sync + async). Observability can never fail an app call (guarded; no-op when unconfigured); the two backends never interfere.

Behavior (Turn 2 of the 3-turn example)

business step	before (obs_span_id)	with wrapper
get_state	rB	wB1
retrieve_docs	rB	wB2
chat_completion	rB	wB3
create_message	rB	wB4

Distinct, named obs_span_id per step; all under the one turn obs trace B.

✅ Live validation — infra-staging, dd_only, real rocket-mock turn (52 spans)

Deployed this branch (git-dep) to sgp-rocket-mock-agent on sgp-infra-staging, fired a real message/send, and read the spans straight out of egp application_trace_span.operation_metadata:

• Edge populates: 52/52 business spans carry obs_trace_id + obs_span_id in operation_metadata — no migration.

• Wrapper works: obs_span_id is distinct per step (52 distinct) — a dedicated named span per business step, not the coarse request span.

• child_of fix — before/after:

  	DISTINCT obs_trace_id	DISTINCT obs_span_id
  before (start_span no child_of)	52 (each a new root)	52
  after (child_of=current_ctx)	1 (per-turn roll-up)	52

  → all 52 spans now share the one request/distributed trace id (ca5e90d9…), each a distinct named wrapper span nested under it — the per-turn (TurnTrace) shape.

• Sample operation_metadata: {"turn":0,"agent":"rocket_mock_agent","__source__":"agentex","obs_trace_id":"ca5e90d9…","obs_span_id":"…","__agent_id__":"c7a9692f…"}

Tests

test_obs_ids.py + test_obs_span.py: mode degrade + keys, both backends, non-interference, never-fails, reverse tag, error status, child_of nesting, and the Turn-2 example. Full tracing suite green.

Not in this PR (follow-ups)

• lgtm/Tempo population needs an OTel TracerProvider in the agent (operator auto-instrumentation or app-level) — SDK side is complete and backend-agnostic.
• The sgp-obs-tracing-middleware library still carries dual/bridge.py — separate cleanup.

🤖 Generated with Claude Code

Greptile Summary

Adds dedicated observability wrapper spans for business spans.

• Selects OTel or Datadog according to the configured observability mode and records bidirectional correlation identifiers.
• Integrates wrapper creation, error propagation, and closure into synchronous and asynchronous tracing.
• Adds coverage for identifier formatting, backend isolation, lifecycle behavior, and cross-instance closure.

Confidence Score: 4/5

The synchronous startup cleanup failure should be fixed before merging because an ordinary processor exception can permanently leak an active wrapper span.

The new handle is registered before fallible synchronous processor callbacks, while the only normal cleanup path requires start_span to return successfully.

Files Needing Attention: src/agentex/lib/core/tracing/trace.py

Important Files Changed

Filename	Overview
src/agentex/lib/core/tracing/obs_ids.py	Standardizes correlation metadata keys, removes dual-mode selection, and makes identifier lookup best-effort.
src/agentex/lib/core/tracing/obs_span.py	Implements backend-specific wrapper-span activation, correlation, reverse tags, error propagation, and closure.
src/agentex/lib/core/tracing/trace.py	Integrates wrapper handles into both tracing paths, but synchronous processor startup failures can leak activated wrappers.

Sequence Diagram

sequenceDiagram
    participant App
    participant Trace
    participant Obs as OTel / Datadog
    participant Processor
    App->>Trace: start_span(name)
    Trace->>Obs: start and activate wrapper
    Obs-->>Trace: obs_trace_id, obs_span_id
    Trace->>Processor: on_span_start(business span)
    App->>Trace: end_span(span)
    Trace->>Obs: propagate error and close
    Trace->>Processor: on_span_end(span)

Loading

Comments Outside Diff (1)

1. src/agentex/lib/core/tracing/trace.py, line 124-130 (link)

   Processor failures leak wrapper spans

   When a synchronous on_span_start processor raises, the newly activated observability handle remains in _OBS_HANDLES because start_span never returns and the context manager cannot call end_span, leaving the wrapper unexported and its active context able to parent subsequent instrumentation incorrectly.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/agentex/lib/core/tracing/trace.py
   Line: 124-130
   
   Comment:
   **Processor failures leak wrapper spans**
   
   When a synchronous `on_span_start` processor raises, the newly activated observability handle remains in `_OBS_HANDLES` because `start_span` never returns and the context manager cannot call `end_span`, leaving the wrapper unexported and its active context able to parent subsequent instrumentation incorrectly.
   
   
   
   ---
   
   For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

   

Prompt To Fix All With AI

### Issue 1
src/agentex/lib/core/tracing/trace.py:124-130
**Processor failures leak wrapper spans**

When a synchronous `on_span_start` processor raises, the newly activated observability handle remains in `_OBS_HANDLES` because `start_span` never returns and the context manager cannot call `end_span`, leaving the wrapper unexported and its active context able to parent subsequent instrumentation incorrectly.

```suggestion
        if obs_handle is not None:
            _OBS_HANDLES[span.id] = obs_handle

        try:
            for processor in self.processors:
                processor.on_span_start(span)
        except Exception:
            close_obs_span(_OBS_HANDLES.pop(span.id, None))
            raise

        return span
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(tracing): satisfy pyright in obs-cor..." | Re-trigger Greptile}