Observability LLM payload + GenAI semconv (v0.17.0)

[chris-colinsky]

Summary

Realizes spec v0.17.0 §5.5 expansion (proposal 0024) and the five python-only ergonomics items from the observability friction roundup. Triggered by a downstream agent integrating OA with Langfuse over OTLP: LLM spans rendered "naked" (model + tokens only), prompt linkage silently dropped at the dispatch-worker task boundary, and every backend needed a per-service attribute-mapping shim. All eight friction items resolved here.

What's in

LLM span attribute expansion (spec v0.17.0):

• §5.5.1 input/output payload — openarmature.llm.input.messages, openarmature.llm.output.content, openarmature.llm.request.extras. JSON-encoded with sorted keys, default-off via disable_llm_payload: bool = True, truncated at payload_max_bytes (default 64 KiB, minimum 256). Inline image bytes redacted at the provider before reaching the event payload — not gated by any observer flag, defence-in-depth that applies to every custom observer too.
• §5.5.2 / §5.5.3 GenAI semconv — gen_ai.system, gen_ai.request.model, gen_ai.response.{model,id,finish_reasons}, gen_ai.usage.{input,output}_tokens, plus per-set gen_ai.request.{temperature,max_tokens,top_p,seed}. Opt out via disable_genai_semconv. gen_ai.system is caller-set per OpenAIProvider instance (default "openai"; override to "vllm" / "lm_studio" / etc. when hitting non-OpenAI endpoints that speak the OpenAI wire format).

OTelObserver ergonomics:

• resource=Resource(...) constructor field.
• span_processor now accepts SpanProcessor | Sequence[SpanProcessor].
• attribute_enrichers: Sequence[Callable[[Span, NodeEvent | None], None]] hook fires before every span.end() the observer issues. Exceptions are caught and warned, never propagated.
• shutdown() docstring documents the BatchSpanProcessor flush gotcha under fast teardown.

Public surface:

• openarmature.observability.LLM_NAMESPACE and openarmature.observability.LlmEventPayload. The payload type now lives at openarmature.observability.llm_event (moved from the provider module to break a circular import that the lazy __getattr__ was masking; subclasses plain Pydantic BaseModel since NodeEvent.pre_state is Any). _LlmEventState retained as a deprecated alias on the provider module for one release.
• Response.response_id and Response.response_model typed fields sourcing the new gen_ai.response.* attributes.

Cross-task ContextVar bug fix:

• The dispatch worker (asyncio.create_task(deliver_loop(queue))) snapshots its Context at invoke() entry — before any node body opens a with_active_prompt(...) block. The observer's current_prompt_result() / current_prompt_group() reads from the worker task saw None even when a node body had set them. Fixed by capturing both ContextVars at dispatch time inside OpenAIProvider.complete() (in the node task, where the writes are live) and putting the snapshot on LlmEventPayload. Observer reads from the payload.

Tests:

• 10 new conformance fixture drivers (012-021) consuming the new harness primitives (content_repeat, base64_data_synthetic, attribute_parses_as_messages, attribute_does_not_contain, attribute_truncation, etc.).
• tests/conformance/harness/llm_attribute_assertions.py assertion-helper module.
• End-to-end graph: parameterize GraphBuilder/CompiledGraph on StateT (PEP 695 generics) #3 regression test exercising the real cross-task boundary via provider.complete() inside a node body inside invoke().
• Fixed a pre-existing test-isolation issue: OTel SDK 1.x makes set_tracer_provider one-shot, so the unit-test finally blocks silently failed to restore the global, leaking it to fixture 005 sub-case 3. Reset via the SDK's private _TRACER_PROVIDER_SET_ONCE primitive in both call sites.

Docs + examples:

• CHANGELOG [Unreleased] section.
• docs/concepts/observability.md — seven new subsections under OTel coverage.
• docs/model-providers/authoring.md — expanded "Observability spans" bullet to a runnable dispatch sketch.
• README — sibling pitch bullet under the existing "doesn't double-export" framing.
• Example 03 (observer-hooks) — added Resource for service.name.
• Example 07 (multimodal-prompt) — wired OTelObserver so the example's headline claim ("OTel observers stamp openarmature.prompt.group_name onto every LLM-call span") is now actually demonstrable end-to-end.

What's out

Per the coord-thread scope (discuss-observability-friction-roundup 02-03):

• test: handle model instances in fixture discriminators #8 Langfuse-native backend, ci: tag-driven release pipeline with TestPyPI RC track #11 langfuse.trace.name on invocation root — separate proposal coming in discuss-observability-langfuse-mapping.
• chore: support plural subgraphs: form, pull spec v0.8.1 #10 maybe_install_log_bridge wrapper — kept the explicit two-line caller-side conditional.
• Tool-call observability — out of scope for v0.17.0 per spec scope answers Q3.

Test plan

• uv run pytest tests/ -q — full suite (729 passed, 70 skipped on this branch).
• uv run pytest tests/conformance/test_observability.py -v — confirm all 21 observability fixtures green (11 v0.7.0 + 10 new).
• uv run pytest tests/unit/test_observability_otel.py::test_prompt_context_propagates_cross_task_via_provider_complete -v — the graph: parameterize GraphBuilder/CompiledGraph on StateT (PEP 695 generics) #3 regression test (would have failed on pre-fix code).
• uv run pyright and uv run ruff check clean.
• Spot-check LLM_NAMESPACE and LlmEventPayload import cleanly from openarmature.observability in both load orders (observability-first, llm-first).
• Run example 07 against any OpenAI-compatible endpoint and confirm the console-exporter JSON for the two openarmature.llm.complete spans carries openarmature.prompt.group_name = "lunar-image-analysis" plus the per-call openarmature.prompt.* attributes and the new gen_ai.* set.

[Copilot]

Pull request overview

Implements OpenArmature observability spec v0.17.0 §5.5 expansion by enriching openarmature.llm.complete spans with optional input/output payload attributes and default-on OpenTelemetry GenAI semantic conventions, plus several OTelObserver ergonomics improvements (resource, multi span processors, enrichers) and a cross-task prompt-context propagation fix. The PR also updates conformance + unit tests and expands docs/examples to reflect the new observability behavior.

Changes:

• Add LLM payload attributes (default-off, truncation, provider-side inline image redaction) and GenAI semconv (gen_ai.*) on LLM spans.
• Fix prompt-context propagation across the dispatch-worker task boundary by snapshotting prompt ContextVars into the LLM event payload.
• Improve OTelObserver ergonomics (Resource support, multiple SpanProcessors, pre-end attribute enrichers) and extend conformance/unit coverage + documentation.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/openarmature/observability/otel/observer.py	Adds payload/semconv emission, truncation, Resource support, multi-processors, and attribute enrichers.
src/openarmature/observability/llm_event.py	Introduces the typed LLM event payload model used between providers and observers.
src/openarmature/llm/providers/openai.py	Captures prompt context at dispatch, emits expanded LLM payload fields, adds genai_system, and records response id/model.
src/openarmature/llm/response.py	Adds typed response_id and response_model fields.
src/openarmature/observability/__init__.py	Exports LLM event contract symbols (but currently imports from OTel backend).
tests/conformance/test_observability.py	Adds v0.17.0 fixture drivers and helper logic for payload/semconv fixtures; fixes tracer provider global reset in fixture 005.
tests/conformance/harness/llm_attribute_assertions.py	New assertion helpers for LLM payload/semconv fixtures (shape, truncation, redaction).
tests/conformance/harness/directives.py	Adds calls_llm.config schema support for RuntimeConfig parameters/extras.
tests/conformance/harness/fixtures.py	Extends fixture schema with observer/provider flags for new v0.17.0 attributes.
tests/unit/test_observability_otel.py	Adds tracer-provider reset helper and a cross-task prompt propagation regression test; updates prompt attribute unit test to use payload snapshot.
tests/test_smoke.py	Updates pinned spec version assertion to 0.17.0.
src/openarmature/__init__.py	Bumps __spec_version__ to 0.17.0.
pyproject.toml	Bumps [tool.openarmature].spec_version to 0.17.0.
README.md	Documents improved LLM span readability via GenAI semconv + optional payloads.
docs/concepts/observability.md	Adds detailed sections on LLM spans, payloads (truncation/redaction), Resource, multi-export, enrichers, and teardown flushing.
docs/model-providers/authoring.md	Documents public LLM event dispatch contract (LLM_NAMESPACE, LlmEventPayload).
docs/examples/index.md	Updates examples install instructions for OTel SDK usage.
docs/examples/07-multimodal-prompt.md	Updates example doc to describe OTelObserver output and GenAI attributes.
examples/03-observer-hooks/main.py	Adds Resource usage and updates commentary for GenAI semconv.
examples/07-multimodal-prompt/main.py	Wires OTelObserver + Resource so prompt/group attributes are visible in exported spans.
CHANGELOG.md	Adds an Unreleased entry describing the v0.17.0 observability changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.