RFC: Universal LLM Observability Semantic Convention (v0.4)

[sauravGit]

Hi everyone,

I'm proposing a vendor-neutral, OpenTelemetry-compatible semantic convention for LLM observability — and I'd love early feedback from this community.

The problem

Every LLM platform and observability tool today uses different field names, KPI definitions, and export formats. Developers who want end-to-end visibility across multiple providers or backends have to re-instrument every time.

What I'm proposing

A canonical schema — built on top of OpenTelemetry semantic conventions — that standardizes:

• Metric names (e.g., gen_ai.latency, gen_ai.usage.cost, gen_ai.time_to_first_token)
• Span names and required attributes for requests, tool calls, retrieval, guardrails
• Resource attributes for provider, model, app, environment
• Derived KPIs (success rate, cost per request, token efficiency, etc.)
• Interoperability rules so backends can transform labels without losing semantic meaning

The mandatory core covers: latency, tokens, cost, errors, retries, rate limits, and trace coverage. An optional domain extension layer covers quality signals like groundedness, relevance, and safety flags.

Why OpenTelemetry

OTEL is already the standard for distributed tracing and metrics. Defining LLM semantic conventions on top of it means any OTEL-compatible backend works without additional integration: Prometheus, Grafana, Datadog, GCP, Honeycomb, etc.

Full RFC

See RFC.md in this repo for the full v0.1 spec.

What I'm asking for

• Does the mandatory core make sense? Anything missing or unnecessary?
• Are the canonical metric names (gen_ai.*) clear and unambiguous?
• Does the OTEL mapping approach align with how the GenAI SIG is thinking about this?
• Who else is working on this problem and wants to collaborate?

Thanks — looking forward to the feedback.

[sauravGit]

Update: RFC-0001 v0.4 is now live 🎉

Just shipped v0.4 with all upstream feedback incorporated:

• Aligned metric names with OTel GenAI SIG (gen_ai.* namespace)
• Added OpenInference interop column to README
• Resolved open questions on error/retry/rate-limit signals
• Python SDK (open_llm_obs) updated to v0.4 — all CI checks green ✅

Full spec: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md

Would love continued feedback, especially on the streaming metrics and cost estimation approach!