Complete 0067 embedding metrics#239
Merged
Merged
Conversation
Wire the section 11 GenAI metric instruments to the embedding event, flipping proposal 0067 from partial to implemented. - otel: _record_embedding_metrics records the same token.usage + operation.duration histograms from the terminal EmbeddingEvent / EmbeddingFailedEvent, with operation="embeddings" and input-only tokens (an embedding call has no output tokens); duration every call including a failure (error.type), token only when usage is present. - conformance: un-defer fixtures 089 (usage) and 143 (no-usage); mark 0067 implemented. Rerank metrics (109) stay deferred, out of scope per the proposal.
There was a problem hiding this comment.
Pull request overview
This PR completes proposal 0067’s remaining gap by recording OTel GenAI metric instruments for embedding calls, using the terminal EmbeddingEvent/EmbeddingFailedEvent path introduced in v0.16.0. It aligns embedding metrics behavior with the existing LLM attempt metrics (same instruments, different openarmature.gen_ai.operation).
Changes:
- Add
_record_embedding_metrics(event)toOTelObserverand invoke it for embedding success/failure events whenenable_metricsis on. - Un-defer and run conformance fixtures 089 and 143 by extending the embedding fixture runner to capture and assert metric points + embedding-specific metric invariants.
- Add unit tests covering embedding metrics for usage present, usage present with
input_tokens=0, usage absent, and failure (error.typeon duration).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/openarmature/observability/otel/observer.py | Records token.usage (input-only) and operation.duration for embedding events under openarmature.gen_ai.operation="embeddings" when metrics are enabled. |
| tests/conformance/test_observability.py | Enables embedding metrics fixtures (089/143) and adds an embedding-metrics invariant assertion layer over captured metric points. |
| tests/unit/test_observability_otel.py | Adds unit coverage for embedding metrics across usage/no-usage/zero-usage/failure scenarios. |
| conformance.toml | Marks proposal 0067 as implemented in v0.17.0 and updates the proposal note to reflect embedding metrics completion. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes proposal 0067 (OTel GenAI metrics, observability §11) by wiring the metric instruments to the embedding event, flipping 0067 from
partialtoimplemented. This is the last item in v0.17.0's Wave 2.Context
0067's LLM path (fixtures 088/090/091) shipped in v0.15.0, recording two OA-namespaced instruments per LLM attempt. The embedding-call metrics were deferred because no embedding event existed yet; v0.16.0's retrieval work now dispatches
EmbeddingEvent/EmbeddingFailedEvent, so this wires the same instruments to that path.What it adds
observability/otel/observer.py—_record_embedding_metrics(event)records the sametoken.usage+operation.durationhistograms from the terminalEmbeddingEvent/EmbeddingFailedEvent(theoperationdimension separatesembeddingsfromchat), withoperation="embeddings"andtoken.type="input"— an embedding call has input tokens only (EmbeddingUsage.input_tokens, no output). Duration records on every call including a failure (error.type);token.usageonly when a usage record is present. Dispatched beside_handle_embedding, gated onenable_metricsand independent ofdisable_llm_spans.implemented.Rerank metrics (fixture 109) stay deferred: out of scope per the proposal.
Tests
Fixtures 089/143 plus unit coverage for embedding metrics: with usage, a reported
input_tokens=0(records a zero observation, distinct from absent usage), no-usage, and a failure (duration witherror.type— the one path no fixture covers). Full suite green; ruff and pyright clean.