CAMEL-24559: Extend GenAI observability to OpenAI embeddings, moderation, and responses - #26106
Conversation
…n, and responses Extend camel-openai producers beyond chat-completion with OpenTelemetry spans and Micrometer metrics using the existing GenAiObservability API. - Instrument OpenAIEmbeddingsProducer, OpenAIModerationProducer, and OpenAIResponsesProducer with try/recordSuccess/recordError/close - Add GenAiOperationName.MODERATION for content-policy operations - Add OpenAIEmbeddingsObservabilityTest, OpenAIModerationObservabilityTest, OpenAIResponsesObservabilityTest, and shared test support - Document OpenAI operation coverage in ai-observability.adoc and 4.23 upgrade guide Co-authored-by: Cursor Agent <noreply@cursor.com>
- Scope GenAI spans to SDK calls only (embeddings/moderation) - Map openai:responses to gen_ai.operation.name=chat per OTel OpenAI usage - Clarify MODERATION as Camel extension in enum javadoc and docs - Strengthen tests: body/header assertions, exact token tags, override properties Co-authored-by: Cursor Agent <noreply@cursor.com>
|
Links CAMEL-24559 — first follow-up slice for extending GenAI observability beyond chat/streaming. Review status: Bugbot (no bugs) and Grok review feedback addressed in commit AI-generated comment on behalf of @atiaomar1978-hub |
|
Links CAMEL-24559 — first follow-up slice. Bugbot + Grok review feedback addressed in 38b10e2. |
|
🌟 Thank you for your contribution to the Apache Camel project! 🌟 🐫 Apache Camel Committers, please review the following items:
|
|
🧪 CI tested the following changed modules:
🔬 Scalpel shadow comparison — Scalpel: 73 tested, 25 compile-only — current: 71 all testedMaveniverse Scalpel detected 98 affected modules (current approach: 71).
|
gnodet
left a comment
gnodet
left a comment
There was a problem hiding this comment.
Review summary: Solid and well-structured extension of GenAI observability to three additional OpenAI producers. The pattern follows the established createChatCompletion in OpenAIProducer closely. Two issues worth addressing before merge.
This review was generated by an automated reviewer (Hermès) on behalf of @gnodet.
| calculateSimilarityIfRequested(exchange, embeddings); | ||
| } | ||
|
|
||
| private static Integer toTokenCount(long tokens) { |
There was a problem hiding this comment.
Math.toIntExact(tokens) will throw ArithmeticException if the token count exceeds Integer.MAX_VALUE. Since this is called inside the try block that catches Exception, a hypothetical overflow would abort the entire exchange even though the API call itself succeeded.
More importantly, this narrowing is unnecessary. GenAiUsage accepts Long directly — the existing createChatCompletion in OpenAIProducer passes usage.promptTokens() (a long) straight to GenAiUsage.of(Long, Long, ...) without any conversion:
| private static Integer toTokenCount(long tokens) { | |
| private static Long toTokenCount(long tokens) { | |
| return tokens; | |
| } |
Or just inline the long values directly and drop toTokenCount entirely, matching the OpenAIProducer pattern.
| finishReason, | ||
| response.model().toString())), | ||
| () -> observation.recordSuccess(GenAiUsage.of(null, null, finishReason, response.model().toString()))); | ||
| } |
There was a problem hiding this comment.
Same issue as in OpenAIEmbeddingsProducer — Math.toIntExact is both unnecessary (the Long overload of GenAiUsage.of exists) and risky (throws ArithmeticException on overflow, which the catch block would treat as a failed operation).
| } | |
| private static Long toTokenCount(long tokens) { | |
| return tokens; | |
| } |
| .system("openai") | ||
| .requestModel(model) | ||
| .componentScheme("openai") | ||
| .build(); |
There was a problem hiding this comment.
createResponse and createStructuredResponse are nearly identical — same observation context, same error handling, same finally. The only difference is the SDK call. Consider extracting a common helper, e.g.:
private Response observedCall(Exchange exchange, String model,
ThrowingSupplier<Response> call) throws Exception {
GenAiObservationContext ctx = GenAiObservationContext.builder()
.operationName(GenAiOperationName.CHAT)
.system("openai").requestModel(model)
.componentScheme("openai").build();
GenAiObservation observation = GenAiObservability.start(exchange, ctx);
try {
Response response = call.get();
recordResponseSuccess(observation, response);
return response;
} catch (Exception e) {
GenAiErrorSupport.apply(exchange, e);
observation.recordError(e);
throw e;
} finally {
observation.close();
}
}Then createResponse becomes observedCall(exchange, model, () -> getEndpoint().getClient().responses().create(params)) and createStructuredResponse becomes observedCall(exchange, model, () -> getEndpoint().getClient().responses().create(structuredParams).rawResponse()).
Not blocking, but reduces ~40 lines of duplication to ~2.
| observation.recordError(e); | ||
| throw e; | ||
| } finally { | ||
| observation.close(); |
There was a problem hiding this comment.
The comment that was here ("this operation is used to gate untrusted content, so a missing verdict must fail the exchange...") explained a security invariant — why a mismatched result count throws rather than silently proceeding. Worth keeping; it's not redundant with the observability changes.
3 participants
Summary
Follow-up to CAMEL-23861 / CAMEL-24559: extend GenAI observability to additional OpenAI producer operations beyond
chat-completion.This PR instruments OpenAI embeddings, moderation, and responses with the existing
GenAiObservabilityAPI.gen_ai.operation.nameopenai:embeddingsembeddingsopenai:moderationmoderation(Camel extension)openai:responseschatopenai:chat-completionchat(unchanged)Changes
OpenAIEmbeddingsProducer,OpenAIModerationProducer,OpenAIResponsesProducerGenAiOperationName.MODERATIONOpenAIEmbeddingsObservabilityTest,OpenAIModerationObservabilityTest,OpenAIResponsesObservabilityTestai-observability.adoc, catalog mirror, 4.23 upgrade guideTest plan
./mvnw -pl components/camel-ai/camel-openai -am test -Dtest=OpenAIEmbeddingsObservabilityTest,OpenAIModerationObservabilityTest,OpenAIResponsesObservabilityTestImages, audio, Spring AI modules, and cloud LLMs remain for future CAMEL-24559 follow-ups.
AI-generated PR description on behalf of @atiaomar1978-hub