Skip to content

Add LLMObs integration tests for Java OpenAI SDK#6364

Draft
ygree wants to merge 6 commits intomainfrom
ygree/openai-java-tests
Draft

Add LLMObs integration tests for Java OpenAI SDK#6364
ygree wants to merge 6 commits intomainfrom
ygree/openai-java-tests

Conversation

@ygree
Copy link
Contributor

@ygree ygree commented Feb 20, 2026

Motivation

Add system tests for LLM Observability integration with the Java OpenAI SDK, covering chat completions, tool calls, responses API, and embedding interactions.

Changes

  • Add a new Java test app (openai_app) built with Gradle/Javalin that implements OpenAI proxy endpoints for
    testing
  • Add openai-java.Dockerfile to build and run the Java test app with dd-java-agent
  • Bump dd-apm-test-agent from v1.38.0 to v1.42.0
  • Add support for /evp_proxy/v4 endpoint in llmobs_requests(); add gzip decompression and MessagePack
    decoding for Java tracer payloads

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@ygree ygree self-assigned this Feb 20, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2026

CODEOWNERS have been resolved as:

tests/integration_frameworks/utils/vcr-cassettes/openai/test_chat_completion_stream_False_openai_chat_completions_post_424f6218.json  @DataDog/system-tests-core
tests/integration_frameworks/utils/vcr-cassettes/openai/test_chat_completion_tool_call_stream_False_openai_chat_completions_post_ec6364ef.json  @DataDog/system-tests-core
tests/integration_frameworks/utils/vcr-cassettes/openai/test_responses_create_stream_False_openai_responses_post_661cf6eb.json  @DataDog/system-tests-core
utils/build/docker/java/openai-java.Dockerfile                          @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/.gitignore                           @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/build.gradle                         @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/gradle/wrapper/gradle-wrapper.jar    @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/gradle/wrapper/gradle-wrapper.properties  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/gradlew                              @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/settings.gradle                      @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/src/main/java/SingleFileServer.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/openai_app/system_tests_library_version.sh      @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
tests/integration_frameworks/llm/openai/test_openai_llmobs.py           @DataDog/ml-observability
tests/integration_frameworks/llm/utils.py                               @DataDog/ml-observability
utils/_context/_scenarios/integration_frameworks.py                     @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core

@ygree ygree force-pushed the ygree/openai-java-tests branch 2 times, most recently from 335d036 to 5e40f8a Compare February 21, 2026 00:03
@datadog-official
Copy link

datadog-official bot commented Feb 21, 2026

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d26893b | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@ygree ygree force-pushed the ygree/openai-java-tests branch from 5e40f8a to 53695b1 Compare February 21, 2026 00:35
output_tokens n/a thus not set
java uses base64 encoding format instead of float
@ygree ygree changed the title openai-java tests Add LLMObs integration tests for Java OpenAI SDK Mar 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant