Use canonical FFE fixtures by leoromanovsky · Pull Request #11355 · DataDog/dd-trace-java

leoromanovsky · 2026-05-12T16:40:41Z

Motivation

Make DataDog/ffe-system-test-data the canonical source for FFE/OpenFeature JSON fixtures in dd-trace-java. This removes copied smoke-test fixtures and moves evaluator behavior coverage to the shared corpus. Depends on DataDog/ffe-system-test-data#9.

Changes

Add dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data as a git submodule pinned to the canonical fixture update.
Remove copied OpenFeature fixture JSON resources.
Update the smoke test to load canonical ufc-config.json and all sorted evaluation-cases/*.json files.
Assert canonical result values and reasons from the JSON corpus.
Remove the stale programmatic evaluator fixture set in favor of the canonical JSON-driven loop.
Fix the OpenFeature smoke CodeNarc violation by removing an unnecessary Groovy import.
Add weekly Dependabot gitsubmodule updates.

📖 https://datadoghq.atlassian.net/wiki/spaces/PANA/pages/6713639211/FFE+SDK+Fixture+Contribution+Guide

Decisions

Shared evaluator behavior must come from JSON fixtures in ffe-system-test-data, not programmatic or copied fixture sets.
dd-trace-java already supports submodules and recursive checkout in CI; this PR uses the same pattern for the OpenFeature fixture corpus.
PHP is intentionally excluded until its OpenFeature client lands.

Validation

JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH ./gradlew :dd-smoke-tests:openfeature:codenarcTest
JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH ./gradlew :products:feature-flagging:feature-flagging-api:test --tests datadog.trace.api.openfeature.DDEvaluatorTest

…0512

dd-octo-sts · 2026-05-18T12:21:19Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 057cdefcb8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-datadog-prod-us1 · 2026-05-20T14:44:36Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 56.93% (-0.04%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f27c243 | Docs | Datadog PR Page | Give us feedback!}

…0512

dougqh

Two review comments authored by Claude (Opus 4.7), posted on @dougqh's behalf.

dougqh

Claude detected a couple behavioral changes that I'm uncertain are intentional.
I'll leave those to your discretion.
Otherwise, looks good to me.

leoromanovsky · 2026-05-26T16:27:26Z

Claude detected a couple behavioral changes that I'm uncertain are intentional. I'll leave those to your discretion. Otherwise, looks good to me.

@dougqh Thanks for the review; agreed with the assessment to take a closer look and remediate. I'll merge after someone on FFE reviews.

sameerank

Maybe one documentation thing, not strictly required

dd-trace-java/BUILDING.md

Line 182 in fbf651e

    
           > This will keep the submodule in `dd-java-agent/agent-jmxfetch/integrations-core` up-to-date.

could mention both submodules: This will keep the submodules in `dd-java-agent/agent-jmxfetch/integrations-core` and `dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data` up-to-date.

leoromanovsky · 2026-05-26T18:43:33Z

Maybe one documentation thing, not strictly required

dd-trace-java/BUILDING.md

Line 182 in fbf651e

> This will keep the submodule in `dd-java-agent/agent-jmxfetch/integrations-core` up-to-date.

could mention both submodules: This will keep the submodules in `dd-java-agent/agent-jmxfetch/integrations-core` and `dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data` up-to-date.

@sameerank Good idea!

dd-octo-sts · 2026-05-29T01:28:42Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-29T01:28:46Z

View all feedbacks in Devflow UI.

2026-05-29 01:28:45 UTC ℹ️ Start processing command /merge

2026-05-29 01:28:50 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-05-29 02:06:08 UTC ❌ MergeQueue: This merge request was updated

This PR is rejected because it was updated

dd-octo-sts · 2026-05-29T01:40:59Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-29T01:41:03Z

View all feedbacks in Devflow UI.

2026-05-29 01:41:02 UTC ℹ️ Start processing command /merge

2026-05-29 01:41:05 UTC ❌ MergeQueue

PR already in the queue with status in_progress

…0512

leoromanovsky · 2026-05-29T02:09:20Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-29T02:09:25Z

View all feedbacks in Devflow UI.

2026-05-29 02:09:24 UTC ℹ️ Start processing command /merge

2026-05-29 02:09:29 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-05-29 06:55:14 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

dd-octo-sts · 2026-05-29T02:24:13Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.01 s	13.90 s	[-0.0%; +1.6%] (no difference)
startup:insecure-bank:tracing:Agent	12.94 s	12.99 s	[-1.2%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	16.94 s	16.74 s	[+0.3%; +2.1%] (maybe worse)
startup:petclinic:iast:Agent	16.95 s	16.89 s	[-0.5%; +1.2%] (no difference)
startup:petclinic:profiling:Agent	16.88 s	16.98 s	[-1.7%; +0.4%] (no difference)
startup:petclinic:sca:Agent	16.99 s	16.84 s	[-0.2%; +2.0%] (no difference)
startup:petclinic:tracing:Agent	16.06 s	16.08 s	[-1.2%; +1.0%] (no difference)

Commit: f27c2436 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…canonical-fixtures-20260512 # Conflicts: # products/feature-flagging/feature-flagging-api/src/test/java/datadog/trace/api/openfeature/DDEvaluatorTest.java

dd-octo-sts · 2026-07-06T04:58:37Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-06T04:58:41Z

View all feedbacks in Devflow UI.

2026-07-06 04:58:41 UTC ℹ️ Start processing command /merge

2026-07-06 04:58:46 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-06 06:02:47 UTC ℹ️ MergeQueue: This merge request was merged

leoromanovsky added 7 commits May 12, 2026 12:32

Use canonical FFE fixtures

9328a52

Catch up FFE missing flag behavior

12c4a57

Align Java smoke fixtures with canonical schema

8c7b1e9

Use canonical fixtures in Java evaluator tests

afd889b

Update FFE fixture submodule

e43944c

Fix OpenFeature smoke CodeNarc

14f68be

Merge branch 'master' into leo.romanovsky/ffe-canonical-fixtures-2026…

057cdef

…0512

leoromanovsky marked this pull request as ready for review May 18, 2026 12:21

leoromanovsky requested review from a team as code owners May 18, 2026 12:21

leoromanovsky requested review from dougqh, sameerank and typotter and removed request for a team May 18, 2026 12:21

chatgpt-codex-connector Bot reviewed May 18, 2026

View reviewed changes

Comment thread ...e-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DDEvaluator.java Outdated

leoromanovsky added the type: refactoring label May 20, 2026

Merge branch 'master' into leo.romanovsky/ffe-canonical-fixtures-2026…

82670f9

…0512

dougqh reviewed May 26, 2026

View reviewed changes

Comment thread ...e-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DDEvaluator.java Outdated

Comment thread ...openfeature/src/test/groovy/datadog/smoketest/springboot/OpenFeatureProviderSmokeTest.groovy Outdated

dougqh approved these changes May 26, 2026

View reviewed changes

Expand OpenFeature smoke exposure coverage

18581f1

leoromanovsky added the comp: openfeature OpenFeature label May 26, 2026

Restore missing flag error reporting

8f6ec04

Apply OpenFeature smoke test formatting

508b941

sameerank approved these changes May 26, 2026

View reviewed changes

Document OpenFeature fixture submodule

a299b27

leoromanovsky enabled auto-merge May 26, 2026 18:43

leoromanovsky added this pull request to the merge queue May 29, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 29, 2026

leoromanovsky added this pull request to the merge queue May 29, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 29, 2026

Merge branch 'master' into leo.romanovsky/ffe-canonical-fixtures-2026…

fbfd381

…0512

Merge remote-tracking branch 'origin/master' into leo.romanovsky/ffe-…

f27c243

…canonical-fixtures-20260512 # Conflicts: # products/feature-flagging/feature-flagging-api/src/test/java/datadog/trace/api/openfeature/DDEvaluatorTest.java

leoromanovsky enabled auto-merge July 6, 2026 04:15

leoromanovsky added this pull request to the merge queue Jul 6, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 6, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 936282c into master Jul 6, 2026
588 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffe-canonical-fixtures-20260512 branch July 6, 2026 06:02

github-actions Bot added this to the 1.64.0 milestone Jul 6, 2026

leoromanovsky mentioned this pull request Jul 6, 2026

Limit OpenFeature fixture submodule updates #11867

Open

PerfectSlayer mentioned this pull request Jul 6, 2026

chore(deps): bump dd-java-agent/agent-jmxfetch/integrations-core from 3189af0 to 0e7cf17 #11863

Closed

Uh oh!

Conversation

leoromanovsky commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Validation

Uh oh!

dd-octo-sts Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented May 20, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dougqh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dougqh left a comment

Choose a reason for hiding this comment

Uh oh!

leoromanovsky commented May 26, 2026

Uh oh!

sameerank left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leoromanovsky commented May 26, 2026

Uh oh!

dd-octo-sts Bot commented May 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented May 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

leoromanovsky commented May 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

dd-octo-sts Bot commented Jul 6, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leoromanovsky commented May 12, 2026 •

edited

Loading

dd-octo-sts Bot commented May 18, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented May 20, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

sameerank left a comment •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 •

edited

Loading

dd-octo-sts Bot commented May 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 6, 2026 •

edited

Loading