Skip to content

feat(feature-flagging): FFE APM feature-flag span enrichment (experimental, gated) #11658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leoromanovsky wants to merge 24 commits into
base: master
Choose a base branch
from
Open

feat(feature-flagging): FFE APM feature-flag span enrichment (experimental, gated) #11658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
411b4f5
feat(02-02): surface UFC split serialId in eval metadata + add span-e…
leoromanovsky Jun 16, 2026
c076e23
feat(02-02): ULeb128 codec + accumulator + capture hook + write inter…
leoromanovsky Jun 16, 2026
44f51c5
test(02-02): JUnit 5 L0 suite for span enrichment (7 required cases +…
leoromanovsky Jun 16, 2026
545f72b
fix(02-02): correct span-enrichment lifecycle (partial-flush, leak, r…
leoromanovsky Jun 16, 2026
8491123
test(02-02): regression tests for span-enrichment lifecycle defects
leoromanovsky Jun 16, 2026
35aba86
chore(feature-flagging): strip internal review labels from span-enric…
leoromanovsky Jun 17, 2026
109eab8
fix(feature-flagging): address span-enrichment code-review findings
leoromanovsky Jun 17, 2026
c9cc06d
Merge master
pavlokhrebto Jul 2, 2026
7b4da18
Adjustments + clean up; add FeatureFlaggingConfig.SPAN_ENRICHMENT_ENA…
pavlokhrebto Jul 2, 2026
d29a8b3
Rework FFE span-enrichment state store to weak-keyed-by-span map
pavlokhrebto Jul 7, 2026
2a6926c
Merge branch 'master' of github.com:DataDog/dd-trace-java into leo.ro…
pavlokhrebto Jul 7, 2026
99422b5
Enhancements after techdebt skill
pavlokhrebto Jul 7, 2026
4be86c6
Refactor: Move span-enrichment write tier out of the published dd-op…
pavlokhrebto Jul 8, 2026
ac8195b
Use JsonWriter instead of hand-written solution
pavlokhrebto Jul 8, 2026
77a8ec2
Fix unicode char
pavlokhrebto Jul 8, 2026
5ac008d
Address review nits: gate enrichment metadata, name constants, reuse …
pavlokhrebto Jul 8, 2026
22255cb
feat(ffe): Introduce config module to decouple feature flagging from …
PerfectSlayer Jul 9, 2026
3b3ae47
Fix 3 bugs: SpanEnrichmentWriter is now a process-wide singleton; add…
pavlokhrebto Jul 9, 2026
f56665b
Fix nits: add debug logging; fix comments
pavlokhrebto Jul 9, 2026
aa31d10
Merge branch 'master' into leo.romanovsky/ffe-apm-span-enrichment
sarahchen6 Jul 9, 2026
d3b867a
Add gate on registration; adjust - add typed metadata
pavlokhrebto Jul 10, 2026
8399678
Merge branch 'master' of github.com:DataDog/dd-trace-java into leo.ro…
pavlokhrebto Jul 10, 2026
bc2b945
Merge branch 'leo.romanovsky/ffe-apm-span-enrichment' of github.com:D…
pavlokhrebto Jul 10, 2026
2d55eb9
Merge branch 'master' into leo.romanovsky/ffe-apm-span-enrichment
pavlokhrebto Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dd-java-agent/agent-bootstrap/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ dependencies {
api project(':dd-java-agent:agent-debugger:debugger-bootstrap')
api project(':components:environment')
api project(':components:json')
api project(':products:feature-flagging:feature-flagging-config')
api project(':products:metrics:metrics-agent')
api libs.instrument.java
api libs.slf4j
Expand Down
2 changes: 1 addition & 1 deletion dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@
import datadog.trace.api.config.CrashTrackingConfig;
import datadog.trace.api.config.CwsConfig;
import datadog.trace.api.config.DebuggerConfig;
import datadog.trace.api.config.FeatureFlaggingConfig;
import datadog.trace.api.config.GeneralConfig;
import datadog.trace.api.config.IastConfig;
import datadog.trace.api.config.JmxFetchConfig;
Expand All @@ -44,6 +43,7 @@
import datadog.trace.api.config.TraceInstrumentationConfig;
import datadog.trace.api.config.TracerConfig;
import datadog.trace.api.config.UsmConfig;
import datadog.trace.api.featureflag.config.FeatureFlaggingConfig;
import datadog.trace.api.gateway.RequestContextSlot;
import datadog.trace.api.gateway.SubscriptionService;
import datadog.trace.api.git.EmbeddedGitInfoBuilder;
Expand Down
1 change: 0 additions & 1 deletion dd-trace-api/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ extra["excludedClassesCoverage"] = listOf(
"datadog.trace.api.civisibility.noop.NoOpDDTestSession",
"datadog.trace.api.civisibility.noop.NoOpDDTestSuite",
"datadog.trace.api.config.AIGuardConfig",
"datadog.trace.api.config.FeatureFlaggingConfig",
"datadog.trace.api.config.ProfilingConfig",
"datadog.trace.api.interceptor.MutableSpan",
"datadog.trace.api.profiling.Profiling",
Expand Down
6 changes: 0 additions & 6 deletions dd-trace-api/src/main/java/datadog/trace/api/config/FeatureFlaggingConfig.java
View file
Open in desktop

This file was deleted.

8 changes: 8 additions & 0 deletions metadata/supported-configurations.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1505,6 +1505,14 @@
"aliases": []
}
],
"DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED": [
{
"version": "A",
"type": "boolean",
"default": "false",
"aliases": []
}
],
"DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED": [
{
"version": "B",
Expand Down
13 changes: 13 additions & 0 deletions ...g/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ public class FeatureFlaggingSystem {

private static volatile RemoteConfigService CONFIG_SERVICE;
private static volatile ExposureWriter EXPOSURE_WRITER;
private static volatile SpanEnrichmentWriter SPAN_ENRICHMENT_WRITER;

private FeatureFlaggingSystem() {}

Expand All @@ -27,10 +28,22 @@ public static void start(final SharedCommunicationObjects sco) {
EXPOSURE_WRITER = new ExposureWriterImpl(sco, config);
EXPOSURE_WRITER.init();

// APM span enrichment: agent-side listener for flag-evaluation seam events. Uses the process-
// wide singleton so a subsystem restart reuses the one already-registered trace interceptor
// (which the tracer cannot remove) instead of registering a second, rejected one. Cheap: it
// only accumulates once the provider's gate-on capture hook dispatches events, and registers
// its interceptor lazily on the first such event.
SPAN_ENRICHMENT_WRITER = SpanEnrichmentWriter.getInstance();
SPAN_ENRICHMENT_WRITER.init();

LOGGER.debug("Feature Flagging system started");
}

public static void stop() {
if (SPAN_ENRICHMENT_WRITER != null) {
SPAN_ENRICHMENT_WRITER.close();
SPAN_ENRICHMENT_WRITER = null;
}
if (EXPOSURE_WRITER != null) {
EXPOSURE_WRITER.close();
EXPOSURE_WRITER = null;
Expand Down
3 changes: 3 additions & 0 deletions products/feature-flagging/feature-flagging-api/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,12 @@ dependencies {
api("dev.openfeature:sdk:1.20.1")

compileOnly(project(":products:feature-flagging:feature-flagging-bootstrap"))
compileOnly(project(":products:feature-flagging:feature-flagging-config"))
compileOnly(project(":utils:config-utils"))
compileOnly("io.opentelemetry:opentelemetry-api:1.47.0")

testImplementation(project(":products:feature-flagging:feature-flagging-bootstrap"))
testImplementation(project(":utils:config-utils"))
testImplementation("io.opentelemetry:opentelemetry-api:1.47.0")
testImplementation(libs.bundles.junit5)
testImplementation(libs.bundles.mockito)
Expand Down
21 changes: 21 additions & 0 deletions ...lagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DDEvaluator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,16 @@ class DDEvaluator implements Evaluator, FeatureFlaggingGateway.ConfigListener {
private static final Set<Class<?>> SUPPORTED_RESOLUTION_TYPES =
new HashSet<>(asList(String.class, Boolean.class, Integer.class, Double.class, Value.class));

// Evaluation-metadata keys consumed by the span-enrichment capture hook (see
// SpanEnrichmentHook). Emitted only when the span-enrichment gate is on.
static final String METADATA_SPLIT_SERIAL_ID = "__dd_split_serial_id";
static final String METADATA_DO_LOG = "__dd_do_log";

// Read once: when off, the __dd_* span-enrichment metadata is not attached to evaluations, so an
// enabled provider pays nothing extra unless span enrichment is also enabled. The gate does not
// change at runtime, and this class is loaded lazily (well after startup) so config is ready.
private static final boolean SPAN_ENRICHMENT_ENABLED = SpanEnrichmentGate.isEnabled();

private final Runnable configCallback;
private final AtomicReference<ServerConfiguration> configuration = new AtomicReference<>();
private final CountDownLatch initializationLatch = new CountDownLatch(1);
Expand Down Expand Up @@ -392,6 +402,17 @@ private static <T> ProviderEvaluation<T> resolveVariant(
.addString("flagKey", flag.key)
.addString("variationType", flag.variationType.name())
.addString("allocationKey", allocation.key);
// Surface the UFC split's serial id and the allocation's doLog flag for APM span enrichment —
// only when span enrichment is on, so a provider without enrichment pays nothing extra.
// __dd_split_serial_id is omitted when the split carries no serial id; __dd_do_log is always
// present (when enrichment is on) so the span-enrichment hook can decide whether to record the
// subject.
if (SPAN_ENRICHMENT_ENABLED) {
if (split.serialId != null) {
metadataBuilder.addInteger(METADATA_SPLIT_SERIAL_ID, split.serialId);
}
metadataBuilder.addBoolean(METADATA_DO_LOG, allocation.doLog != null && allocation.doLog);
}
final ProviderEvaluation<T> result =
ProviderEvaluation.<T>builder()
.value(mappedValue)
Expand Down
61 changes: 56 additions & 5 deletions ...e-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/Provider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import dev.openfeature.sdk.exceptions.OpenFeatureError;
import dev.openfeature.sdk.exceptions.ProviderNotReadyError;
import java.lang.reflect.Constructor;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
Expand All @@ -28,13 +29,19 @@ public class Provider extends EventProvider implements Metadata {
private static final Logger log = LoggerFactory.getLogger(Provider.class);
static final String METADATA = "datadog-openfeature-provider";
private static final String EVALUATOR_IMPL = "datadog.trace.api.openfeature.DDEvaluator";

private static final Options DEFAULT_OPTIONS = new Options().initTimeout(30, SECONDS);
private volatile Evaluator evaluator;
private final Options options;
private final AtomicReference<InitializationState> initializationState =
new AtomicReference<>(InitializationState.NOT_STARTED);
private final FlagEvalMetrics flagEvalMetrics;
private final FlagEvalHook flagEvalHook;
// Span enrichment: null unless the gate is on, so the feature has no idle overhead when off.
private final SpanEnrichmentHook spanEnrichmentHook;
// Precomputed hook list returned by getProviderHooks() on every evaluation. Immutable and built
// once so gate-off evaluation allocates nothing on this hot path.
private final List<Hook> providerHooks;

public Provider() {
this(DEFAULT_OPTIONS, null);
Expand All @@ -45,6 +52,17 @@ public Provider(final Options options) {
}

Provider(final Options options, final Evaluator evaluator) {
this(options, evaluator, null);
}

/**
* @param spanEnrichmentEnabledOverride when non-null, forces the span-enrichment gate (test
* seam); when null, the gate is read via {@link SpanEnrichmentGate}.
*/
Provider(
final Options options,
final Evaluator evaluator,
final Boolean spanEnrichmentEnabledOverride) {

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Could you explain more about this override mode? Why we need it?

@pavlokhrebto pavlokhrebto Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a test-only seam: the spanEnrichmentEnabledOverride constructor param lets tests force the gate on/off deterministically without mutating global config or system properties (when null, the gate is read from config as normal). Not a user-facing feature.

this.options = options;
this.evaluator = evaluator;
FlagEvalMetrics metrics = null;
Expand All @@ -58,6 +76,34 @@ public Provider(final Options options) {
}
this.flagEvalMetrics = metrics;
this.flagEvalHook = hook;

// Span enrichment is wired ONLY when the gate is on — off means no capture hook and no idle
// per-evaluation overhead.
final boolean spanEnrichmentEnabled =
spanEnrichmentEnabledOverride != null
? spanEnrichmentEnabledOverride
: SpanEnrichmentGate.isEnabled();
this.spanEnrichmentHook = spanEnrichmentEnabled ? new SpanEnrichmentHook() : null;

// Precompute the immutable hook list once so getProviderHooks() (called on every evaluation)
// allocates nothing, including when the gate is off.
final List<Hook> hooks = new ArrayList<>(2);
if (flagEvalHook != null) {
hooks.add(flagEvalHook);
}
if (spanEnrichmentHook != null) {
hooks.add(spanEnrichmentHook);
}
this.providerHooks =
hooks.isEmpty() ? Collections.emptyList() : Collections.unmodifiableList(hooks);

// Announce the span-enrichment state at startup (matches the reference implementation).
// "enabled" only when the gate is on (the capture hook was constructed), otherwise "disabled".
if (spanEnrichmentHook != null) {
log.info("{} span enrichment enabled", METADATA);
} else {
log.info("{} span enrichment disabled", METADATA);
}
}

@Override
Expand Down Expand Up @@ -167,22 +213,27 @@ private Evaluator buildEvaluator() throws Exception {

@Override
public List<Hook> getProviderHooks() {
if (flagEvalHook == null) {
return Collections.emptyList();
}
return Collections.singletonList(flagEvalHook);
return providerHooks;
}

@Override
public void shutdown() {
if (flagEvalMetrics != null) {
flagEvalMetrics.shutdown();
}
// Span enrichment needs no provider-close cleanup here: the capture hook holds no tracer state.
// The agent-side write tier owns the interceptor and per-trace state and is torn down with the
// feature-flagging subsystem, not per provider.
if (evaluator != null) {
evaluator.shutdown();
}
}

// Visible for tests: expose whether span enrichment is wired (gate-on) without leaking the impl.
SpanEnrichmentHook spanEnrichmentHook() {
return spanEnrichmentHook;
}

@Override
public Metadata getMetadata() {
return this;
Expand Down Expand Up @@ -223,7 +274,7 @@ public ProviderEvaluation<Value> getObjectEvaluation(
return evaluator.evaluate(Value.class, key, defaultValue, ctx);
}

@SuppressForbidden // Class#forName(String) used to lazy load internal-api dependencies
@SuppressForbidden // Class#forName(String) used to lazy-load the evaluator implementation
protected Class<?> loadEvaluatorClass() throws ClassNotFoundException {
return Class.forName(EVALUATOR_IMPL);
}
Expand Down
25 changes: 25 additions & 0 deletions .../feature-flagging-api/src/main/java/datadog/trace/api/openfeature/SpanEnrichmentGate.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package datadog.trace.api.openfeature;

import datadog.trace.api.featureflag.config.FeatureFlaggingConfig;
import datadog.trace.bootstrap.config.provider.ConfigProvider;

/**
* Single source for reading the experimental span-enrichment gate, with full {@link ConfigProvider}
* precedence (system property > stable config > env var {@code
* DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED}). OFF by default; distinct from the
* provider-enabled gate. Shared so {@link Provider} (per construction) and {@link DDEvaluator}
* (once at class load) read it the same way.
*/
final class SpanEnrichmentGate {

private SpanEnrichmentGate() {}

static boolean isEnabled() {
try {
return ConfigProvider.getInstance()
.getBoolean(FeatureFlaggingConfig.EXPERIMENTAL_SPAN_ENRICHMENT_ENABLED, false);
} catch (final Throwable t) {
return false; // never let config reading break construction
}
}
}
Loading