Skip to content

Add a two-tier presence bloom to the dense tag store (phase 2)#12046

Draft
dougqh wants to merge 1 commit into
dougqh/dense-store-v2from
dougqh/bloom-v2
Draft

Add a two-tier presence bloom to the dense tag store (phase 2)#12046
dougqh wants to merge 1 commit into
dougqh/dense-store-v2from
dougqh/bloom-v2

Conversation

@dougqh

@dougqh dougqh commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What

Adds a per-map bloom presence filter (knownBloom, a single long) to the dense tag store so a "definitely absent" known tag is answered without scanning knownIds. Each dense write ORs in knownBloomBit(tagId); reads (knownRawValue, removeKnown, and the read-through shadow check parentDenseVisible) test the bit first and skip knownIndexOf on a miss.

  • Superset semantics: a set bit means maybe present (scan is authoritative); a clear bit means definitely absent (no scan). Never a false negative.
  • knownBloom is intentionally not cleared on removal — a stale-set bit only costs a scan, whereas clearing could drop a bit still shared (via collision) by a present id, which would be a false negative.
  • Read-through stays chain-aware: parentDenseVisible prunes each ancestor level's scan with that level's own bloom.

Why

The dense store's knownIndexOf is a linear scan. The bloom prune brings dense presence checks to HashMap-insertion parity at ~7 tags while keeping the allocation win — see the tag-id/bloom session notes. Alloc unchanged; this is the CPU-side complement to dense.

Stack

tagset → dense-store (#12045) → **bloom** → generator. Base is dougqh/dense-store-v2. Supersedes the bloom portion of the old reconcile stack (#11900).

Test

TagMapDenseForkedTest, TagMapDenseFuzzForkedTest, KnownTagsTest green; spotbugsMain + spotlessJavaCheck clean.

🤖 Generated with Claude Code

@dougqh dougqh added comp: core Tracer core tag: no release notes Changes to exclude from release notes type: refactoring tag: ai generated Largely based on code generated by an AI or LLM labels Jul 23, 2026
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 23, 2026

Copy link
Copy Markdown

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

Wrapped the dense-tag initialization comment in dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java to match the repository formatter.

Commit fix to this PR


View in Datadog | Reviewed commit d6e7088 · Any feedback? Reach out in #deveng-pr-agent

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 23, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 59.77%
Overall Coverage: 57.39% (-0.04%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d6e7088 | Docs | Datadog PR Page | Give us feedback!

Replace the dense store's linear knownIndexOf scan gate with a two-tier
presence filter driven by the tag-id's declaration coordinate: a per-map
knownGroupMask (1L << group-decl) checked first, then a knownFieldMask
(field-decl & 63). Either bit clear proves the tag definitely absent, so an
append is O(1) and a read-through parent whose groups are disjoint from ours
never pays a scan. The scan stays authoritative (superset semantics); the masks
are never cleared on removal (a stale-set bit only costs a scan, whereas
clearing could drop a bit still shared via collision -> false negative).

Splits the tag-id encoding into group-decl (6 bits) + field-decl (10 bits) in
KnownTagCodec (dropping nameHash from known ids) so the coordinate feeds the
filter, and carries an interim hand-maintained coordinate KnownTags table (the
tag-registry generator replaces it with a byte-equal generated one in a
follow-up). Read-through stays chain-aware: parentDenseVisible prunes each
ancestor level's scan with that level's own two-tier filter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/bloom-v2 branch from 85380de to d6e7088 Compare July 23, 2026 01:01
@dougqh dougqh changed the title Add bloom-filter fast-path to the dense tag store Add a two-tier presence bloom to the dense tag store Jul 23, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.09 s 13.93 s [+0.3%; +2.0%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.95 s 13.01 s [-1.3%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 16.78 s 16.52 s [+0.6%; +2.6%] (maybe worse)
startup:petclinic:iast:Agent 16.94 s 17.00 s [-1.1%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 16.71 s 16.83 s [-1.7%; +0.4%] (no difference)
startup:petclinic:sca:Agent 17.04 s 16.88 s [+0.1%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 16.01 s 16.02 s [-1.0%; +0.9%] (no difference)

Commit: d6e7088a · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh changed the title Add a two-tier presence bloom to the dense tag store Add a two-tier presence bloom to the dense tag store (phase 2) Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant