Integrate stored fields format bloom filter with synthetic _id #138515

fcofdez · 2025-11-24T15:25:57Z

This PR integrates ES93BloomFilterStoredFieldsFormat with the synthetic _id lookups. For that, it introduces a new set of Codecs meant to be used only by TIME_SERIES indices. These new codecs are necessary to cover the case when the codec is loaded through SPI (i.e. after a shard relocation or node restarts).

The new codecs just wrap the existing codecs and extend them with the necessary plumbing to populate the bloom filter during indexing.

elasticsearchmachine · 2025-11-24T15:26:45Z

Hi @fcofdez, I've created a changelog YAML for you.

tlrx

I need to take a deeper look but overall approach looks good.

...er/src/main/java/org/elasticsearch/index/codec/ES93TSDBDefaultCompressionLucene103Codec.java

server/src/main/java/org/elasticsearch/index/codec/ES93TSDBLuceneDefaultCodec.java

tlrx · 2025-11-24T15:34:31Z

server/src/main/java/org/elasticsearch/index/IndexSettings.java

        Property.Final
    );

+    public static final boolean USE_STORED_FIELDS_BLOOM_FILTER_FOR_ID_FEATURE_FLAG = new FeatureFlag("stored_field_bloom_filter")


Do you think we need another feature flag, or could it be folded with the existing one for synthetic id?

I'm not sure it makes a lot of sense to test one without the other, but maybe I'm missing a point.

My idea is that the bloom filter is an optimization on top of the synthetic id. But happy to get rid of the feature flag and the index.mapping.use_stored_field_bloom_filter_id index setting if we think that's redundant. It'll simplify the code a bit.

My idea is that the bloom filter is an optimization on top of the synthetic id

I agree but I think we won't use synthetic ids without a bloom filter on top of it, and having two features flags complicate the code. If that's OK, I would prefer use only one feature flag for both.

I won't block the PR for this so if you want to keep it that's OK too.

I got rid of the setting and feature flag in 799fb3a

tlrx · 2025-11-24T15:44:32Z

server/src/main/java/org/elasticsearch/index/codec/CodecService.java


    }
+
+    private enum StorageMode {


I find this storage mode a bit confusing. Maybe a useBloomFilterSyntheticId local variable would be simpler?

I got rid of it in a72a66a

…-codec

elasticsearchmachine · 2025-11-25T13:08:10Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

fcofdez · 2025-11-25T14:59:38Z

server/src/main/java/org/elasticsearch/index/codec/TSDBCodecWithSyntheticId.java

+    private final TSDBStoredFieldsFormat storedFieldsFormat;
+
+    TSDBCodecWithSyntheticId(String name, Codec delegate, BigArrays bigArrays) {
+        super(name, new TSDBSyntheticIdCodec(delegate));


I'm planning to incorporate the code from TSDBSyntheticIdCodec into this class in a follow-up PR. But I wanted to keep the change size under control.

tlrx

Great work @fcofdez ! I only left minor comments, the direction makes sense to me and we can improve in follow ups. I'd like to have Martijn or Alan review the codec part before merging though.

tlrx · 2025-11-25T14:12:35Z

server/src/main/java/org/elasticsearch/index/codec/CodecService.java

+        boolean useSyntheticId = IndexSettings.TSDB_SYNTHETIC_ID_FEATURE_FLAG
+            && mapperService != null
+            && mapperService.getIndexSettings().useTimeSeriesSyntheticId()
+            && mapperService.getIndexSettings().getMode() == IndexMode.TIME_SERIES;


mapperService.getIndexSettings().useTimeSeriesSyntheticId() already ensure that the index is a time-series index and that the feature flag is enabled.

Simplified in d0d94ef

tlrx · 2025-11-25T14:28:24Z

server/src/main/java/org/elasticsearch/index/codec/TSDBCodecWithSyntheticId.java

+    private final TSDBStoredFieldsFormat storedFieldsFormat;
+
+    TSDBCodecWithSyntheticId(String name, Codec delegate, BigArrays bigArrays) {
+        super(name, new TSDBSyntheticIdCodec(delegate));


We can merge TSDBSyntheticIdCodec and TSDBCodecWithSyntheticId together in a follow up.

Ok, I just saw your #138515 (comment) 👍

tlrx · 2025-11-25T14:40:18Z

...main/java/org/elasticsearch/index/codec/bloomfilter/DelegatingBloomFilterFieldsProducer.java

+        return new FilterLeafReader.FilterTerms(delegate.terms(field)) {
+            @Override
+            public TermsEnum iterator() throws IOException {
+                return new LazyFilterTermsEnum() {
+                    private TermsEnum delegate;
+
+                    @Override
+                    protected TermsEnum getDelegate() throws IOException {
+                        if (delegate == null) {
+                            delegate = in.iterator();
+                        }
+                        return delegate;
+                    }


nit: I've been confused by the two delegate (the on in lazy and the one in the bloom filter) and what in was referencing to.

Maybe something like this would help?

final Terms terms = delegate.terms(field); return new FilterLeafReader.FilterTerms(terms) { @Override public TermsEnum iterator() throws IOException { return new LazyFilterTermsEnum() { private TermsEnum termsEnum; @Override protected TermsEnum getDelegate() throws IOException { if (termsEnum == null) { termsEnum = terms.iterator(); } return termsEnum; } @Override public boolean seekExact(BytesRef text) throws IOException { if (bloomFilter.mayContainTerm(field, text) == false) { return false; } return getDelegate().seekExact(text); } }; } };

Good idea, changed in 92a6daa

server/src/main/java/org/elasticsearch/index/codec/storedfields/TSDBStoredFieldsFormat.java

tlrx · 2025-11-25T14:53:27Z

server/src/main/java/org/elasticsearch/index/codec/CodecService.java

+        var legacyBestSpeedCodec = new LegacyPerFieldMapperCodec(Lucene103Codec.Mode.BEST_SPEED, mapperService, bigArrays);
        if (ZSTD_STORED_FIELDS_FEATURE_FLAG) {
-            codecs.put(DEFAULT_CODEC, new PerFieldMapperCodec(Zstd814StoredFieldsFormat.Mode.BEST_SPEED, mapperService, bigArrays));
+            PerFieldMapperCodec defaultZstdCodec = new PerFieldMapperCodec(


If we want to reduce the scope of this change, we could create our own default_code_with_synthetic_id and hard-coded this in INDEX_CODEC_SETTING for all time-series with use_synthetic_id enabled.

Here we go for the complete solution immediately, for which I'm ok too.

I don't have a strong opinion about this. I'm ok with both approaches. The downside of an extra codec is that we need to maintain it indefinitely whereas with this change as long as the feature flag is off we keep the current behaviour.

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

tlrx · 2025-11-25T15:07:27Z

server/src/main/java/org/elasticsearch/index/codec/storedfields/TSDBStoredFieldsFormat.java

+ * @see StoredFieldsFormat
+ */
+public class TSDBStoredFieldsFormat extends StoredFieldsFormat {
+    private final StoredFieldsFormat storedFieldsFormat;


nit: I would call this delegate

Tackled in e1cbce6

tlrx · 2025-11-25T15:10:21Z

server/src/main/java/org/elasticsearch/index/codec/storedfields/TSDBStoredFieldsFormat.java

+
+        TSDBStoredFieldsWriter(Directory directory, SegmentInfo si, IOContext context) throws IOException {
+            boolean success = false;
+            List<Closeable> toClose = new ArrayList<>();


nit:

Suggested change

List<Closeable> toClose = new ArrayList<>();

List<Closeable> toClose = new ArrayList<>(2);

Tackled in 680edf1

tlrx · 2025-11-25T15:10:45Z

server/src/main/java/org/elasticsearch/index/codec/storedfields/TSDBStoredFieldsFormat.java

+
+        TSDBStoredFieldsReader(Directory directory, SegmentInfo si, FieldInfos fn, IOContext context) throws IOException {
+            boolean success = false;
+            List<Closeable> toClose = new ArrayList<>();


Suggested change

List<Closeable> toClose = new ArrayList<>();

List<Closeable> toClose = new ArrayList<>(2);

Tackled in 680edf1

...a-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/TSDBSyntheticIdsIT.java

tlrx

LGTM, nice work @fcofdez !

Let's wait for @martijnvg approval too before merging.

Integrate stored fields format bloom filter with synthetic _id

429c14a

fcofdez requested review from burqen, martijnvg and tlrx November 24, 2025 15:25

fcofdez added >enhancement Team:StorageEngine :StorageEngine/Codec labels Nov 24, 2025

elasticsearchmachine added the v9.3.0 label Nov 24, 2025

fcofdez mentioned this pull request Nov 24, 2025

Add PerFieldStoredFieldsFormat to allow multiple stored field formats #138299

Closed

Update docs/changelog/138515.yaml

38984b6

This was referenced Nov 24, 2025

Add new codecs to support PerFieldStoredFieldsFormat and bloom filter stored fields #138301

Closed

Integrate bloom filter checks with TSDBSyntheticIdPostingsFormat #138357

Closed

tlrx reviewed Nov 24, 2025

View reviewed changes

fcofdez added 5 commits November 24, 2025 17:25

Wrong version

24fc7e8

Merge branch 'new-codec' of github.com:fcofdez/elasticsearch into new…

60b1eb7

…-codec

Use same name for codecs

7f6d6f3

Get rid of StorageMode

a72a66a

Some renaming

9d0a1f8

fcofdez marked this pull request as ready for review November 25, 2025 13:07

fcofdez requested a review from a team as a code owner November 25, 2025 13:07

fcofdez requested a review from kkrik-es November 25, 2025 13:23

fcofdez added 2 commits November 25, 2025 15:19

Remove setting

799fb3a

Merge remote-tracking branch 'origin/main' into new-codec

089f6f5

fcofdez commented Nov 25, 2025

View reviewed changes

tlrx reviewed Nov 25, 2025

View reviewed changes

fcofdez added 3 commits November 25, 2025 16:31

Improve use syntheticId check

d0d94ef

Add comment about read order

e01019b

Rename to delegate

e1cbce6

fcofdez added 7 commits November 25, 2025 16:36

Initial capacity

680edf1

Add clarifying comment

3e632b8

Add TODO

279cff3

Add javadoc and rename

acc82c1

Improve javadoc

cfec7f9

Clarify delegates

92a6daa

Merge remote-tracking branch 'origin/main' into new-codec

ad711c3

tlrx approved these changes Nov 26, 2025

View reviewed changes

fcofdez mentioned this pull request Nov 26, 2025

Ensure that synthetic _id is usable after restarts/relocations #138678

Open

	List<Closeable> toClose = new ArrayList<>();
	List<Closeable> toClose = new ArrayList<>(2);

Integrate stored fields format bloom filter with synthetic _id #138515

Are you sure you want to change the base?

Integrate stored fields format bloom filter with synthetic _id #138515

Conversation

fcofdez commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 24, 2025

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Nov 25, 2025

Uh oh!

fcofdez Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fcofdez commented Nov 24, 2025 •

edited

Loading

fcofdez Nov 25, 2025 •

edited

Loading