[CI] DataStreamsClientYamlTestSuiteIT test {p0=data_stream/120_data_streams_stats/Multiple data stream} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #14067 / part1
• elasticsearch-pull-request #44760 / rest-compatibility

Reproduction Line:

./gradlew ":modules:data-streams:yamlRestTest" --tests "org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT.test {p0=data_stream/120_data_streams_stats/Multiple data stream}" -Dtests.seed=4882C459601E52C4 -Dtests.locale=bem-ZM -Dtests.timezone=America/St_Thomas -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [data_stream/120_data_streams_stats:198]: field [data_streams.1.maximum_timestamp] doesn't match the expected value
Expected: <1.593639450943E12>
     but: was <0.0>

Issue Reasons:

• [main] 2 failures in test test {p0=data_stream/120_data_streams_stats/Multiple data stream} (0.2% fail rate in 803 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test test {p0=data_stream/120_data_streams_stats/Multiple data stream} (0.2% fail rate in 803 executions)

Build Scans:

• elasticsearch-intake #14067 / part1
• elasticsearch-pull-request #44760 / rest-compatibility

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[gmarouli]

Looking at this test, I observe the following:

• We create two data streams
• We index documents with refresh: true so theoretically the data should have been available for searching, meaning also available for stats.

Looking at the code of the DataStreamsStatsTransportAction:

           long maxTimestamp = 0L;
           try (Engine.Searcher searcher = indexShard.acquireSearcher(ReadOnlyEngine.FIELD_RANGE_SEARCH_SOURCE)) {
                IndexReader indexReader = searcher.getIndexReader();
                byte[] maxPackedValue = PointValues.getMaxPackedValue(indexReader, DataStream.TIMESTAMP_FIELD_NAME);
                if (maxPackedValue != null) {
                    maxTimestamp = LongPoint.decodeDimension(maxPackedValue, 0);
                }
            }
            return new DataStreamsStatsAction.DataStreamShardStats(indexShard.routingEntry(), storeStats, maxTimestamp);

So, even if the document is still there, meaning there should also be a non-zero maximum_timestamp, it's possible for the maximum_timestamp to be zero if PointValues.getMaxPackedValue(indexReader, DataStream.TIMESTAMP_FIELD_NAME) returns null.

I will ask the expertise of a search team, maybe they know something about this. If not, I think I will unmute it, this failure has happened only once (the other failure is a suite timeout), so I would like to see if it persists before diving into more.

@dakrone I would classify it as low-risk, but I will let you decide.

[nielsbauman]

See #96711 (comment) for my hypothesis of what is happening here.

In short, all test failures have missing shards in the result:

"_shards" : {
  "total" : 3,
  "successful" : 2,
  "failed" : 0
}

I suspect this to be caused by the rollover and/or docs ingestion after we create the second data stream.