Flaky test report: committed-code failures on 2026-03-24 #218
Description
Committed-code failures on 2026-03-24
The following tests failed in gradle-check builds that ran against committed code (Timer runs on main or Post Merge Actions) within the past 24 hours. Historical failure data across all build types (including PR builds) is included to assess flake rates.
1. MixedClusterClientYamlTestSuiteIT — 310_match_bool_prefix/multi_match multiple fields (complete term & partial term)
Recent build: #73223
First seen: 2024-03-25
Total unique builds affected: ~115 (partial term: 114, complete term: 112; these always fail together)
Pattern: This is a long-standing flaky test dating back to March 2024. It had a massive spike in Sep 2024 (~63 builds), then subsided. It recurs at a low but steady rate of 1–5 builds/month through 2025, with occasional spikes (Jul 2025: 10 builds, Jan 2026: 8 builds). The same underlying YAML test also fails in ClientYamlTestSuiteIT (~64 builds each variant) and occasionally in SmokeTestMultiNodeClientYamlTestSuiteIT (~4 builds).
Trend: Stable — persistent low-level flake with periodic spikes. No sign of improvement.
2. AzureBlobStoreRepositoryTests.testWriteRead
Recent build: #73222
First seen: 2024-04-29
Total unique builds affected: 75
Pattern: Consistent flake since April 2024. Fails in 1–9 builds per month with no month completely free of failures since it started. Recent months show a slight uptick: Nov 2025 (7), Dec 2025 (7), Jan 2026 (5), Feb 2026 (9), Mar 2026 (5).
Trend: Worsening — failure rate has increased in recent months compared to mid-2025.
3. MixedClusterClientYamlTestSuiteIT — 110_strict_allow_templates
Recent build: #73215
First seen: 2024-06-26
Total unique builds affected: 48 (MixedCluster) + 58 (ClientYamlTestSuiteIT) + 3 (SmokeTest)
Pattern: Had a large spike in Sep 2024 (39 builds in MixedCluster alone), then went quiet for months. Resurfaced in Apr 2025 and has been sporadically appearing since. The ClientYamlTestSuiteIT variant is more consistently flaky with a big spike in Jan 2026 (13 builds).
Trend: Stable — intermittent flake with long quiet periods punctuated by bursts.
4. ClusterDisruptionIT.testAckedIndexing
Recent build: #73190
First seen: 2024-04-05
Total unique builds affected: 34
Pattern: Low-frequency flake that has persisted for nearly 2 years. Typically 1–3 builds/month, with occasional spikes (Sep 2025: 5, Dec 2025: 5, Apr 2025: 4). Never fully goes away.
Trend: Stable — consistent low-level flake with no improvement over time.
5. Netty4Http3IT.testThatNettyHttpServerSupportsHttp2OrHttp3Post
Recent build: #73181
First seen: 2026-01-26
Total unique builds affected: 23
Pattern: Relatively new test failure. Appeared in late January 2026 and has been accelerating: Jan (4 builds), Feb (10 builds), Mar (9 builds so far).
Trend: Worsening — rapidly increasing failure rate since introduction. This is a new flake that needs attention.
6. SegmentReplicationWithNodeToNodeIndexShardTests.classMethod
Recent build: #73191
First seen: 2024-08-21
Total unique builds affected: 10
Pattern: Rare but bursty. First appeared Aug 2024 (1 build), then quiet until Feb 2025 (1 build). Had a burst in Jun 2025 (4 builds), went quiet again, then reappeared in Mar 2026 (4 builds). The high doc_count relative to unique builds (58 docs / 10 builds) suggests multiple test methods fail together in the same build.
Trend: Worsening — Mar 2026 shows renewed activity after months of quiet.
7. WarmIndexSegmentReplicationIT.testNodeDropWithOngoingReplication
Recent build: #73194
First seen: 2025-03-17
Total unique builds affected: 10
Pattern: Low-frequency flake appearing roughly once every 1–2 months since March 2025. Typically 1 build per month when it appears, with a small spike in Oct 2025 (2) and Dec 2025 (2).
Trend: Stable — consistent low-frequency flake.
8. NodeJoinLeftIT.testClusterStabilityWhenDisconnectDuringSlowNodeLeftTask
Recent build: #73232
First seen: 2025-06-09
Total unique builds affected: 8
Pattern: Intermittent flake since June 2025. Appears in 1–2 builds per month when it surfaces, with some quiet months in between (Jul, Sep, Dec 2025 through Feb 2026 all zero).
Trend: Stable — low-frequency intermittent flake.
9. SegmentReplicationWithNodeToNodeIndexShardTests.testPrimaryPromotionWithConcurrentTranslogRecovery
Recent build: #73191
First seen: 2026-01-21
Total unique builds affected: 6
Pattern: New flake. First appeared Jan 2026 (1 build), then 5 builds in Mar 2026 so far.
Trend: Worsening — sharp increase in March 2026. Likely related to the classMethod failures in the same test class.
10. IndicesRequestCacheCleanupIT.testCacheWithInvalidation
Recent build: #73168
First seen: 2025-04-04
Total unique builds affected: 4
Pattern: Very rare flake. Only 4 occurrences in nearly a year: Apr 2025 (2), Aug 2025 (1), Mar 2026 (1).
Trend: Stable — very low frequency, not a significant concern.
Summary Table
| Test | Recent Build | First Seen | Unique Builds | Trend |
|---|---|---|---|---|
MixedCluster..310_match_bool_prefix (partial term) |
#73223 | 2024-03-25 | 114 | Stable |
MixedCluster..310_match_bool_prefix (complete term) |
#73223 | 2024-03-25 | 112 | Stable |
AzureBlobStoreRepositoryTests.testWriteRead |
#73222 | 2024-04-29 | 75 | |
MixedCluster..110_strict_allow_templates |
#73215 | 2024-06-26 | 48 | Stable |
ClusterDisruptionIT.testAckedIndexing |
#73190 | 2024-04-05 | 34 | Stable |
Netty4Http3IT..Http2OrHttp3Post |
#73181 | 2026-01-26 | 23 | |
SegRepWithNodeToNode..classMethod |
#73191 | 2024-08-21 | 10 | |
WarmIndexSegRepIT.testNodeDropWithOngoingReplication |
#73194 | 2025-03-17 | 10 | Stable |
NodeJoinLeftIT..SlowNodeLeftTask |
#73232 | 2025-06-09 | 8 | Stable |
SegRepWithNodeToNode..testPrimaryPromotion |
#73191 | 2026-01-21 | 6 | |
IndicesRequestCacheCleanupIT.testCacheWithInvalidation |
#73168 | 2025-04-04 | 4 | Stable |
Key takeaways:
- 3 tests are worsening:
Netty4Http3IT,AzureBlobStoreRepositoryTests.testWriteRead, and twoSegmentReplicationWithNodeToNodeIndexShardTestsmethods - The
310_match_bool_prefixYAML test is the most prolific flake by volume, affecting 100+ builds across multiple test suites over 2 years Netty4Http3ITis the fastest-growing new flake, averaging ~10 builds/month since appearing in Jan 2026