Flaky test report: committed-code failures on 2026-04-24

[andrross]

Summary

Automated scan of committed-code (Timer/Post Merge) test failures from the past 24 hours (2026-04-24), with historical flakiness analysis across all build types.

None of the 10 failures reproduced locally with the original seed, consistent with non-deterministic (flaky) behavior.

Summary Table

#	Test	Builds Affected	First Failure	Trend	Reproduced?
1	IndexingIT.testIndexingWithSegRep	246	2024-03-25	Stable	N/A (BWC rolling-upgrade test, cannot isolate with --tests locally)
2	RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs	189	2024-03-27	Improving	No
3	SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase	168	2024-03-26	Worsening	No
4	SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase	158	2024-03-28	Worsening	No
5	SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase	141	2024-03-26	Worsening	No
6	IndexingPressureRestIT.testIndexingPressureStats	68	2024-07-28	Worsening	No
7	ResponseHeaderPluginIT.testThatSettingHeadersWorks	66	2024-12-10	Stable	No
8	SystemIndexRestIT.testSystemIndexCreatedWithoutAnyTemplates	64	2025-11-18	Stable	No
9	FlightMetricsTests.testComprehensiveMetrics	63	2025-07-25	Worsening	No
10	RareClusterStateIT.testDisassociateNodesWhileShardInit	28	2024-11-04	Worsening	No

Detailed Findings

1. IndexingIT.testIndexingWithSegRep

• Full name: org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep
• Recent build(s): 75028
• Seed: AB11946717F60E1A
• Error: expected:<0> but was:<1>
• Reproduced locally: N/A (BWC rolling-upgrade test, cannot isolate with --tests locally)
• First failure: 2024-03-25
• Total unique builds affected: 246
• Pattern: Stable flake — consistently failing 4-18 builds/month since Mar 2024 with no improvement trend. Peaked at 29 builds in Sep 2024.

2. RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

• Full name: org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs
• Recent build(s): 75028, 75037
• Seed: AB11946717F60E1A
• Error: java.lang.AssertionError
• Reproduced locally: No
• First failure: 2024-03-27
• Total unique builds affected: 189
• Pattern: Long-running flake — failing since Mar 2024. Was worse in mid-2024 (14-19 builds/month), improved to 2-9 builds/month since mid-2025. Slight uptick in Mar-Apr 2026.

3. SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase

• Full name: org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase
• Recent build(s): 75037, 75061
• Seed: 53F4587599DF31C
• Error: 1 channels still being tracked in RestCancellableNodeClient while there should be none
• Reproduced locally: No
• First failure: 2024-03-26
• Total unique builds affected: 168
• Pattern: Long-running flake — consistent since Mar 2024. Major spike to 43 builds in Nov 2025. Currently 12 builds in Apr 2026 (partial month).

4. SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase

• Full name: org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase
• Recent build(s): 75037
• Seed: 53F4587599DF31C
• Error: 1 channels still being tracked in RestCancellableNodeClient while there should be none
• Reproduced locally: No
• First failure: 2024-03-28
• Total unique builds affected: 158
• Pattern: Long-running flake — same RestCancellableNodeClient channel leak pattern. Spike to 44 builds in Nov 2025. Currently 12 builds in Apr 2026.

5. SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase

• Full name: org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase
• Recent build(s): 75037
• Seed: 53F4587599DF31C
• Error: 1 channels still being tracked in RestCancellableNodeClient while there should be none
• Reproduced locally: No
• First failure: 2024-03-26
• Total unique builds affected: 141
• Pattern: Long-running flake — same RestCancellableNodeClient channel leak. Spike to 42 builds in Nov 2025. Currently 11 builds in Apr 2026.

6. IndexingPressureRestIT.testIndexingPressureStats

• Full name: org.opensearch.http.IndexingPressureRestIT.testIndexingPressureStats
• Recent build(s): 75037
• Seed: 53F4587599DF31C
• Error: 1 channels still being tracked in RestCancellableNodeClient while there should be none
• Reproduced locally: No
• First failure: 2024-07-28
• Total unique builds affected: 68
• Pattern: Sporadic until Nov 2025 spike (40 builds). Appears to be collateral damage from the same RestCancellableNodeClient channel leak affecting the smoke-test-http suite.

7. ResponseHeaderPluginIT.testThatSettingHeadersWorks

• Full name: org.opensearch.http.ResponseHeaderPluginIT.testThatSettingHeadersWorks
• Recent build(s): 75037
• Seed: 53F4587599DF31C
• Error: 1 channels still being tracked in RestCancellableNodeClient while there should be none
• Reproduced locally: No
• First failure: 2024-12-10
• Total unique builds affected: 66
• Pattern: Emerged Dec 2024, spiked to 40 builds in Nov 2025. Same RestCancellableNodeClient channel leak. Steady at 2-9 builds/month since.

8. SystemIndexRestIT.testSystemIndexCreatedWithoutAnyTemplates

• Full name: org.opensearch.http.SystemIndexRestIT.testSystemIndexCreatedWithoutAnyTemplates
• Recent build(s): 75037
• Seed: 53F4587599DF31C
• Error: 1 channels still being tracked in RestCancellableNodeClient while there should be none
• Reproduced locally: No
• First failure: 2025-11-18
• Total unique builds affected: 64
• Pattern: Appeared Nov 2025 with 40-build spike. Same RestCancellableNodeClient channel leak. Steady at 2-9 builds/month since.

9. FlightMetricsTests.testComprehensiveMetrics

• Full name: org.opensearch.arrow.flight.stats.FlightMetricsTests.testComprehensiveMetrics
• Recent build(s): 75022
• Seed: DD5058BCFD287C89
• Error: BindTransportException: Failed to bind to address (Address already in use)
• Reproduced locally: No
• First failure: 2025-07-25
• Total unique builds affected: 63
• Pattern: Consistent flake since Jul 2025 — 4-9 builds/month. Port binding conflict (address already in use). Slightly worsening in recent months (9 builds in both Mar and Apr 2026).

10. RareClusterStateIT.testDisassociateNodesWhileShardInit

• Full name: org.opensearch.cluster.coordination.RareClusterStateIT.testDisassociateNodesWhileShardInit
• Recent build(s): 75037, 75045
• Seed: 53F4587599DF31C
• Error: java.lang.AssertionError
• Reproduced locally: No
• First failure: 2024-11-04
• Total unique builds affected: 28
• Pattern: Low-frequency flake until recently. Was 0-2 builds/month through most of 2025. Sharp increase to 10 builds in Apr 2026 (partial month) — significant worsening.

Notable Patterns

1. RestCancellableNodeClient channel leak — 6 of the 10 failures share the same root symptom: 1 channels still being tracked in RestCancellableNodeClient while there should be none. These all come from the qa:smoke-test-http suite and likely share a common underlying cause. The Nov 2025 spike (40+ builds) suggests a regression was introduced around that time.

2. IndexingIT.testIndexingWithSegRep — The most prolific flake (246 builds) has been failing consistently for over 2 years with no improvement.

3. RareClusterStateIT.testDisassociateNodesWhileShardInit — Previously low-frequency, but showing a sharp increase in Apr 2026 (10 builds in partial month vs. 0-4/month historically). May indicate a new regression.

4. FlightMetricsTests.testComprehensiveMetrics — Port binding conflict, not seed-dependent. Infrastructure/environment issue.