You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AI Tool Usage Notice
If you used an AI tool to help draft this issue,
please make sure you have reviewed and validated all content before submitting.
You are responsible for the accuracy and quality of everything in this report.
Low-quality or unreviewed AI-generated submissions may be closed without further investigation.
See our Generative AI Contribution Policy for details.
Describe the bug
TestCompactor_DeleteLocalSyncFiles (pkg/compactor) is flaky again on arm64, despite the fix in #7567 (which closed #7565). It now times out on the post-fix poll that #7567 added:
// pkg/compactor/compactor_test.go:1855// Wait for at least two completed cycles so we sample after a steady-state// ownership cycle ...cortex_testutil.Poll(t, 30*time.Second, true, func() any {
returnprom_testutil.ToFloat64(c2.CompactionRunsCompleted) >=2&&len(c2.listTenantsWithMetaSyncDirectories()) >0
})
On arm64 the second compactor does not reliably reach 2 completed cycles with at least one owned tenant within 30s, so the poll added to fix the original flake itself times out. The original symptom (#7565: "Should not be zero, but was 0") is gone, but the test is still non-deterministic on slow/loaded arm64 runners.
go test -count=20 -run TestCompactor_DeleteLocalSyncFiles ./pkg/compactor/
Expected behavior
The test passes deterministically on arm64; the second compactor reaches the required steady-state ownership cycles within the poll window (or the wait condition is made robust to arm64 CI timing).
Related: #7565 (original report, closed), #7567 (fix that proved insufficient).
Filed from CI failure-log analysis with AI assistance; the run link and compactor_test.go:1855 were reviewed and verified against master before submitting.
AI Tool Usage Notice
If you used an AI tool to help draft this issue,
please make sure you have reviewed and validated all content before submitting.
You are responsible for the accuracy and quality of everything in this report.
Low-quality or unreviewed AI-generated submissions may be closed without further investigation.
See our Generative AI Contribution Policy for details.
Describe the bug
TestCompactor_DeleteLocalSyncFiles(pkg/compactor) is flaky again on arm64, despite the fix in #7567 (which closed #7565). It now times out on the post-fix poll that #7567 added:The poll added by #7567:
On arm64 the second compactor does not reliably reach 2 completed cycles with at least one owned tenant within 30s, so the poll added to fix the original flake itself times out. The original symptom (#7565: "Should not be zero, but was 0") is gone, but the test is still non-deterministic on slow/loaded arm64 runners.
To Reproduce
Steps to reproduce the behavior:
40a27ador later — i.e. with fix(compactor): fix flaky TestCompactor_DeleteLocalSyncFiles on arm64 #7567 merged)Expected behavior
The test passes deterministically on arm64; the second compactor reaches the required steady-state ownership cycles within the poll window (or the wait condition is made robust to arm64 CI timing).
Environment:
ubuntu-24.04-arm(arm64),test-no-racejobAdditional Context
Observed on
masterCI run (2026-06-08), i.e. after #7567 merged: https://github.com/cortexproject/cortex/actions/runs/27123579544 (jobtest-no-race (arm64)).Related: #7565 (original report, closed), #7567 (fix that proved insufficient).
Filed from CI failure-log analysis with AI assistance; the run link and
compactor_test.go:1855were reviewed and verified againstmasterbefore submitting.