Flaky TestRulerKeepFiring integration test

[yeya24]

Describe the bug
Flaky test run https://github.com/cortexproject/cortex/actions/runs/8619846095/job/23625587487.

Error message:

    ruler_test.go:1135: 
        	Error Trace:	/home/runner/work/cortex/cortex/integration/ruler_test.go:1135
        	Error:      	Should NOT be empty, but was <nil>
        	Test:       	TestRulerKeepFiring

Expected behavior
Test should pass.

[mustafain117]

Another failure: https://github.com/cortexproject/cortex/actions/runs/8758051411/job/24038143808

Trying to reproduce locally

[yeya24]

Probably fixed by #5971.
Let's close this one and reopen if we still see it in CI