Skip to content

Conversation

@pracucci
Copy link
Contributor

@pracucci pracucci commented Jun 22, 2021

What this PR does:
The TestStoreGateway_InitialSyncWithWaitRingStability is flaky and fails with this error:

=== RUN   TestStoreGateway_InitialSyncWithWaitRingStability/shuffle_sharding_strategy,_20_gateways,_RF_=_3,_SS_=_3_(bucket_index_enabled_=_false)
    gateway_test.go:300: random generator seed: 1624370314782121000
    gateway_test.go:347:
        	Error Trace:	gateway_test.go:347
        	Error:      	Received unexpected error:
        	            	invalid service state: Failed, expected: Running, failure: unable to start store-gateway dependencies: not healthy, 0 terminated, 1 failed: [register instance in the ring: failed to CAS store-gateway]
        	Test:       	TestStoreGateway_InitialSyncWithWaitRingStability/shuffle_sharding_strategy,_20_gateways,_RF_=_3,_SS_=_3_(bucket_index_enabled_=_false)

We've experienced a similar issue CASing the in-memory consul in the test TestShuffleShardWithCaching and, at that time, we solved it increasing retries and adding a delay between retries. Getting back to TestStoreGateway_InitialSyncWithWaitRingStability, I'm able to reproduce the issue locally in master but I can't reproduce it after this PR's change 🤞

Which issue(s) this PR fixes:
Fixes #4290

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci mentioned this pull request Jun 22, 2021
3 tasks
@pracucci pracucci merged commit c1023f3 into cortexproject:master Jun 22, 2021
@pracucci pracucci deleted the fix-flaky-TestStoreGateway_InitialSyncWithWaitRingStability branch June 22, 2021 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky Test: TestStoreGateway_InitialSyncWithWaitRingStability

2 participants