Fix the intermittent failure of TestRemovePrimaryValidSecondaries #10581
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anastasiamac
approved these changes
Sep 1, 2019
worker/peergrouper/worker.go
Outdated
| @@ -303,7 +310,11 @@ func (w *pgWorker) loop() error { | |||
| // Update the replica set members occasionally to keep them up to | |||
| // date with the current replica-set member statuses. | |||
| logger.Tracef("succeeded, waking up after: %v", pollInterval) | |||
| if updateChan == nil { | |||
| // If we had previously failed to update to update the replicaset, | |||
There was a problem hiding this comment.
only need one 'to update'
howbazaar
force-pushed
the
peer-grouper-test-fix
branch
from
ae0c34e
to
88a949b
Compare
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There were two different causes of the intermittent failure. Both had to do with the expectations of the worker main loop, and things actually happening differently.
Given that all the inputs to the worker were controlled, we should be able to be deterministic.
The first failure case was when passing through the loop we had a success then a failure to write. This would give two waiters. However if we had failure followed by success, there would be only one waiter. This was fixed by the change in logic that says if there was a pevious failure when we get a success, update the wait time to be the poll interval.
The second failure case was during the initialize function call where the call would return before the time.After channel had been signalled. There was the assumption that init wasn't complete until there had been a poll happen, but it was possible to have this return before the updateChan was signalled. This caused the rest of the test to progress before things were ready.
QA steps
Bug reference
https://bugs.launchpad.net/juju/+bug/1802590