-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix flakyness of replace node e2e #1126
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would the readiness check @rzetelskik is building help here?
AFAIR it allows for one DN, we have 2 nodes in this e2e, so it wouldn't help. |
Yes, it wouldn't help in this case. After it's been merged you could just increase the number of nodes in this test to 3 though. |
7ffac2c
to
3e8f6d9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/approve
Full quorum check is sensitive to seeing nodes not being part of the cluster, when such node is spotted it immediately returns an error. In replace node e2e, it could happen that cluster become ready before replaced node was acknowledged by the others. In this case, full quorum check returned an error, because it saw a IP not bound to any cluster Service. To fix it, test waits until cluster is rolled out, and replaced node is seen by the other one, before the full quorum check is validated.
3e8f6d9
to
dfea5a7
Compare
Full quorum check is sensitive to seeing nodes not being part of the cluster, when such node is spotted it immediately returns an error.
In replace node e2e, it could happen that cluster become ready before replaced node was acknowledged by the others.
In this case, full quorum check returned an error, because it saw a IP not bound to any cluster Service.
To fix it, test waits until cluster is rolled out, and replaced node is seen by the other one, before the full quorum check is validated.