cpg: Inform clients about left nodes during pause #347
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Patch tries to fix incorrect behaviour during following test-case: - 3 nodes - Node 1 is paused - Node 2 and 3 detects node 1 as failed and informs CPG clients - Node 1 is unpaused - Node 1 clients are informed about new membership, but not about Node 1 being paused, so from Node 1 point-of-view, Node 2 and 3 failure Solution is to: - Remove downlist master choose and always choose local node downlist. For Node 1 in example above, downlist contains Node 2 and 3. - Keep code which informs clients about left nodes - Use joinlist as a authoritative source of nodes/clients which exists in membership This patch doesn't break backwards compatibility. I've walked thru all the patches which changed behavior of cpg to ensure patch does not break CPG behavior. Most important were: - 058f503 - Base. Code was significantly changed to handle double free by split group_info into two structures cpg_pd (local node clients) and process_info (all clients). Joinlist was - 97c28ea - This patch removed confchg_fn and made CPG sync correct - feff0e8 - I've tested described behavior without any issues - 6bbbfcb - Added idea of using heuristics to choose same downlist on all nodes. Sadly this idea was beginning of the problems described in 040fda8, ac1d79e, 559d408, 02c5dff, 64d0e5a and b55f32f - 02c5dff - Made joinlist as authoritative source of nodes/clients but left downlist_master_choose as a source of information about left nodes Long story made short. This patch basically reverts idea of using heuristics to choose same downlist on all nodes. Signed-off-by: Jan Friesse <jfriesse@redhat.com>
chrissie-c
approved these changes
Apr 27, 2018
|
@chrissie-c Thank you a lot for the review (I know how hard this one was)! I've merged patch into needle as 9c2a97f and ported also to master as 23e1795. Honza |
ProxBot
pushed a commit
to proxmox/corosync-pve
that referenced
this pull request
Sep 8, 2020
The cpg config change callback where not made correctly for nodes with a low member id (lowest id == master node) after corosync on said node hung, due to IO, artifical suspend or other scheduling related hangs. This is releated to an heuristic for choosing/syncing the CPG member downlist added in 6bbbfcb6b4af72cf35ab9fdb4412fa6c6bdacc12 (corosync repository). See whole issue thread: https://lists.clusterlabs.org/pipermail/users/2018-March/014594.html Upstream pull-request: corosync/corosync#347
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patch tries to fix incorrect behaviour during following test-case:
being paused, so from Node 1 point-of-view, Node 2 and 3 failure
Solution is to:
For Node 1 in example above, downlist contains Node 2 and 3.
in membership
This patch doesn't break backwards compatibility.
I've walked thru all the patches which changed behavior of cpg to ensure
patch does not break CPG behavior. Most important were:
changed to handle double free by split group_info into two structures
cpg_pd (local node clients) and process_info (all clients). Joinlist
was
confchg_fn and made CPG sync correct
behavior without any issues
heuristics to choose same downlist on all nodes. Sadly this idea
was beginning of the problems described in
040fda8,
ac1d79e,
559d408,
02c5dff,
64d0e5a and
b55f32f
authoritative source of nodes/clients but left downlist_master_choose
as a source of information about left nodes
Long story made short. This patch basically reverts
idea of using heuristics to choose same downlist on all nodes.
Signed-off-by: Jan Friesse jfriesse@redhat.com