Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Member attributes goes missing after merging from split brain scenario #7697

Closed
hemikak opened this issue Mar 9, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@hemikak
Copy link

commented Mar 9, 2016

Hi All,

I currently have a setup where there are 2 nodes are clustered using hazelcast. Each node has their member attributes added through code. Example :

hazelcastInstance.getCluster().getLocalMember().setStringAttribute("foo", "bar");

When a network disconnection occur, each node will get notified that the other member has left the cluster. This creates a split brain scenario. After a while, once the network is back online, each node will recognize each other and I see the MERGED log as well on a node.

INFO {com.hazelcast.core.LifecycleService} - [10.100.5.197]:4001 [wso2.carbon.domain] [3.5.2] Address[10.100.5.197]:4001 is MERGED

The above log could appear on either of the node(disregarding the IP).

But some of the member attributes that was assigned to members of the cluster has been assigned null. Is this the expected behaviour ?

Is there an event which we can reassigned the member attributes after the merging occur ? . I tried using a LifecycleListener to reassigned the member attributes when the LifecycleState.MERGED event happens. But still the member attributes are null. Is there an event afterwards ?

Regards,

@jerrinot jerrinot added this to the Backlog milestone Mar 9, 2016

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Mar 9, 2016

Hi @hemikak,

thanks for reporting this! This is not an expected behaviour to me. We'll have look.

@hemikak

This comment has been minimized.

Copy link
Author

commented Mar 10, 2016

Hi @jerrinot ,

To elaborate more regarding the issue,

Lets say node "x" has an member attribute value as "a" for key "myKey".

// node "x"
hazelcastInstance.getCluster().getLocalMember().setStringAttribute("myKey", "a");

And node "y" has an member attribute value as "b" for key "myKey".

// node "y"
hazelcastInstance.getCluster().getLocalMember().setStringAttribute("myKey", "b");

After the split brain occurs and then recovers, If I were to loop through all the members within node "x" and get their "myKey" property, I see that the node "myKey" value in node "y" is null. In which the value should actually be "b".

But if I check the "myKey" properties of all the members within node "y", they will have their expected values. No nulls.

Therefore you will have to check this in both nodes to see that a member's property has gone null.

Regards,

@jerrinot jerrinot modified the milestones: 3.8, Backlog Aug 22, 2016

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2016

@hemikak: can you try it with Hazelcast 3.7? I tried it and I can reproduce it. It works for me. It works for me when even I set the attributes at the time when there is an ongoing split-brain situation.

package com.hazelcast.cluster;

import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.Member;
import com.hazelcast.spi.impl.SplitBrainTestSupport;
import com.hazelcast.test.HazelcastParallelClassRunner;
import com.hazelcast.test.annotation.ParallelTest;
import com.hazelcast.test.annotation.QuickTest;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;

import java.util.Set;

import static org.junit.Assert.assertEquals;

@RunWith(HazelcastParallelClassRunner.class)
@Category({QuickTest.class, ParallelTest.class})
public class MemberAttributeSplitBrainTest extends SplitBrainTestSupport {

    @Override
    protected int clusterSize() {
        return 2;
    }

    @Override
    protected void onBeforeSplitBrainCreated() {
//        setMemberAttributeForAllMembers();
    }

    @Override
    protected void onAfterSplitBrainCreated() {
        setMemberAttributeForAllMembers();
    }

    private void setMemberAttributeForAllMembers() {
        HazelcastInstance[] allInstances = getAllInstances();
        for (HazelcastInstance hz : allInstances) {
            setAttribute(hz);
        }
    }

    @Override
    protected void onAfterSplitBrainHealed() {
        HazelcastInstance[] allInstances = getAllInstances();
        for (HazelcastInstance hz : allInstances) {
            Set<Member> members = hz.getCluster().getMembers();
            for (Member member : members) {
                String attributeValue = member.getStringAttribute("foo");
                assertEquals(attributeValue, "bar");
            }
        }
    }

    private void setAttribute(HazelcastInstance isolatedInstance) {
        isolatedInstance.getCluster().getLocalMember().setStringAttribute("foo", "bar");
    }
}
@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2016

I'm closing it now, feel free to re-open it if you still can reproduce the issue.

@jerrinot jerrinot closed this Aug 22, 2016

Andy2003 added a commit to kiwigrid/vert.x that referenced this issue Oct 19, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.