Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loosing topic listeners after merging from a split brain scenario #7742

Closed
Asitha opened this issue Mar 14, 2016 · 6 comments

Comments

Projects
None yet
3 participants
@Asitha
Copy link

commented Mar 14, 2016

Test:

  • Create 3 node Hazelcast cluster with topic listeners on each node listening on a topic
  • Stop one instance (use kill -SIGSTOP ) and get the member left notification on other two nodes
  • Continue the stopped process (use kill -SIGCONT ) and the node will get merged in to the main cluster
  • Send a topic message to nodes.

Observation:

  • MERGED node/nodes won't receive any topic messages sent.

ISSUE:

  • Topic listeners on MERGED nodes get removed after the merge event of a split brain scenario.

@jerrinot jerrinot added this to the 3.7 milestone Mar 14, 2016

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2016

@Asitha: Thank you for reporting ths issue.

It appears we do not re-register listeners from the smaller cluster after a merge.
It appears as a bug to me, but there might be a reason for this. It's certainly a surprising behaviour; I'll find out more and will update this issue.

@jerrinot jerrinot modified the milestones: 3.8, 3.7 May 30, 2016

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Aug 23, 2016

Hello @Asitha,

I am trying to reproduce it, but I can't. Can you please describe it with more details? What Hazelcast version you are using? Are all members listening on the same topic? What member is sending the message once split-brain is resolved? Obviously a reproducer would be great. You can use our test support classes to write one. Just add this dependency into your project:

        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <scope>test</scope>
            <version>3.7</version>
            <classifier>tests</classifier>
        </dependency>

and then you can write a test like this:

public class TopicSplitBrainTest extends SplitBrainTestSupport {

    private static final String TOPIC_NAME = "myTopic";

    private Semaphore semaphore;

    @Before
    public void initListeners() {
        semaphore = new Semaphore(0);
    }

    @Override
    protected void onBeforeSplitBrainCreated()  {
        //
    }

    @Override
    protected void onAfterSplitBrainCreated() {
        HazelcastInstance isolatedMember = getAllInstances()[0];
        ITopic<String> topic = isolatedMember.getTopic(TOPIC_NAME);
        topic.addMessageListener(new MyMessageListener<String>(semaphore));
    }

    @Override
    protected void onAfterSplitBrainHealed() {
        getAllInstances()[1].getTopic(TOPIC_NAME).publish("foo");
        semaphore.acquire();
    }

    private static class MyMessageListener<T> implements MessageListener<T> {
        private final Semaphore semaphore;

        public MyMessageListener(Semaphore semaphore) {
            this.semaphore = semaphore;
        }

        @Override
        public void onMessage(Message<T> message) {
            System.out.println("Received a message");
            semaphore.release();
        }
    }
}

It will create a cluster with 3 members by default then it will isolate the very first member and then it will heal the cluster and you can add a callback method. This should make a reproducer easier.

@Asitha

This comment has been minimized.

Copy link
Author

commented Aug 24, 2016

Hi @jerrinot

I'm using Hazelcast version 3.5.0. I'll get back to you with the details to reproduce

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Dec 15, 2016

hi @Asitha: Did you have a chance to have another look?

@mdogan

This comment has been minimized.

Copy link
Member

commented Jan 4, 2017

A very simple test that shows topic message listener gets notified after split merge;

import com.hazelcast.test.AssertTask;
import com.hazelcast.test.SplitBrainTestSupport;

import java.util.concurrent.atomic.AtomicInteger;

import static org.junit.Assert.assertEquals;

public class TopicSplitBrainTest extends SplitBrainTestSupport {

    private static final String NAME = "topic";
    private static final int MESSAGE_COUNT = 1000;

    private AtomicInteger[] counters;

    @Override
    protected void onBeforeSplitBrainCreated(HazelcastInstance[] instances) throws Exception {
        counters = new AtomicInteger[instances.length];
        for (int i = 0; i < instances.length; i++) {
            counters[i] = new AtomicInteger();
            instances[i].getTopic(NAME).addMessageListener(new MyMessageListener(counters[i]));
        }
    }

    @Override
    protected void onAfterSplitBrainHealed(HazelcastInstance[] instances) throws Exception {
        ITopic<Object> topic = instances[0].getTopic(NAME);
        for (int i = 0; i < MESSAGE_COUNT; i++) {
            topic.publish("message" + i);
        }

        assertTrueEventually(new AssertTask() {
            @Override
            public void run() throws Exception {
                for (AtomicInteger counter : counters) {
                    assertEquals(MESSAGE_COUNT, counter.get());
                }
            }
        });
    }

    private static class MyMessageListener implements MessageListener<Object> {
        private final AtomicInteger counter;

        MyMessageListener(AtomicInteger counter) {
            this.counter = counter;
        }

        @Override
        public void onMessage(Message<Object> message) {
            counter.incrementAndGet();
        }
    }
}

Message listeners are registered before split and messages are published after merge.

@jerrinot

This comment has been minimized.

Copy link
Contributor

commented Jan 6, 2017

I'm closing the issue. @Asitha: feel free to re-open in if it's still not working for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.