Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Merge after a Split-Brain Network doesn't work #9358
I've experimented a critical problem by using just one map with 10 items and 3 nodes. After isolating 1 node, the merge occurs only with some of the keys, not all of them. To be sure, I've also printed the merge calls by using my own merge policy strategy and for some of the keys, the merge is not called.
After the merge is completed (Lifecycle=MERGED), the content of the map of the joined server is different, verified by printing the values after the merge.
I'm using IMap. This is my config:
<?xml version="1.0" encoding="UTF-8"?> <!-- ~ Copyright (c) 2008-2012, Hazel Bilisim Ltd. All Rights Reserved. ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.3.xsd" xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <group> <name>orientdb</name> <password>orientdb</password> </group> <properties> <property name="hazelcast.mancenter.enabled">false</property> <property name="hazelcast.memcache.enabled">false</property> <property name="hazelcast.rest.enabled">false</property> <property name="hazelcast.wait.seconds.before.join">2</property> <property name="hazelcast.operation.thread.count">1</property> <property name="hazelcast.operation.generic.thread.count">1</property> <property name="hazelcast.client.event.thread.count">1</property> <property name="hazelcast.event.thread.count">1</property> <property name="hazelcast.heartbeat.interval.seconds">5</property> <property name="hazelcast.max.no.heartbeat.seconds">30</property> <property name="hazelcast.icmp.enabled">true</property> <property name="hazelcast.icmp.timeout">5000</property> <property name="hazelcast.icmp.ttl">3</property> </properties> <network> <port auto-increment="false">2434</port> <join> <multicast enabled="false"> <multicast-group>22.214.171.124</multicast-group> <multicast-port>2434</multicast-port> <multicast-timeout-seconds>1</multicast-timeout-seconds> </multicast> <tcp-ip enabled="true" connection-timeout-seconds="3"> <member>127.0.0.1:2435</member> <member>127.0.0.1:2436</member> </tcp-ip> </join> <interfaces enabled="true"> <interface>127.0.0.1</interface> </interfaces> </network> </hazelcast>
And at runtime I apply this:
We have 2 problems here:
Hi guys, testing with a network with 4 nodes, I isolated the network of 2 servers, then remerged only one and it worked. Then I merged the 2nd node and they never merge. This is the log on the merging server:
And this is the log on one of the other 3 servers:
@lvca: it depends on your logging framework. Hazelcast uses JUL by default.
I tend to use log4j or slf4j/logback. You can switch logging via the
You can use this property to change frequency of the split-brain handler.
I think that this issue can be related to the cluster merge mechanism, not the map service. Maybe the last node which could not merge back still has the master of the bigger side in its member list. Because of this reason, it can be ignoring split brain join requests coming from the bigger side.
More logs would be very useful.