Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hazelcast 3.7.*: MigrationListener migrationStarted/migrationCompleted is called for all partitions #9919

Closed
hochraldo opened this issue Feb 15, 2017 · 3 comments
Assignees
Milestone

Comments

@hochraldo
Copy link

@hochraldo hochraldo commented Feb 15, 2017

The change in behavior is maybe easiest to explain with the following code snippet:

import com.hazelcast.config.Config;
import com.hazelcast.config.NetworkConfig;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IMap;
import com.hazelcast.core.MigrationEvent;
import com.hazelcast.core.MigrationListener;
import com.hazelcast.core.Partition;
import com.hazelcast.instance.GroupProperties;
import org.junit.Test;

import java.util.HashSet;
import java.util.Set;

public class HazelcastTest {

    @Test
    public void migrationListener() throws InterruptedException {
        Config config = new Config();
        config.setProperty(GroupProperties.PROP_PREFER_IPv4_STACK, "" + true);
        config.setProperty(GroupProperties.PROP_PARTITION_COUNT, "" + 10);

        NetworkConfig netConfig = new NetworkConfig();
        Set<String> interfaces = new HashSet<>();
        interfaces.add("127.0.0.1");
        netConfig.getJoin().getMulticastConfig().setEnabled(false);
        netConfig.getJoin().getTcpIpConfig().addMember("127.0.0.1").setEnabled(true);
        netConfig.setPort(55555);
        netConfig.getInterfaces().addInterface("127.0.0.1").setEnabled(true);
        config.setNetworkConfig(netConfig);
        config.getGroupConfig().setName("cluster-test" + 55555);
        HazelcastInstance instance1 = Hazelcast.newHazelcastInstance(config);

        IMap<String, String> map1 = instance1.getMap("foo");
        map1.put("bla", "bar");
        map1.put("foo", "bar");
        map1.put("bar", "bar");

        Set<Partition> partitions = instance1.getPartitionService().getPartitions();
        System.out.println("total partitions: " + partitions);

        instance1.getPartitionService().addMigrationListener(new MigrationListener() {
            @Override
            public void migrationStarted(MigrationEvent migrationEvent) {
                System.out.println("migration: " + migrationEvent);
            }

            @Override
            public void migrationCompleted(MigrationEvent migrationEvent) {
                System.out.println("migration: " + migrationEvent);
            }

            @Override
            public void migrationFailed(MigrationEvent migrationEvent) {
                System.out.println("migration: " + migrationEvent);
            }
        });

        HazelcastInstance instance2 = Hazelcast.newHazelcastInstance(config);
        Thread.sleep(500);
    }
}

If above code is run in 3.6.x then 5 partitions are moved from node1 to node2 (because 10 are configured to be in total).

migration: MigrationEvent{partitionId=0, status=STARTED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=1, status=STARTED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=0, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=1, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=2, status=STARTED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=3, status=STARTED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=2, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=3, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=4, status=STARTED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}
migration: MigrationEvent{partitionId=4, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 this, newOwner=Member [127.0.0.1]:55556}

In version 3.7.x or 3.8-RC1 the migration listener is called for 10 partitions and it looks like that the new node is the owner of all partitions.

migration: MigrationEvent{partitionId=0, status=STARTED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=0, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=1, status=STARTED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=1, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=2, status=STARTED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=2, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=3, status=STARTED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=3, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=4, status=STARTED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=4, status=COMPLETED, oldOwner=Member [127.0.0.1]:55555 - 20d799eb-9338-4753-b559-788240b59dc7 this, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=5, status=STARTED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=5, status=COMPLETED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=6, status=STARTED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=6, status=COMPLETED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=7, status=STARTED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=7, status=COMPLETED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=8, status=STARTED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=8, status=COMPLETED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=9, status=STARTED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
migration: MigrationEvent{partitionId=9, status=COMPLETED, oldOwner=null, newOwner=Member [127.0.0.1]:55556 - 368ca374-ed6d-4e53-87e3-1f1360e59f28}
@mdogan mdogan self-assigned this Feb 15, 2017
@mdogan mdogan modified the milestones: 3.9, 3.8.1 Feb 15, 2017
@mdogan mdogan added the Team: Core label Feb 15, 2017
@mdogan
Copy link
Contributor

@mdogan mdogan commented Feb 15, 2017

@hochraldo, thanks for the report. Migration system has changed significantly in 3.7 release. Alongside with primary replicas, backup replicas are managed by migration system by 3.7. But unintentionally migration listener started to receive events for backup migrations too. That's what you observe.

@hochraldo
Copy link
Author

@hochraldo hochraldo commented Feb 16, 2017

@mdogan Thx for your answer. Is it safe to say that if the oldOwner is null that it is an event about a backup migration? Or in other words is it possible that the partition size grows dynamically (in which case no old owner would be available either)?

@metanet
Copy link
Contributor

@metanet metanet commented Feb 16, 2017

@hochraldo
With Mehmet's fix, you will only get partition owner migrations. When a backup is migrated, it will not be published to the MigrationListener. In later releases, we are planning to refactor MigrationListener and provide a better view about migrations.

Regards,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants