Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiMapValue can not be serialized #13559

Closed
rvega-arg opened this issue Aug 10, 2018 · 4 comments
Closed

MultiMapValue can not be serialized #13559

rvega-arg opened this issue Aug 10, 2018 · 4 comments

Comments

@rvega-arg
Copy link

@rvega-arg rvega-arg commented Aug 10, 2018

I'm using vertx with the latest version of hazelcast 3.10.4. Vertx use hazelcast as default implementation for its cluster manager service. However i am getting a few errors when running vertx+hazelcast in a distributed cluster.

After a couple of hours of research i found that vertx cluster goes into an inconsistent state because it use a MultiMap to hold some internal information. However when split brain happens and this multimap should be merged, hazelcast fail with

SEVERE: [10.0.14.7]:5701 [dev] [3.10.4] Failed to serialize 'com.hazelcast.multimap.impl.MultiMapValue'
com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.multimap.impl.MultiMapValue'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:75)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:106)
	at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:317)
	at com.hazelcast.multimap.impl.MultiMapEventsPublisher.publishEntryEvent(MultiMapEventsPublisher.java:60)
	at com.hazelcast.multimap.impl.MultiMapService.publishEntryEvent(MultiMapService.java:241)
	at com.hazelcast.multimap.impl.operations.AbstractMultiMapOperation.publishEvent(AbstractMultiMapOperation.java:71)
	at com.hazelcast.multimap.impl.operations.MergeOperation.run(MergeOperation.java:78)
	at com.hazelcast.spi.Operation.call(Operation.java:148)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:202)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:191)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:120)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:100)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable serializer for class com.hazelcast.multimap.impl.MultiMapValue
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.serializerFor(AbstractSerializationService.java:487)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:146)
	... 13 more

I think MultiMapValue should implement IdentifiedDataSerializable. I made a quick implementation and it works, but i 'm not sure if my code is good enough

    @Override
    public int getFactoryId() {
        return MultiMapDataSerializerHook.F_ID;
    }

    @Override
    public int getId() {
        return MultiMapDataSerializerHook.MULTIMAP_VALUE;
    }

    @Override
    public void writeData(ObjectDataOutput out) throws IOException {
        out.writeLong(hits);

        for (MultiMapRecord record : collection)
            out.writeObject(record);
    }

    @Override
    public void readData(ObjectDataInput in) throws IOException {
        this.hits = in.readLong();
        for (int i = 0; i < this.hits; i++)
            this.collection.add(in.readObject(MultiMapRecord.class));
    }
@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Sep 6, 2018

@rvega-arg thanks for the great report! let's see what we can do about it.

@jerrinot jerrinot self-assigned this Sep 6, 2018
jerrinot added a commit to jerrinot/hazelcast that referenced this issue Sep 6, 2018
Fixes hazelcast#13559

Multimap does not have MERGE listener anyway hence it's pointless
to fire MERGE events.

I also added additional parameters to the MultiMap split-brain test
to test every merge policy for both BINARY and OBJECT formats.
@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Sep 6, 2018

@rvega-arg: once again thanks for your bug report!

It turns out the event should not be published at all hence no need to have the value serializable.

Before we release the fix you can disable multimap merging after split-brain: Configure DiscardMergePolicy in MultiMap configuration. If you are using XML configuration then this is the line: <merge-policy>DiscardMergePolicy</merge-policy> This will revert the split-brain healing behaviour as it was in Hazelcast 3.9 and older versions: When 2 clusters are merging together then it will discard entries from the smaller cluster.

@rvega-arg
Copy link
Author

@rvega-arg rvega-arg commented Sep 7, 2018

@jerrinot i couldn't found the gitlab issue but at some point i remember that hazelcast is now supporting MultiMap merge after split brain. This change is breaking that feature? Maybe i miss understood something.

Thanks!

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Sep 7, 2018

@rvega-arg: yes, you are right. Hazelcast 3.10 introduced merging policies for many structures. A merge policy is a function to merge entries from 2 clusters when a partitioned cluster is healed.

If you configure the DiscardMergePolicy policy then MultiMap entries wont be merged when a split-bran is healed. Entries from the smaller cluster will be discarded and only entries from the bigger cluster will survive. That's the same behaviour as in Hazelcast 3.9.x and olders. It's a workaround before we release a fixed version.

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Sep 10, 2018
Fixes hazelcast#13559

Multimap does not have MERGE listener anyway hence it's pointless
to fire MERGE events.

I also added additional parameters to the MultiMap split-brain test
to test every merge policy for both BINARY and OBJECT formats.
jerrinot added a commit to jerrinot/hazelcast that referenced this issue Sep 10, 2018
Fixes hazelcast#13559

Multimap does not have MERGE listener anyway hence it's pointless
to fire MERGE events.

I also added additional parameters to the MultiMap split-brain test
to test every merge policy for both BINARY and OBJECT formats.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.