Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

serialize-messages =on doesn't work with DistributedPubSub #4120

Open
MaximG1234 opened this issue Dec 22, 2019 · 5 comments · May be fixed by #4271
Open

serialize-messages =on doesn't work with DistributedPubSub #4120

MaximG1234 opened this issue Dec 22, 2019 · 5 comments · May be fixed by #4271
Labels
akka-cluster-tools confirmed bug serialization up for grabs

Comments

@MaximG1234
Copy link

Akka.NET 1.4.0-beta3

It appears that enabling 'serialize-messages = on' while using clustering breaks the actor system with the following error.

Looks like clustering has some difficulty correctly serializing its messages when turned on.

Here is a project which reproduces the error https://www.dropbox.com/s/o9al6adwl609p7o/AkkaTest.zip?dl=0


[ERROR][22/12/2019 12:32:41 AM][Thread 0005][[akka://PubSubSystem/system/cluster/core#342317816]] Swallowing exception during message send
Cause: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type System.Collections.Immutable.ImmutableHashSet`1[System.Type]. Path 'To.$values', line 1, position 376.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewList(JsonReader reader, JsonArrayContract contract, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Akka.Serialization.NewtonSoftJsonSerializer.FromBinary(Byte[] bytes, Type type)
   at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)
[DEBUG][22/12/2019 12:32:41 AM][Thread 0005][akka://PubSubSystem/system/clusterEventBusListener] Started (Akka.Cluster.ClusterReadView+EventBusListener)
[ERROR][22/12/2019 12:32:41 AM][Thread 0019][[akka://PubSubSystem/system/cluster/core#342317816]] Swallowing exception during message send
Cause: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type System.Collections.Immutable.ImmutableHashSet`1[System.Type]. Path 'To.$values', line 1, position 368.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewList(JsonReader reader, JsonArrayContract contract, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Akka.Serialization.NewtonSoftJsonSerializer.FromBinary(Byte[] bytes, Type type)
   at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)
[DEBUG][22/12/2019 12:32:41 AM][Thread 0019][akka://PubSubSystem/system/remote-watcher] Started (Akka.Cluster.ClusterRemoteWatcher)
[DEBUG][22/12/2019 12:32:41 AM][Thread 0009][EventStream] subscribing [akka://PubSubSystem/system/cluster/core/daemon#73339279] to channel Akka.Remote.QuarantinedEvent
[ERROR][22/12/2019 12:32:41 AM][Thread 0007][[akka://PubSubSystem/system/cluster/core#342317816]] Swallowing exception during message send
Cause: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type System.Collections.Immutable.ImmutableHashSet`1[System.Type]. Path 'To.$values', line 1, position 379.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewList(JsonReader reader, JsonArrayContract contract, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Akka.Serialization.NewtonSoftJsonSerializer.FromBinary(Byte[] bytes, Type type)
   at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)
@MaximG1234
Copy link
Author

MaximG1234 commented Dec 22, 2019 •
edited

Can confirm that the cluster system does not correctly form and end up getting the following at the publisher. If I turn serialization back off, everything works as expected.

[INFO][22/12/2019 12:44:39 AM][Thread 0005][Cluster (akka://PubSubSystem)] Cluster Node [akka.tcp://PubSubSystem@192.168.0.29:123] - Received InitJoin message from [[akka.tcp://PubSubSystem@0.0.0.0:3212/system/cluster/core/daemon/joinSeedNodeProcess-1#1491684597]], but this node is not initialized yet
[INFO][22/12/2019 12:44:44 AM][Thread 0005][Cluster (akka://PubSubSystem)] Cluster Node [akka.tcp://PubSubSystem@192.168.0.29:123] - Received InitJoin message from [[akka.tcp://PubSubSystem@0.0.0.0:3212/system/cluster/core/daemon/joinSeedNodeProcess-1#1491684597]], but this node is not initialized yet

@Aaronontheweb
Copy link
Member

@MaximG1234 thanks for reporting this - thought we had fixed it already in #3725 but that must not be the case.

@Aaronontheweb Aaronontheweb added this to the 1.4.0 milestone Dec 27, 2019
@Aaronontheweb
Copy link
Member

#3791 might contain a fix

@Aaronontheweb
Copy link
Member

So this isn't an issue with Akka.Remote or Akka.Cluster - it appears to be an issue with Akka.Tools.DistributedPubSub, based on your reproduction sample. I'm going to push a reproduction spec here shortly.

Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this issue Feb 27, 2020
@Aaronontheweb
Reproduction spec for akkadotnet#4120 - Turning on message serializat…
1583678 
…ion breaks clustering
@Aaronontheweb Aaronontheweb linked a pull request Feb 27, 2020 that will close this issue
Draft
@Aaronontheweb
Copy link
Member

Taking this off of the 1.4.0 milestone - it's a minor issue and ultimately a problem because Newtonsoft.Json can't handle Systems.Collections.Immutable. Not getting a clear beat on which specific messages are causing the issue in #4721 but I'll leave this up for grabs.

@Aaronontheweb Aaronontheweb changed the title Turning on message serialization breaks clustering serialize-messages =on doesn't work with DistributedPubSub Feb 27, 2020
@Aaronontheweb Aaronontheweb removed this from the 1.4.0 milestone Feb 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
akka-cluster-tools confirmed bug serialization up for grabs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix serialize-messages for DistributedPubSub Aaronontheweb/akka.net
2 participants
@Aaronontheweb @MaximG1234