Orleans fails to perform JsonPatchDocument operations

[Aashish93-stack-zz]

We are in the process of upgrading our projects to orleans 3.5.0 and donet 5.0 and we observed that all of our JsonPatchDocument requests started to fail with the following
Unhandled exception. System.Runtime.Serialization.SerializationException: Reference with no referred object at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task1 asyncTask)

While trying to reproduce the same Issue on a the HelloWorld Sample project. On reproducing the issue, we noticed some interesting logs related to deserialization on the SiloHost and we believe we are seeing the same in our internal projects.

         System.Runtime.Serialization.SerializationException: Reference with no referred object
         at Orleans.Serialization.DeserializationContext.FetchReferencedObject(Int32 offset)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.SerializationManager.DeserializeArray(Type resultType, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, IDeserializationContext context)
         at TablesDeserializer(Type , IDeserializationContext )
         at Orleans.Serialization.ILBasedSerializer.Deserialize(Type expectedType, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, IDeserializationContext context)
         at ConcurrentDictionary`2Deserializer(Type , IDeserializationContext )
         at Orleans.Serialization.ILBasedSerializer.Deserialize(Type expectedType, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, IDeserializationContext context)
         at ThreadSafeStore`2Deserializer(Type , IDeserializationContext )
         at Orleans.Serialization.ILBasedSerializer.Deserialize(Type expectedType, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, IDeserializationContext context)
         at DefaultContractResolverDeserializer(Type , IDeserializationContext )
         at Orleans.Serialization.ILBasedSerializer.Deserialize(Type expectedType, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, IDeserializationContext context)
         at JsonPatchDocumentDeserializer(Type , IDeserializationContext )
         at Orleans.Serialization.ILBasedSerializer.Deserialize(Type expectedType, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Serialization.BuiltInTypes.DeserializeInvokeMethodRequest(Type expected, IDeserializationContext context)
         at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
         at Orleans.Runtime.Messaging.MessageSerializer.OrleansSerializer`1.Deserialize(ReadOnlySequence`1 input, T& value)
         at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message)
         at Orleans.Runtime.Messaging.Connection.ProcessIncoming()

Versions used in the project

donet sdk version: 5.0.401
Target Framework: net5.0
Orleans Version: 3.5.0
Microsoft.AspNetCore.JsonPatch Version: 5.0.10

I have uploaded the zip file for the reproducer. Instructions for running are in the readme file
HelloWorld.zip

[ReubenBond]

Thanks for reporting, @Aashish93-stack.
The primary issue here is that BinaryFormatter is disabled by default on .NET 5.0 and above.
Orleans 3.5.0 attempts to detect this and mitigate it by enabling the in-built ILBasedFormatter in that case, however it is not working for you. It may be that the payload in your case was serialized by BinaryFormatter but is being deserialized by ILBasedSerializer.

One workaround is to reenable BinaryFormatter temporarily. That workaround is presented here: #6805 (comment)

[ReubenBond]

There are a couple of issues here:

1. ILBasedSerializer is enabled as a "keyed serializer" even when BinaryFormatterSerializer is enabled as the fallback, and it will be used preferentially. I will see what kinds of fixes we can have for this case. It may be that we offer some kind of opt-in so that this serializer is only used if either it's been enabled or if it's explicitly referenced in the data.
2. In this case, the ILBasedSerializer is deserializing an array referenced in your repro which is self-referential (one of the array members is the array itself). The type comes from DefaultContractResolver, which arguably should not be serialized anyway. This is a bug which we should be able to safely fix without hurting any other cases.

[Aashish93-stack-zz]

Thanks for the update @ReubenBond

[britaincarter]

Updating the example HelloWorld app to orleans 3.5.1 did not resolve the serialization error. Is there another work around needed or a working example? @ReubenBond @Aashish93-stack