What is the approach for non-POCOs in 4.X

[SebastianStehle]

Hi,

The Release describes what to do with POCOs. But what is the approach for Value types and collections and everything else that is not a POCO?

I understand how POCOs get serialized. You loop over all the properties and then you have default serializers for the most common value types (DateTime, Date, Guid and so on) and collections (List, Dictionary, HashSet ...).

But what is, when I have types, that I cannot annotate, for example a custom value type for a Guid, or NodaTime. In 3.X it works, because the underlaying fields are often serialized (in my understanding). But what do I have to do now? Is there a new interface for a custom converter / serializer?

[ReubenBond]

Hi Sebastian, could you elaborate?

[SebastianStehle]

I updated my question

[ReubenBond]

A surrogate can be used in place of the original type. Below is an example of creating a serializer for a type which cannot be annotated because it comes from an assembly which you don't own.

The general pattern is to create a surrogate type, have Orleans generate a serializer for that, and then create a converter. In the example below, I've also defined a copier. Often times, the type is immutable, so the copier can simply return the input.

[RegisterSerializer]
public sealed class UriCodec : GeneralizedReferenceTypeSurrogateCodec<Uri, UriSurrogate>
{
    private readonly TypeConverter _uriConverter = TypeDescriptor.GetConverter(typeof(Uri));

    public UriCodec(IValueSerializer<UriSurrogate> surrogateSerializer) : base(surrogateSerializer)
    {
    }

    public override Uri ConvertFromSurrogate(ref UriSurrogate surrogate) => surrogate.Value switch
    {
        null => null,
        _ => (Uri)_uriConverter.ConvertFromInvariantString(surrogate.Value)
    };

    public override void ConvertToSurrogate(Uri value, ref UriSurrogate surrogate) => surrogate = value switch
    {
        null => default,
        _ => new UriSurrogate
        {
            Value = _uriConverter.ConvertToInvariantString(value)
        },
    };
}

[GenerateSerializer]
public struct UriSurrogate
{
    [Id(1)]
    public string Value { get; set; }
}

[RegisterCopier]
public sealed class UriCopier : IDeepCopier<Uri>, IGeneralizedCopier
{
    public Uri DeepCopy(Uri input, CopyContext context) => input;

    // Uri has derived types and these are handled by declaring this as a "generalized copier" which
    // knows how to copy them. Usually, you don't need to do this. The "System.Type" type also needs something similar
    public object DeepCopy(object input, CopyContext context) => input;
    public bool IsSupportedType(Type type) => typeof(Uri).IsAssignableFrom(type);
}

Note that if the type lives in an assembly which you do own, then you have the option of annotating the type using the correct attributes but generating code in a separate assembly. In order to accomplish that, the assembly with the types should not have the code generator installed (only Microsoft.Orleans.Serialization.Abstractions), and another assembly should have both the code generator installed as well as an attribute in the form of [assembly: GenerateCodeForDeclaringAssembly(typeof(SomeTypeInTheOtherAssembly))].

[ReubenBond]

perhaps something like what I described above could help?

[GenerateAssemblyVersionedSerializer(typeof(AnyType), "1.1.0")

That would be a whole new mechanism and I'm not sure yet how it would work: would it break compilation when the target assembly's version changes? If it does change and the developer bumps the version in the attribute, what would the behavior be? Assembly versions are generally unreliable.

We could have something like [assembly: AllowType(typeof(X))] and [assembly: AllowTypesFromAssembly(typeof(X))]. I was thinking of the lower-level mechanism: adding a more user-friendly interface for checking individual types. I'm considering whether or not we should allow all types by default: the new serializer is simply unable to serialize/deserialize/copy arbitrary/unknown types, and nested codecs must be responsible for their own type loading logic.

[VincentH-Net]

how it would work: would it break compilation when the target assembly's version changes?

Yes; I outlined how it could work above:

include a (mandatory?) 'string assemblyVersion` parameter, validated against Version, that can be included in the serialized instance and enforced at codegen time (so it won't compile if the version of the type's containing assembly does not match - the compile error can again contain / link to a clear description of the risk involved in updating). This will prevent unwanted updates at compile time and give a clear exception at runtime (indicating the type name, payload assembly version and serializer assembly version) if a dev makes a wrong choice

If it does change and the developer bumps the version in the attribute, what would the behavior be?

Bumping the version in [GenerateAssemblyVersionedSerializer] would indicate a breaking change in the serialization of the type, so the grain interfaces that include this type must be versioned as well, and their versions must be bumped at the same time. The configured GrainVersioningOptions and deployment approach will then handle things as usual - if the change in serialization is backwards compatible (unless Orleans has a means to handle non-backwards compatible grain versions - is there such a thing as a custom version selector strategy?).

By adding a ushort serializationVersion to [GenerateAssemblyVersionedSerializer] it is possible to have the relevant grain interfaces version increase automatically when there is a breaking change in the serialization of an external type:

static class ExternalTypeVersions
{
    public const ushort MyExternalType = 1;
}

[assembly: GenerateAssemblyVersionedSerializer(typeof(MyExternalType), "1.1.0", ExternalTypeVersions.MyExternalType)

[Version(1 + ExternalTypeVersions.MyExternalType)]
public interface IMyGrain : IGrainWithStringKey
{
    // Signature(s) that contain MyExternalType
}

The developer only needs to increase ExternalTypeVersions.MyExternalType when updating the dependency and while doing so concluding that there is in fact a breaking change in the serialization of that type (should be guarded by unit tests). So the assemblyVersion parameter for the GenerateAssemblyVersionedSerializer attribute can be updated while the serializationVersion parameter remains unchanged.

So [GenerateAssemblyVersionedSerializer) can add safety both at compile time and runtime.
My thinking was that it would be more safe than simply allowing an external type without any versioning as [assembly: AllowType] would do. Even though assembly versions are not fully reliable, it will be better than nothing and provide a middle ground.

It would be a new mechanism; perhaps we even want all of the above so we have three safety levels for serialization:

• High: [GenerateSerializer] for owned types and surrogates
• Medium: [GenerateAssemblyVersionedSerializer]
• None: [AllowType]

Btw instead of separate attributes we could us a single one for external types, and also support generating serializers for many types with a single attribute, by adding an additional parameter to match against the type names in the assembly of the specified type:

[assembly: GenerateExternalTypeSerializer(typeof(MyExternalType), "1.1.0", 1)]                // Single external type, assembly versioned
[assembly: GenerateExternalTypeSerializer(typeof(MyExternalType), "1.1.0", 1, "Namespace.*")] // External types in namespace within assembly, assembly versioned
[assembly: GenerateExternalTypeSerializer(typeof(MyExternalType))]                            // Single external type, unversioned
[assembly: GenerateExternalTypeSerializer(typeof(MyExternalType), "*")]                       // All external types in assembly, unversioned

I would be happy with just supporting unversioned external types; however also supporting assembly versioned external types may be a useful middle ground; it would offer devs a measure of safety while not needing to write a surrogate for every external type they use; especially if that are many types like domain objects created by another team.

[ReubenBond]

Interesting idea, @VincentH-Net. It would require the code generator to inspect the foreign assembly and assign ids to the properties/fields in it. I think it's an interesting idea, but I'm concerned that it's too magic and too difficult to get right, since downstream developers don't know what the upstream developers have changed.

Additionally, if you were ever storing any of these types, then the version number wouldn't stop things from blowing up since you would still encounter cases where field orders have changed. Ultimately, the version-number approach could work, but I think it would require a schema repository, eg checked into source control.

I've opened a PR for surrogates here: #7776

[VincentH-Net]

Valid points, thx!

• I had not thought about the likelihood of those types being stored as well. This makes it likely that issues will not be prevented - so the added security when compared to [AllowType] would be less than I thought.
• Maintaining a schema repo reduces/nullifies the effort saved when compared to creating surrogates

All in all I agree that the idea is not likely to add enough value to justify adding the mechanism (and educating devs about it).

Thx for mentioning me in the surrogate PR; @ReubenBond how could I keep track of below?

and follow it with a better solution for allowing types

I'd like to follow [assembly: AllowType(typeof(X))] and [assembly: AllowTypesFromAssembly(typeof(X))] or equivalent

[ReubenBond]

Part of it is here: #7781
I'm yet to implement those attributes, but I don't think they're needed anymore: any serializer will now have the ability to allow a given type to be serialized by implementing ITypeFilter

[SebastianStehle]

Btw: What is the approach for Exceptions? The old ISerializable thing?

[SebastianStehle]

So you cannot use something like ValidationException anymore?

[SebastianStehle]

They are dozens of exceptions where I would be interested in the other properties as well.

[ReubenBond]

That's correct: you wouldn't be able to sensibly use ValidationException unless you wrote a custom serializer for it. Perhaps we can add certain exception types in-the-box.

The priority is to enable version-tolerance. I'd say we could use ISerializable to support it, but as you can see here, ValidationException does not implement it correctly.

I'm open to your suggestions.

[SebastianStehle]

Okay the example with ValidationException was stupid, because I have my own ValidationException.

There are a lot of exception types that have custom properties and I would say it should be a best practice. Something like a database error code can be handled with a property easily and it should not be necessary to parse the message to get the result.

I would recommend the following:

1. Support SerializationInfo in the serializer. It is basically just a dictionary, so it should not be too complicated.
2. If an exception supports ISerializable serialize the exception.
3. If the exception cannot be serialized use a wrapper exception and try to obtain as much infos from the original exception as possible. Perhaps with reflection, json serializer or whatever.

For me it happened a few times that I lost the information from an exception because it could not be serialized. I have therefore written a filter for that: https://github.com/Squidex/squidex/blob/master/backend/src/Squidex.Infrastructure/Orleans/ExceptionWrapperFilter.cs.

This is an typical error that often happens in production only, because locally or on the CI machine you will not see it if you do not run a cluster. Therefore we should avoid to loose error details.

I understand and support the idea to move away from the "It works magically until it does not work" approach but exceptions are another story. When you have a look to a method or grain signature you see the input and output. But exceptions do not follow this rule. Some exceptions just flow from an class trough a grain to asp.net core middleware. It is very easy to forget to write a serializer for such an exception.

[ReubenBond]

Support SerializationInfo in the serializer. It is basically just a dictionary, so it should not be too complicated.

We have some support for that already, it's not too difficult. It will work as long as the inner types are serializable, too.

If an exception supports ISerializable serialize the exception.

Sure, we can do that.

If the exception cannot be serialized use a wrapper exception and try to obtain as much infos from the original exception as possible. Perhaps with reflection, json serializer or whatever.

We can special case Exception to use a fallback if the original type is not serializable, but I don't want to introduce the fragility of reflection or attempt to fall back to JSON serialization when the developer hasn't configured it.

Most exceptions are not designed with remoting in mind, which unfortunately leads to things like an HttpException which carries the whole world with it. Using reflection to try to maintain fidelity for these exceptions often results in additional problems, so I think it's the wrong solution.

[oising]

What about using a CustomReflectionContext ( https://docs.microsoft.com/en-us/dotnet/api/system.reflection.context.customreflectioncontext?view=dotnet-plat-ext-6.0 ) to synthesize the [GenerateSerializer] and [Id] attributes needed? This would allow Hagar to remain mostly unmolested -- you'd have to provide an way to provide an optional reflection context -- while allowing a configuration - maybe a manifest config text file - which explains how to annotate external types for which you don't have the source.

[ReubenBond]

The code generator inspects the assemblies/code via Roslyn at build time, so CustomReflectionContext is not available/applicable. If having external annotations (eg on a dummy mirror type or possibly in some IDL file) is satisfactory, then something like that might be an option.

I'm considering other options, too, like a more user-friendly way to construct serializers programmatically, eg using a builder pattern to map names to fields/properties.

[koenbeuk]

Related question, are there plans on properly supporting record types? Currently we'll have to define al Members explicitly and mark them with an Id attribute, e.g.

[GenerateSerializer]
record Person(string FirstName, string LastName) {
    [Id(0)]
    public string FirstName { get; init; }

    [Id(1)]
    public string LastName { get; init; }

    [Id(2)]
    public string? MiddleName { get; init; }
}

This could hypothetically be simplified to something in the lines of:

[GenerateSerializer]
record Person(string FirstName, string LastName) {
    [Id(0)]
    public string? MiddleName { get; init; }
}

Where Orleans codegen automatically includes primary constructor arguments before including marked properties. Version tolerance will still be a thing since changing the primary constructor would be a breaking change anyway.

[koenbeuk]

To emphasize on my last point: If we were to introduce an Age property afterwards through non-breaking manners, the record would ideally look like this:

record Person(string FirstName, string? MiddleName, string LastName)
{
    // Optional constructor
    public Person(string firstName, string middleName, string lastName, int age) {          
        FirstName = firstName;
        MiddleName= middleName;
        LastName = lastName;
        Age = age;
    }

    [Id(0)]
    public int? Age { get; init; }
}

The minimum amount of work to have both simplicity and version tolerance.

[ReubenBond]

I'm ok with this. The [Id(0)] bit is odd, though: I imagine we'd have to error-out if you tried to do that, since the FirstName property would likely have that id. Either that, or the default ids for the constructor properties would have to start at some higher value (eg, 1000). The CodeFix could auto-add the right id for you.

[koenbeuk]

I would think that primary constructor properties are automatically assigned an Id (probably in a reserved range as you suggested). Perhaps the negative range can be reserved for primary constructor properties? that way, we can enforce explicit properties to start from 0. Happy to contribute in this area.

[ReubenBond]

Negative ids are not valid. Ids starting at 0 and incrementing by 1 are encoded more efficiently, since the difference between subsequent ids are encoded (and properties are encoded in ascending order) and differences of less than 7 between ids can be encoded without consuming extra space (they are encoded directly into the 'tag' byte for the property).

Help is appreciated. If you're willing to tackle this, please feel free to message me on Discord so that I can assist you in case you run into any issue or if you'd like to know where to look

[VincentH-Net]

@ReubenBond

I'm ok with this.

I would appreciate this improvement; the current supported functionality with repeated [property: Id()] feels like a workaround and unnecessary ceremony; it detracts from the excellent readability of this record syntax.

Having to mark a record as [Immutable] explicitly seems like unnecessary ceremony as well. I am aware that records are not really guaranteed immutable, but I think it makes more sense for records to be immutable (in the Orleans sense) by default instead of mutable. It will make Orleans applications more performant by default when Co-Hosting - no serialization at all in-process.

I.e. now almost every record in my Orleans project looks like:

[GenerateSerializer, Immutable]
public record UserInfo([property: Id(0)] Guid Id, [property: Id(1)] string Name);

All those records could be:

[GenerateSerializer]
public record UserInfo(Guid Id, string Name);

You could theoretically still support mutable records with a [MutableRecord] attribute, but even though this is technically possible I think it would be better not to support mutable records. The fact that you would need to apply an attribute means you own the record type, so you also should be able to define the record safe for immutability within Orleans.

Finally - since [Immutable] is just an optimization for serialization, and Orleans 4 now has the [GenerateSerializer] attribute, would it make more sense to make [Immutable] obsolete and instead add an optional bool? immutable parameter to [GenerateSerializer]? The default would be mutable for classes and immutable for records. So an immutable class would get:

[GenerateSerializer(immutable: true)]
public class MyClass

[oising]

Having to mark a record as [Immutable] explicitly seems like unnecessary ceremony as well. I am aware that records are not really guaranteed immutable, but I think it makes more sense for records to be immutable (in the Orleans sense) by default instead of mutable. It will make Orleans applications more performant by default when Co-Hosting - no serialization at all in-process.

This is a foot gun. A plain record is a reference type, and will lead to disgusting bugs if people forget this. If you want immutable records, use readonly record struct. The [immutable] attribute is there for a reason - if you're using a byref record and you know what you're doing, then you should have to explicitly opt-in.

[VincentH-Net]

@oising using struct is only recommended for type instance sizes under 16 bytes (guideline).

In a real-world application only a small part of the [Immutable] types would be under 16 bytes, so using readonly record struct is not a realistic solution to prevent that type of errors.

I see the point you make; safe Immutable by default on record would require analyzing the property types for immutability while processing [GenerateSerializer] - e.g. ImmutableArray<T> for collections.

It could be fairly simple analysis, just have a known list of framework types that can be considered safe for Orleans immutability and only consider records types immutable if they contain no other types. Devs can still override the default either way with an optional bool? immutable parameter for the [GenerateSerializer] attribute.

I do think that the vast majority of record types could be perf optimized automatically for Orleans immutability that way, and also safe. And with less ceremony / noise in the code.