How to use built primitive type serialization instead of writing wrapper types for every possible value

[tedbarth]

Deep in many of my static networked data types (all implement my own ISerializable) I have the following type.

public partial class NetworkVariableData : ISerializable {
  public readonly ushort Index;
  public readonly ISerializable? Value;
}

Value here can be almost anything (I write a library, so it is up to the user of the library). Of course I want to support as many types as possible. Right now I have Implemted ISerializable as union:

[MemoryPackable(GenerateType.NoGenerate)]
public partial interface ISerializable {
  // Just a marker
}

    (ushort, Type t)[] types = AppDomain.CurrentDomain.GetAssemblies()
      .SelectMany(a => a.ExportedTypes)
      .Where(t => t.GetInterfaces().Contains(typeof(ISerializable)) && !t.IsAbstract)
      .Select((t, i) => ((ushort)i, t))
      .ToArray();

    var formatter = new DynamicUnionFormatter<ISerializable>(types);

    MemoryPackFormatterProvider.Register(formatter);

This works fine, but I still have to write basic wrapper type for every known types, which already is becoming tedious, inflexible and error prone.

Example of how I would convert to ISerializable:

public static ISerializable? ToSerializable(object? value) {
    switch (value) {
      case null:
        return null;
      case ISerializable serializable:
        return serializable;
      default: {
        var type = value.GetType();
        return value switch {
          double v => new DoubleData(v),
          short v => new ShortData(v),
          ushort v => new UnsignedShortData(v),
          int v => new IntData(v),
          bool v => new BoolData(v),
          string v => new StringData(v),
          // Oh god, there are so many! :'(
          _ => throw new UnreachableException($"Cannot serialize type '{type}'")
        };
      }
    }
  }

Example of a wrapper ISerializable:

[MemoryPackable]
public partial class ShortData : ISerializable {
  [MemoryPackInclude] public readonly short? Value;

  public ShortData(short? value) {
    Value = value;
  }
}

MemoryPack already supports many standard types (at least the primitives). How can I make use of those without having to write those wrapper types?

[tedbarth]

@neuecc Proposal: Add API to get an index (or id) of a formatter and a formatter by index (or id):

  public class KnownObjectFormatter : MemoryPackFormatter<Object?> {
    public override void Serialize<TBufferWriter>(
      ref MemoryPackWriter<TBufferWriter> writer,
      scoped ref Object? value) {
      if (value == null) {
        writer.WriteNullObjectHeader();
        return;
      }

      Type type = value.GetType();
      ushort index = MemoryPackFormatterProvider.GetFormatterIndex(type);
      var formatter = MemoryPackFormatterProvider.GetFormatter(type);

      writer.WriteValue(index);
      formatter.Serialize(writer, value);
    }

    public override void Deserialize(ref MemoryPackReader reader, scoped ref Object? value) {
      if (!reader.TryReadObjectHeader(out var count)) {
        value = null;
        return;
      }

      ushort index = reader.ReadValue<ushort>();
      var formatter = MemoryPackFormatterProvider.GetFormatterByIndex();

      value = formatter.Deserialize(reader);
    }
  }

It is slower than when using static types, but not that much. I would expect it to be much faster that when having to wrap every value into an object on the heap. And there is no need to write boiler plate wrapper classes for primitives.

[tedbarth]

I helped myself with the following formatter:

  public class KnownObjectFormatter : MemoryPackFormatter<Object?> {
    public class KnownObjectFormatterException(string message) : Exception(message);
    private static readonly Bictionary<Type, ushort> RegisteredTypes = new Bictionary<Type, ushort>();

    public static void RegisterType(Type type) {
      if (!RegisteredTypes.ContainsKeyForward(type)) {
        ushort index = (ushort)RegisteredTypes.Count;
        RegisteredTypes.Add(type, index);
      }
    }

    public override void Serialize<TBufferWriter>(
      ref MemoryPackWriter<TBufferWriter> writer,
      scoped ref Object? value) {
      if (value == null) {
        writer.WriteNullObjectHeader();
        return;
      }

      Type type = value.GetType();

      if (!RegisteredTypes.TryGetValueForward(type, out ushort index)) {
        throw new KnownObjectFormatterException($"Type {type} not registered");
      }

      writer.WriteUnmanaged<ushort>(index);

      var formatter = writer.GetFormatter(type);
      formatter.Serialize(ref writer, ref value);
    }

    public override void Deserialize(ref MemoryPackReader reader, scoped ref Object? value) {
      if (reader.PeekIsNull()) {
        reader.Advance(1);
        value = null;
        return;
      }

      ushort index = reader.ReadUnmanaged<ushort>();
      if (!RegisteredTypes.TryGetValueReverse(index, out var type)) {
        throw new KnownObjectFormatterException($"No type with index {index} registered");
      }

      var formatter = reader.GetFormatter(type!);
      formatter.Deserialize(ref reader, ref value);
    }
    

    public class Attribute : MemoryPackCustomFormatterAttribute<object?> {
      private static readonly KnownObjectFormatter Formatter = new KnownObjectFormatter();

      public override IMemoryPackFormatter<object?> GetFormatter() {
        return Formatter;
      }
    }
  }

That boils down to usage:

KnownObjectFormatter.RegisterType(typeof(int));
KnownObjectFormatter.RegisterType(typeof(int?));
KnownObjectFormatter.RegisterType(typeof(int[]));
KnownObjectFormatter.RegisterType(typeof(bool));
KnownObjectFormatter.RegisterType(typeof(bool?));
KnownObjectFormatter.RegisterType(typeof(bool[]));
KnownObjectFormatter.RegisterType(typeof(string));
KnownObjectFormatter.RegisterType(typeof(string[]));
...
KnownObjectFormatter.RegisterType(typeof(MyCustomType));
KnownObjectFormatter.RegisterType(typeof(MyCustomType?));
KnownObjectFormatter.RegisterType(typeof(MyCustomType[]));
...

It still is tedious to do that for every possible already mapped type, but it at least is better than having to write wrapper classes.

So proposal as changed to:

• Expose types defined in src/MemoryPack.Core/MemoryPackFormatterProvider.WellknownTypes.cs to be consumed by such formatter (above code)

[hadashiA]

Hello.
This issue is probably a duplicate of #186.
Unfortunately, dynamically inferring types is difficult due to the binary spec.

We have looked at your KnownObjectFormatter and would like to recommend using Union for official support. The above sample should be substitutable with Union.