General Unsafe Formatters

[pCYSl5EDgo]

Abstract

There are some unsafe resolvers such as UnsafeBlitResolver.
Such unsafe formatters are affected by the endian and struct layout.

This Pull Request offers the general way to serialize/deserialize the unmanaged struct by copying the binary presentation.

Usage

Add UnsafeUnmanagedStructAttribute to the module to specify which type the programmer wants to serialize/deserialize unsafely.
DynamicUnsafeUnmanagedStructResolver generates unsafe formatter dynamically.

using System;
using System.Numerics;
using MessagePack;
using MessagePack.Resolvers;

[module: UnsafeUnmanagedStruct(typeof(Matrix4x4), UnsafeUnmanagedStructFormatterImplementationKind.Array | UnsafeUnmanagedStructFormatterImplementationKind.Array)]

namespace Hoge
{
    public static class Program
    {
        public static void Main()
        {
            var random = new Random();
            var resolver = CompositeResolver.Create(DynamicUnsafeUnmanagedStructResolver.Instance, StandardResolver.Instance);
            var options = MessagePackSerializerOptions.Standard.WithResolver(resolver);
            const int length = 114514;
            var original = new Matrix4x4[length];
            for (var i = 0; i < original.Length; i++)
            {
                original[i].M11 = (float)random.NextDouble();
                original[i].M12 = (float)random.NextDouble();
                original[i].M13 = (float)random.NextDouble();
                original[i].M14 = (float)random.NextDouble();
                original[i].M21 = (float)random.NextDouble();
                original[i].M22 = (float)random.NextDouble();
                original[i].M23 = (float)random.NextDouble();
                original[i].M24 = (float)random.NextDouble();
                original[i].M31 = (float)random.NextDouble();
                original[i].M32 = (float)random.NextDouble();
                original[i].M33 = (float)random.NextDouble();
                original[i].M34 = (float)random.NextDouble();
                original[i].M41 = (float)random.NextDouble();
                original[i].M42 = (float)random.NextDouble();
                original[i].M43 = (float)random.NextDouble();
                original[i].M44 = (float)random.NextDouble();
            }

            var binary = MessagePackSerializer.Serialize(original, options);
            var decoded = MessagePackSerializer.Deserialize<Matrix4x4[]>(binary, options);
            for (var i = 0; i < original.Length; i++)
            {
                if (original[i] != decoded[i]) throw new Exception();
            }
        }
    }
}

Performance

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.1082 (1909/November2018Update/19H2)
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.402
  [Host]   : .NET Core 3.1.8 (CoreCLR 4.700.20.41105, CoreFX 4.700.20.41903), X64 RyuJIT
  ShortRun : .NET Core 3.1.8 (CoreCLR 4.700.20.41105, CoreFX 4.700.20.41903), X64 RyuJIT

Job=ShortRun  IterationCount=3  LaunchCount=1  
WarmupCount=3  

Method	Size	Mean	Error	StdDev
SerializeUnsafe	1	114.3 ns	9.32 ns	0.51 ns
SerializeNormal	1	207.2 ns	11.51 ns	0.63 ns
DeserializeUnsafe	1	207.5 ns	6.50 ns	0.36 ns
DeserializeNormal	1	447.9 ns	21.05 ns	1.15 ns
SerializeUnsafe	64	478.1 ns	62.93 ns	3.45 ns
SerializeNormal	64	6,525.4 ns	886.36 ns	48.58 ns
DeserializeUnsafe	64	510.0 ns	78.91 ns	4.33 ns
DeserializeNormal	64	21,446.9 ns	1,095.69 ns	60.06 ns
SerializeUnsafe	1024	6,898.4 ns	128.76 ns	7.06 ns
SerializeNormal	1024	108,254.7 ns	1,279.67 ns	70.14 ns
DeserializeUnsafe	1024	5,193.4 ns	386.37 ns	21.18 ns
DeserializeNormal	1024	345,922.9 ns	51,375.10 ns	2,816.04 ns
SerializeUnsafe	16777216	895,800,300.0 ns	162,256,172.06 ns	8,893,808.30 ns
SerializeNormal	16777216	2,942,060,766.7 ns	1,745,869,374.82 ns	95,696,991.61 ns
DeserializeUnsafe	16777216	453,505,666.7 ns	79,486,869.72 ns	4,356,943.55 ns
DeserializeNormal	16777216	5,752,746,566.7 ns	177,602,469.96 ns	9,734,990.67 ns

[AArnott]

Thanks for contributing, again. :)
I like the idea, and Experimental seems like reasonable place to start with this. My main concern is the new attribute, as I explain below.

[AArnott]

Is there any value to defining this as Memory<T> or Span<T> or their readonly counterparts instead of requiring an array?

[pCYSl5EDgo]

Span<T> can not be the generic type argument.
I made Memory<T>, ReadOnlyMemory<T> and ReadOnlySequence<T> formatters.

[pCYSl5EDgo]

Thank you for your review.

Ok, I will remove the resolver, the attribute and ExtensionTypeCode(50/51).
The user programmers should clarify which types to unsafely (de)serilialize.
I'll make the constructor public and make it having one sbyte parameter to specify TypeCode.

Done.

[AArnott]

Looks great. @neuecc do you want to take a look? All the changes are in Experimental.

[neuecc]

Is this same(similar) as https://github.com/Cysharp/MagicOnion/blob/master/src/MagicOnion/UnsafeDirectBlitResolver.cs that before I wrote?
Injecting ExtCode from the outside is a good idea.
OK.

[pCYSl5EDgo]

Wow, very similar!
Mine is slightly different from MagicOnion's in the region of api usage but it is the same.