Guidance on encoder/decoder for compound types?

[marklam]

I have an F# record type which is marked as a ValueType via the StructAttribute

[<Struct; StructLayout(LayoutKind.Sequential, Pack = 1)>]
type Peak<[<Measure>] 'm> =
    {
        mz : double<'m>
        ic : single<intensity>
    }

In terms of the Type, there's 2 public read-only properties, backed by name-mangled fields which are marked as IsPublic:False and IsPrivate:False, and a constructor with parameters named after the properties:

type,"{Name = ""Peak"" FullName = ""XXX.Peak""}"
    DeclaredConstructors,{System.Reflection.ConstructorInfo[1]}
        [0],"{Void .ctor(Double, Single)}"
            ArgumentTypes,{System.RuntimeType[2]}
                [0],"{Name = ""Double"" FullName = ""System.Double""}"
                [1],"{Name = ""Single"" FullName = ""System.Single""}"
            m_parameters,{System.Reflection.ParameterInfo[2]}
                [0],{Double mz}
                [1],{Single ic}
    DeclaredFields,{System.Reflection.FieldInfo[2]}
        [0],{Double mz@}
            BindingFlags,Instance | NonPublic
        [1],{Single ic@}
            BindingFlags,Instance | NonPublic
    DeclaredProperties,{System.Reflection.PropertyInfo[2]}
        [0],{Double mz}
            BindingFlags,Instance | Public
            CanRead,TRUE
            CanWrite,FALSE
        [1],{Single ic}
            BindingFlags,Instance | Public
            CanRead,TRUE
            CanWrite,FALSE

Is there any info on how to either:

• build an automatic encoder/decoder pair for this kind of F# data type; or
• supply hand-coded custom delegates for each data type such as this

Thanks for any advice!

[Apollo3zehn]

I was able to make a successful roundtrip (encode then decode) with the following unit test code:

[Fact]
public void Test()
{
    const string filePath = "test.h5";
    const string datasetName = "data";

    var writeData = new Peak[] {
        new Peak(1.1, 2.2f),
        new Peak(3.1, 4.5f)
    };

    // encode data
    var writeFile = new H5File
    {
        [datasetName] = writeData
    };

    var options = new H5WriteOptions(
        IncludeStructProperties: true
    );

    writeFile.Write(filePath, options);

    // decode data
    var readFile = H5File.OpenRead(filePath);
    var readData = readFile.Dataset(datasetName).Read<Peak[]>();

    Assert.True(writeData.SequenceEqual(readData));
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
readonly record struct Peak(double mz, float ic);

Encoding

The important part is IncludeStructProperties: true. When you add a struct with no reference types to the H5File instance and then call H5File.Write(...), the value type encode path used (instead of the reference type path) where the .NET memory content of the variable is just being copied to the file:

PureHDF/src/PureHDF/VOL/Native/FileFormat/Level2/ObjectHeaderMessages/Datatype/DatatypeMessage.Writing.cs

Line 186 in 2cc8333

: GetTypeInfoForValueType(context, type),

When going this encode path only public fields are being considered to construct the HDF5 compound type while all others (static and private fields and all properties) are being ignored.

By setting IncludeStructProperties: true the reference type encode path is being used instead:

PureHDF/src/PureHDF/VOL/Native/FileFormat/Level2/ObjectHeaderMessages/Datatype/DatatypeMessage.Writing.cs

Lines 184 to 185 in 2cc8333

	=> context.WriteOptions.IncludeStructProperties
	? GetTypeInfoForReferenceLikeType(context, type)

Now the two relevant properties are being detected with their proper names and PureHDF is able to copy over the property values into the HDF5 file. The disadvantage is the lower performance of this operation (now every value is being copied individually instead of bulk copying the whole instance data as it is the case with the value type encoding path).

Decoding

With the reference type code path selected to encode data, PureHDF always packs all HDF5 compound members without any gaps into the file. That is why it is also important to define the Peak struct with Pack = 1 when reading back / decoding the data from the HDF5 file:

[StructLayout(LayoutKind.Sequential, Pack = 1)]

PureHDF does not behave symmetrically here in that for decoding, the value type decoding path is selected instead of the reference type path as it the case for encoding. Here is the line where the value type path is being selected:

PureHDF/src/PureHDF/VOL/Native/FileFormat/Level2/ObjectHeaderMessages/Datatype/DatatypeMessage.Reading.cs

Line 386 in 2cc8333

(false, false) when memoryTypeSize == fileTypeSize => GetDecodeInfoForUnmanagedElement(memoryType),

So to make it more consistent, PureHDF should also select the reference type path when IncludeStructProperties: true is specified in H5File.OpenRead(...) (or when the dataset itself is being read).

The problem which then arises is that the Peak type has read-only properties and so PureHDF cannot access them for writing / decoding. Instead it must use the constructor to instantiate the type. The problem is the same as for the .NET JsonSerializer type and described here:

dotnet/runtime#38539 (comment)

I did not yet implement this "constructor-based" instantiation and so the value-type decode path is currently the only way to properly decode the Peak type.

I will create a new issue to ensure that future versions do not have this asymmetric behavior anymore (i.e. slow reference type encoding vs fast value type decoding) when readonly record struct types are being involved.

---

Regarding your point supply hand-coded custom delegates for each data type such as this:

A hook into the encoding / decoding logic would be nice for the users to have but it would mean for me to make large portions of the internally API public (there are many HDF5 format specific types involved) and I am not sure how such an extension point would look like so that the custom implementation has access to all relevant data structures but at the same time it should not be able to make the serialization too faulty. Debugging a wrong bit in an HDF5 file can take a while.

I`d say it would be possible to build such an API but it requires some effort which I cannot put time into myself right now :-(

[marklam]

IncludeStructProperties: true works, thanks.

I'll give your explanation a proper read soon, but since I'm storing a compressed 2D-dataset where every 'cell' is a variable-length list of compound types, I think my performance is probably doomed anyway.