WIP: More efficient ProtobufNet serializer

[icanhasjonas]

• Reduced the number of buffer copies
• Removed the length prefix/guard (possible breaking change)

These are improvements easily applied to the BinaryFormatter serializer too

[jsteinich]

The length prefix change is breaking since it changes what the 1st 4 bytes mean.
It should be pretty simple (and low perf impact) to make reading of old data still work, but I'm not sure there's a way to go the other direction.

Personally, I think the ability to read old data is critical, but having the old code be able to read new data doesn't seem very important.

[icanhasjonas]

... maybe I'll just use a MemoryStream and do TryGetBuffer() to get the supporting ArraySegment<byte> and then pass that over to the IBinaryTokenStreamWriter.. that way we should get minimal buffer copies while still being able to accurately write the length prefix

[icanhasjonas]

.. and replace the WriteAdapter

[KevinCathcart]

Yeah, I agree back-compat is a concern, here.

Arguably a bigger concern is that protobuf does not have any size-of-message or end of message indication, so it will consume until the end of the stream. I think that could be a problem, since objects serialized with this could be embedded inside an object serialized with the default Orleans serializer, so it needs to stop before the end of the stream.

But protobufnet actually has built-in support for inputting and outputting exactly the format with the 32 bit length prefix, so we can use it to fix both concerns. It only cleanly exposed on the Serializer class when providing a type argument, but you can do it yourself with:

//Deserialize (This DOES handles nulls represented as 0 length.)
return RuntimeTypeModel.Default.DeserializeWithLengthPrefix(stream, value: null, type, style, fieldNumber: 0);

//serialize (unlike Deserialize, this does NOT handle nulls. Write 0 yourself for them.)
RuntimeTypeModel.Default.SerializeWithLengthPrefix(stream, value, type, PrefixStyle.Fixed32, fieldNumber: 0);

Now, you make think that on serialization this is just using a buffer inside protobuf, and well, you would be right. However this is a buffer it already uses, but we are still saving one round of buffer copying like this, although the returns would be diminished by that internal buffer growing more than it normally would. Still one fewer copies is still an improvement.

Deserialization does not have this problem, since it just tracks position, and will not read more than the number of bytes in the prefix. However, in this case, deserialization method will make use of ReadByte() for reading the length prefix, so you should probably implement that on the Adapter stream, to avoid the crazy inefficient base class implementation.

Yes, this is a supported protobufnet API. the serializer class is merely a user friendly front-end over the RuntimeTypeModel, but using it directly is both supported, and potentially necessary if you want to use a custom type model to decide which properties to serialize, and what number they get.

[ReubenBond]

An updated protobuf serializer implementation could be offered either as a separate package, or perhaps the efficiency improvements can be put behind a configuration switch to retain compatibility (unsure how viable that is). Regardless, I'll close this for now, but feel free to open a new one if this is still of interest. Also note that Orleans now includes a fast, version-tolerant serializer out of the box (in 4.0.0-preview1). It also now supports configurable serializers on a per-storage-provider basis (#7416), so hopefully external serializers are easier to implement in future.