Description
My code needs to serialize lot of byte arrays (typically 50-300 elements long)
I've noticed that serialization doesn't scale that well on 32+ core (64+ threads) machines, probably due to protobuf doing locking when getting a buffer?
Let's say im serializing many arrays in parallel, like so:
byte[][] arrays; Stream[] streams; Parallel.For(0, 1000, i => { for (var j=0;j<10000;j++) Serializer.SerializeWithLengthPrefix(streams[i], arrays[j]); });
This is just a pseudocode. There are many (several hundred streams representing different files) and thousands of arrays to be serialized per stream.
This could be much faster if i wrote arrays to the stream directly, or if i created a wrapper class, and serialized that in one go.
But multiple SerializeWithLengthPrefix invokation seem to slowdown due to internal locking
Is there a workaround for that? I'm using protobuf 2.4.0