Deprecating schema-based serializers (ProtoBuf, Avro, Thrift)

[icebob]

The built-in schema-based serializers (ProtoBuf, Avro, Thrift) will be deprecated and removed in the next 0.15 version of Moleculer. These serializers will be moved to an external repository and published as an external NPM package, so if you use them in your project, you will able to use them further, just you should install them in your project.

The reason

The Moleculer transporters use serializers to serialize the calling parameters and other metadata which can be user-defined, so in these serializers, these fields not covered by schema, instead used JSON stringify/parse to convert. It means only a little part of the packet (only protocol used fields) covered by the schema the payload converted with JSON serializer (the rate depends on the size of the payload).

Some tests

Rate: percentage of serializer packet size / payload size

Payload size: 10 bytes

Serializer	Serialized packet length	Rate	ops/sec
JSON	169 bytes	5.9%	487,816
MsgPack	126 bytes	7.9%	104,777
Avro	87 bytes	11.4%	320,970
ProtoBuf	90 bytes	11.1%	447,391

Payload size: 1 kBytes (1,052 bytes)

Serializer	Serialized packet length	Rate	ops/sec
JSON	1211 bytes	86.8%	137,778
MsgPack	1081 bytes	97.3%	29,085
Avro	1130 bytes	93.1%	104,005
ProtoBuf	1133 bytes	92.8%	123,336

Payload size: 100 kByte (101,021 bytes)

Serializer	Serialized packet length	Rate	ops/sec
JSON	101,180 bytes	99.8%	2,444
MsgPack	93,311 bytes	100%	416
Avro	101,100 bytes	99.9%	2,272
ProtoBuf	101,103 bytes	99.9%	2,286

Notepack serializer speed is better than MsgPack: 1,975

Results

If you see the values, you will see that only ~10 bytes payload where schema-based serializers affect the packet size and can reduce it. But this payload size in real projects is rare.

In the 1kB - 100kB range, the schema-based serializers has no effect because the packet 99% serialized with JSON stringify/parse. Therefore the speed and size are the same as the JSON serializer, as well.

However, the schemaless serializers (MsgPack, Notepack) are able to reduce the payload size, as well.

[JordanPawlett]

I have some WIP (nearly finished) solutions of an Avro Serializer, which is serialising the full schema protocol, to fully utilise the transporter, that i should be able to contribute.
I'll be dusting it off in the near-future, it's part of our work to move towards Kafka. However, it's currently on pause.