Fory binary format - is it worthwhile compressing for transferring over the wire

[pjfanning]

I'm looking at create a Fory based serializer for Apache Pekko. Pekko supports comms between Pekko nodes running as a cluster. It already has Jackson support as well as using Protobuf for some internal classes. The serializers used are pluggable so a Fory based one could be an useful option.
The Jackson based serialized for Pekko supports compression (eg LZ4) of the JSON and CBOR formats.
Would it be useful to also support compression for Fory's data format?

[chaokunyang]

Fory use varint for innt32/64 compression, and has a highly-optimized format for struct meta compression. If reference tracking enableld, the shared refrence will be serialized only once like dict encoding. But for other cases, fory doesn't apply compression. Whether compression help will depend on the data distribution. If the object being serialized has some repeated pattern which are not captured by fory, the compression like LZ4 will help.

I think the such high-level compression can be a separate feature, and can be applied the all serialized binary in pekko, whether it is serialized by fory or not

[pjfanning]

Thanks @chaokunyang. We have the compression code in Pekko so I don't need to move it to Fory.
I'll allow users of my serializer to enable the LZ4 compression but it will be optional and I'll probably default to having it turned off.