Zstd experiment

[ericvolp12]

Findings in this test conclude that enabling ZSTD compression for clients reduces BW per message by ~31.5% but increases CPU load an incredible amount.

In the below CPU profiles, Jetstream is serving from the playback buffer as fast as it can to a single client.

• The first profile is without compression, serving at 198k evt/sec
• The second profile is with compression, serving at ~28k evt/sec

The CPU costs of compression seems too high for now for me to want to support. The bandwidth usage of Jetstream even with hundreds of consumers is still well below 1Gbps for now and running multiple instances makes it relatively easy to scale out Jetstream horizontally for more consumers if needed.

This was a fun experiment though!

[ericvolp12]

I made some changes to use a trained dictionary for compression and also modified Jetstream to only compress events once and then store events in both Raw JSON and in compressed JSON in another PebbleDB. This should allow for replay without any round tripping through an encoder/decoder on the server side and allow for emitting events having only compressed them once for all consumers.