ZStandard is not documented

[haf]

Describe the bug
It's not clear how to configure ZStandard despite it being marked fixed and implemented in PR:s and README:

Snappy, LZ4 and ZSTD compression through pluggable codecs

And https://github.com/tulios/kafkajs/pulls?q=is%3Apr+zstd+is%3Aclosed in particular #875

1. snappy has kafkajs-snappy but there's no kafkajs-zstd https://www.npmjs.com/search?q=kafkajs-zstd
2. It's not documented under https://kafka.js.org/docs/1.11.0/producing#a-name-compression-a-compression
3. I can't find a constructor any => () anywhere, for it
4. You cannot not configure it, or it will crash at runtime (yup, tried that)

[Nevon]

The compression codes are not shipped as part of KafkaJS itself, as there's no reason for you to have a dependency on for example snappy if you are not using snappy. That's why we've made the compression codes pluggable, so that you can add whatever compression codec you need and not any others. We had to do some work on our side to support newer versions of the Kafka protocol in order to be able to support zstandard compression, which we have now done.

What's missing is a codec. In the case of zstandard, no one has written a compression codec as of yet. The codecs are generally very simple - they're just a piece of glue to adapt the underlying compression library to KafkaJS' expected interface. If you take a look at kafkajs-snappy, it's very simple: https://github.com/tulios/kafkajs-snappy/blob/master/src/index.js

The challenging bit is the actual compression. For zstandard, there really isn't a good, maintained set of bindings to the zstandard C-library. The closest I've found is https://github.com/zwb-ict/node-zstd, which hasn't been updated since 2016 and certainly won't build with modern node versions.

So when we say that we support zstandard compression, it means that we support all the bits around Kafka to enable zstandard. Writing an actual compression library is quite out of scope for us, but if someone were to update node-zstd or write a new set of bindings, it would just be a matter of plugging it into KafkaJS and everything should work.

[haf]

Well answered! Thanks.

[Nevon]

Turns out there was a bug in kafkajs that made this impossible. I have a PR up (#912) to fix it. Once that's merged, zstd is possible.

I wrote a codec for it that does work. However, it's depending on what looked like the most promising fork of node-zstd I could find (https://github.com/rmw-lib/cppzst), so I can't really guarantee that it's production worthy or that it will necessarily be updated going forward. However, it can easily be adapted if a better package appears at some point. https://github.com/kafkajs/zstd