This proposal introduces zstd as an optional compression codec to address the performance bottlenecks often associated with the current GZIP implementation. Located in the serialization layer of storm-client, GZIP frequently introduces unnecessary latency in real-time topologies; however, industry benchmarks from Cloudflare and Kafka (KIP-390) consistently demonstrate that zstd delivers significantly faster compression and decompression speeds without compromising the compression ratio. To maximize flexibility, I propose making the compression level user-configurable, with a recommended default of 3 to provide an optimal balance between CPU efficiency and data density. Implementing zstd will allow Storm users to reduce serialisation/deserialization processing latency and lower CPU overhead, bringing the project in line with modern data-streaming performance standards.
Resources:
This proposal introduces zstd as an optional compression codec to address the performance bottlenecks often associated with the current GZIP implementation. Located in the serialization layer of storm-client, GZIP frequently introduces unnecessary latency in real-time topologies; however, industry benchmarks from Cloudflare and Kafka (KIP-390) consistently demonstrate that zstd delivers significantly faster compression and decompression speeds without compromising the compression ratio. To maximize flexibility, I propose making the compression level user-configurable, with a recommended default of 3 to provide an optimal balance between CPU efficiency and data density. Implementing zstd will allow Storm users to reduce serialisation/deserialization processing latency and lower CPU overhead, bringing the project in line with modern data-streaming performance standards.
Resources: