Truncating after compression

[matthewfala]

I noticed that truncation is occurring after compression in kinesis streams.

I'm not sure that the plugin can truncate compressed data and expect it to still be in a valid compressed format.

here is the plugin's current assumption

truncate(compress(data)) == compress(truncate(data))

Does that hold true for gzip and zlib?

amazon-kinesis-streams-for-fluent-bit/kinesis/kinesis.go

Line 479 in 94aa8d7

data = data[:maximumRecordSize-partitionKeyLen-len(truncatedSuffix)]

If not then the current truncation system may need to be replaced.

[zhonghui12]

This fix has been included in latest release: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.22.0.