Compression support?

[mannywald]

Is there any plan to add compression support when putting data into Firehose?

Since Firehose charges are incurred based on the volume of data ingress, it would behoove a rational economic actor to minimize the amount of data being sent to Firehose (and thus compress data beforehand rather than using the compression option in Firehose to minimize the amount of data being written to S3).

[IanMeyers]

Hello. No there isn’t. The reason is that Firehose can’t automatically decompress data streams before egress to S3, but automatically will perform Base64 decoding, so compressed data would end up as gibberish is the target. Happy to refer you to someone at AWS who can help with overall cost optimisation if that helps. Email meyersi@amazon.com if you’d like to connect.

[mannywald]

Ah okay - thanks!

For the use case I was thinking about I wouldn't actually want to decompress the data stream before egress to S3, but rather take the compressed data received in Firehose and output that to S3 in its compressed state...