Compress messages before publishing - Google Pub/Sub

[kk2491]

Hi All,

I am new to Google pub/sub.

Considering the maximum limit for the publish message is 10 MB, is there a way to compress the messages before publishing the data to a topic.

While going through I found this article, however I am not really sure if this is recommended by GCP as I was not able to get the resources in GCP.

Your help would be really appreciated.

Thank you

[kir-titievsky]

Pub/Sub PM here. Just a note that this is makes sense as a feature request. I've heard this from another customer who quoted a 5-10X compression ratios for JSON data. Since Kafka can compress data "on the wire", this can mean that without Pub/Sub-side compression the output bandwidth of the sink connector is 5-10X the input.

We'll have to think about how to support this properly.

[kk2491]

@kir-titievsky Thanks for the response. It would be very helpful if some kind compression is in place in pubsub msgs.

Thank you

[kir-titievsky]

@kk2491 understood. We'll keep this bug open as a feature request. I agree this would be a good for us to add, although, cannot at the moment say when we can get to this. In the meantime, consider the recommendation to compress each message body in the application code as official.

[kk2491]

@kir-titievsky Thank you. 👍🏾