options to limit max memory uaage

[vtolstov]

I saw in docs many options to limit memory per partition, broker, fetch and other.
But in microservices we need to limit upper memory size not , and number of brokers and partitions does not matter because it can change over time.
Can you suggest how to limit?

[twmb]

As a consumer or as a producer?

As a producer, you can limit MaxBufferedMessages, and then application side, limit the size of what you're producing, so that you know there's an upper bound of messages*record_size.

As a consumer, you can limit AllowedConcurrentFetches, and then limit FetchMaxBytes, so that you know at most n_concurrent_fetches*fetch_max_bytes will be buffered internally (minus the one caveat that if a single record exceeds FetchMaxBytes, kafka will return it).

[twmb]

(I may change AllowedConcurrentFetches to MaxConcurrentFetches for the 1.0 release--allowed is odd naming considering the rest of the API)

[twmb]

One last note: inflation from decompression cannot really be controlled. If a small fetch inflates to 1G when decompressed, the client can't know that going into the decompression. It's best to limit this on the producer side by setting BatchMaxBytes, which is the maximum uncompressed size when producing before compression is involved.

[vtolstov]

no i'm not use compression =) yes question about consumer

[vtolstov]

all fine thanks