Need the ability to produce/consume messages beyond 1MB in size.

[ajhunyady]

Currently, when a client producer a record of 5881006 bytes, the producer panics:

I have a rust producer and pass a  JSON string. It results in an error: Record too large. What's wrong?
thread 'main' panicked at 'Failed to send message.: Producer(RecordTooLarge(16384))', src/main.rs:39:9

[ajhunyady]

fluvio/crates/fluvio-types/src/defaults.rs

Line 46 in 1dd976c

pub const STORAGE_MAX_BATCH_SIZE: u32 = 1048588;

pub const STORAGE_MAX_BATCH_SIZE: u32 = 1048588;

This value is not configurable as large records, greater than 1Mb impact performance.

[nacardin]

For now, we can increase default size

[ajhunyady]

@nacardin according to @tjtelan and @sehz there are performance considerations that need to be taken into account.

• Can we quantify the impact in performance degradation?
• Is it safe or we expose the product to various side-effects?

[sehz]

record size must be less than batch and segment size

[thinkrapido]

Hi @AJ , I've done compression but it's still too large (7131124 uncompressed, 1253783 compressed)

It is not possible for us to control the input because it comes from an RPC out of our control.

This use case is a real show stopper for us.