GRPC channel: PubSub inbound message size limit

[emanuele-pirro]

Hi everyone,

For various reasons my source connector is trying to ingest a big-boy message.

INFO Error while retrieving records, treating as an empty poll. java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: io.grpc.netty.NettyClientTransport$3: Frame size 5435421 exceeds maximum: 4194304. (com.google.pubsub.kafka.source.CloudPubSubSourceTask)

The data size exceeds the maximum allowed by GRPC (about 4MB) so to allow the connector to ingest the message need to add the option .maxInboundMessageSize(eightMegabytes) to the GRPC channel builder.

Would it be useful to achieve this via configuration? I can contribute on that but I'm very busy lately so that would be released for our nephews :)

Thanks in advance.

[kamalaboulhosn]

I went ahead and made the default 10MB to match the maximum Pub/Sub message size. I don't think we need to make it a configurable option. Thanks for reporting the issue!

[emanuele-pirro]

Thank you! ;)