Services using gRPC will break when hitting the max gRPC message size limit

[fmassot]

gRPC message default size limit (which I think was introduced in tonic 0.9) is set to 4MB.

With a lot of splits or indexes, a gRPC call to the metastore will return an error. For now, we saw a few examples happening on a quickwit cluster:

• a search request on tens thousand splits will hit the limit. The search request errors after calling list_splits (the merge planner too was concerned but we fixed the request to decrease the number of requested splits, see Add maturity date in split metadata and use it to filter mature splits for merges #3585)
• with an index with a lot of partitions, list_indexes_metadatas will error. This call is particularly critical.

[mingshun]

I have encounter this on search request. Is there a way to increase this limit?

[fulmicoton]

@mingshun We can fix it, but we need to make sure there are no underlying problem. That's a lot of splits.
How many splits/documents do you have?

[fmassot]

I have a PR that streams splits somewhere, I can revive it.