Identify root cause for "maximum payload exceeded:"

[th0ger]

We have several pipelines running on the same ISB. All ISB pods started to regularly throw:

main [259] 2026/08/25 06:12:24.359109 [ERR] 10.42.4.236:53232 - cid:32117 - "v0.44.2:rust" - "js/user:******" - maximum payload exceeded: 5012888 vs 1048576

A pipeline is exceeding the default 1MB maximum message size, but we cant figure out which one. How can we do that?

We are not interested in raising the max_payload.

[th0ger]

It was as simple as 10.42.4.236 being the client IP of the offending vertex pod.

(Still need some tooling (metrics/histograms?) to mesure and inspect message sizes, but closing this for now.)

[adarsh0728]

https://numaflow.numaproj.io/user-guide/reference/configuration/max-message-size/#identify-oversized-messages

[th0ger]

Nice :-)
#3612