gadgettracermanager: Make gRPC buffer size configurable.

[eiffel-fl]

This commit adds flags and environment values to set gRPC buffer size rather than having it hardcoded.
The default value size is set to 16384 to improve horizontal scaling.

[flyth]

I think having this number configurable is a really good thing. At which point did it lead to issues? Can you please attach some data/benchmarks around it?

Also, this is about the buffer of events (in number of events, not number of bytes of event data) that we're holding before marshaling them, so it's got nothing to do at that stage with gRPC, yet (gRPC uses buffers on its own, so let's not confuse by calling this gRPC buffer (naming again, I know... hehe) - even though it eventually will get sent over gRPC). So I think eventBufferSize or eventBufferLen would be a better name.

Just fyi, sending looks something like this:

Event -> __Event Buffer__ -> Marshaling -> gRPC -> TCP send buffer -> wire

The buffer is mainly used to throw away events if the client is slow in reading the events (and thus the tcp write on the server is blocking), and it especially handles bursts at the start of a gadget run - it also kicks in if marshaling is too slow to keep up (if the CPU is under heavy load for example). It would be interesting to know if increasing the TCP write buffer would also help.

[eiffel-fl]

Hi.

I think having this number configurable is a really good thing. At which point did it lead to issues? Can you please attach some data/benchmarks around it?

As already discussed, everything is available here:
inspektor-gadget/inspektor-gadget-scaling#2
The first set of results is with #2146, the second with 811a541.

Also, this is about the buffer of events (in number of events, not number of bytes of event data) that we're holding before marshaling them, so it's got nothing to do at that stage with gRPC, yet (gRPC uses buffers on its own, so let's not confuse by calling this gRPC buffer (naming again, I know... hehe) - even though it eventually will get sent over gRPC). So I think eventBufferSize or eventBufferLen would be a better name.

Just fyi, sending looks something like this:

Event -> __Event Buffer__ -> Marshaling -> gRPC -> TCP send buffer -> wire

The whole problem here is that since the refactoring we do not have any document and/or figures which depicts the current architecture regarding of events are sent.
All our current figures are no more up to date.

The buffer is mainly used to throw away events if the client is slow in reading the events (and thus the tcp write on the server is blocking), and it especially handles bursts at the start of a gadget run - it also kicks in if marshaling is too slow to keep up (if the CPU is under heavy load for example). It would be interesting to know if increasing the TCP write buffer would also help.

I still have a 5% difference between how many events where generated by stress-ng and what is reported by Inspektor Gadget, this could be due to the TCP buffer size.
In any case, having hardcoded values for buffer is clearly something wrong and it will clearly lead to troubles as we clearly not are in a situation of "one size fits all".

Best regards.

[burak-ok]

From my side it looks good.

[eiffel-fl]

Thank you for the review!