Enhancement/Question on event loop time

[pioardi]

Hi and thanks to create this amazing piece of software, I was wondering if is possible ( or if it is already in place ) to have a mechanism that logs a WARN message when an event loop cycle takes too much time and/or log the cycle time of the event loop.
This would help people using Netty to figure out when the event loop is used not appropriately, the cycle time may be also a metrics to push somewhere.

I am open to contribute on that but could need some guidance on the implementation.

Thanks in advance

[franz1981]

Hi @pioardi you can just provide your Runnable abstraction that could (with sampling; meaning that not every Runnable perform this metric) measure the time spent while it's executed.
It won't give you how much time is spent while executing application tasks, but it would save Netty to provide such functionality that can have a performance impact.

There's a GSoC in the past aiming to provide event loop lightweight metrics; if you're interested maybe we can think about resurrecting it...
and we love PRs if you wish to contribute on it :P

[pioardi]

Hei @franz1981 thanks for your quick reply, yes I think lightweight metrics would be nice to have with a possibility to enable and disable them so that who is interested can turn them on, do you have any starting point where I can start looking into ?

[franz1981]

Yes @pioardi

• https://github.com/netty/netty/wiki/Google-Summer-of-Code-Ideas-2020#implement-event-loop-telemetry
• Add Event loop queue latency telemetry #11293
• EventExecutorGroup lightweight progress metrics #9080

OT maybe could worth looking at https://www.dynatrace.com/news/blog/all-you-need-to-know-to-really-understand-the-node-js-event-loop-and-its-metrics/ as well

And IIRC https://github.com/eclipse-vertx/vert.x/blob/d711e50174f84a06744c855dd29222beaa97bdeb/src/main/java/io/vertx/core/impl/btc/BlockedThreadChecker.java VErt-x (nudge nudge @vietj ) is using this mechanism

I can help to implement something, wdyt @chrisvest @normanmaurer @NiteshKant ?

[pioardi]

Yep the vertx implementation is exactly what I have in mind, we may start small from that and then add the metrics later ?

[franz1981]

Yep the vertx implementation is exactly what I have in mind, we may start small from that and then add the metrics later ?

if that's cover your use case, this can be implemented outside Netty and I suggest to do so: the lightweight mechanisms in the GSoC are meant to capture things that can be captured only within Netty, while other mechanisms should use whatever application code is required to perform them

[franz1981]

Re

that can be captured only within Netty

I mean, for example, to expose hooks to report the time spent while performing I/O vs application code eg

netty/transport-classes-epoll/src/main/java/io/netty/channel/epoll/EpollEventLoop.java

Lines 383 to 391 in 86004b7

	final long ioStartTime = System.nanoTime();
	try {
	if (processReady(events, strategy)) {
	prevDeadlineNanos = NONE;
	}
	} finally {
	// Ensure we always run tasks.
	final long ioTime = System.nanoTime() - ioStartTime;
	runAllTasks(ioTime * (100 - ioRatio) / ioRatio);

These are not meant to be checks that can alert anyone and will be published when they happen and complete (because captured within the event loop); it's not meant to poll if an existing task is still blocking the event loop while it's happening, as the Vert-x checker.

[franz1981]

@violetagg too, anything that can be interesting for you as well?