Monitoring the EventLoop

[nei]

Guy everyone, I would like to know what you guys are doing to monitor the event loop. I have a few workers running my application which are just handling commands and I would like to have a way to enable like a DEBUG mode and start seen what the event loop is doing, which things are blocking it, etc.. Any idea?

[WyriHaximus]

Ok before I go into answering this, I love this question, doing a lot of SRE work helping teams out onto Kubernetes and mentoring them on how to monitor their application so thank you for asking this, it is a vital part of running your applications!

There is a lot in this question and I don't think there we will ever add a debug mode to the event loop. However, I also don't think that is technically what you are looking for. A few years I started working on https://github.com/reactphp-inspector to get insight into my ReactPHP projects. That is done by decorating the event loop to get event loop insights and doing some monkey patching and function/class hijacking for other metrics. Now, a few years later, I'm not sure if all those metrics still make sense, sure networking, memory, and such stats are very useful. So are the open stream counts and others at times.

As time passed by I've gotten more and more interested in where my HTTP calls are going, and status code rates for both client and server. How my database calls are performing, failing/success ratio, how are fibers/promises doing etc etc.

Some of those exist and some not, and with tracing Otel/EBPF tooling maturing coming in hot there is some tooling out there that can get you insights in how your application is doing. I've been having my eye on tracing for a while now and with fibers it is making it a lot of sense start looking at it.

To get back to your question, it is hard to determine what is blocking, that is a matter of your perspective. All the event loop does it take your streams and asks the kernel "my dude/lady if something needs to happen for this one, you let me know right?". It doesn't know protocol, encryption, who it comes from, and who it goes to, it frankly doesn't give a fuck. (Pardon my French.) It's also not its job, we only need us to tell us when we have some data and to queue writes to somewhere. This makes it incredibly fast, and IMHO the wrong place to figure out what is blocking/lagging. Blocking is often a standard PHP function doing some I/O or sleeping, so if that happens the entire loop halts and nothing happens until that call has completed.

Here are a few things you can do:

• Figure out what exactly you want. I know it sounds like a bounce back at your questions, but think about what you'd want to know and why. This is my list of bare minimum metrics I want to know:
  • CPU usage
  • Memory usage
  • CPU throttling
  • Network In/Out Traffic
  • For every HTTP server status code stats
  • Inflight HTTP requests/command bus handling/outgoing queries/HTTP requests/etc stats
• Have metrics, logs (, and tracing if you're up for that), easily accessible one way or the other. I'm using to centralized systems like Sentry/Loggly/etc but do what works best for you.
• Keep in mind that nothing is certain. Was talking with @clue recently about react/filesystem and the amount of time things can go slightly differently on a system level for just reading from a file is not something most high level developers are used to.
• Once you defined what you want to want, let me know and I'll have a look at how I can add that to ReactPHP Inspector or give you advise on how to tackle that without ReactPHP Inspector.
• Are you looking for this to monitor in production, or during local development?
• Are you hunting down a specific bug or want to make sure you are in the loop of what happens in your application?

P.S. Sorry that this turned a bit into my future ideas for ReactPHP Inspector, but this topic is dear to me and I want to see non-blocking PHP get better at it.

[bartvanhoutte]

As @WyriHaximus said, the need for this probably depends on what you want to achieve. As someone who has implemented graceful shutdown in a ReactPHP application I understand the need for having a bit more insight into what the event loop is doing. When implementing graceful shutdown you want to make sure you don't just kill the loop on a SIGTERM while you're in the middle of processing some data. Therefore I've been using a singleton that tracks how many coroutines are running. It also allows you to clean up after yourself on certain events. It looks something like this:

Tracking coroutines:

$coroutine = Application::coroutine()->start();

// do something...

$coroutine->finished();

Cleaning up:

$timer = Loop::periodicTimer(60, ...);

// Stop doing new stuff on StopEvent.
Application::on(StopEvent::class, fn(StopEvent $event) => Loop::cancelTimer($timer));

$database = ...;

// Remove everything on the loop that stops it from stopping automatically on KillEvent.
Application::on(KillEvent::class, fn(KillEvent $event) => $database->close());

Note that KillEvent is only dispatched when all coroutines have finished ($coroutine->finished()).

Anyway, when implementing something like this it would be useful to have some very basic information on what's happening. Even in production when there's some unexpected behavior it could be useful to get a tiny glimpse of what is going on underneath.

Basically, I'd want to know what the event loop is about to do, and have some form of backtrace to figure out why it's doing that. Therefore, I've added a backtrace to $coroutine. At any point in time, I could get all coroutines created through Application::coroutine() and see where they were created through the backtrace.

I think something similar might be useful for the event loop: how many objects are in the queue, what timers are present, and what streams have been added to the event loop. Include a small backtrace and you could pinpoint quite a bit of issues. I've ran into issues before using a third party library that adds something on the loop but doesn't provide the ability to clean it up, therefore hindering a clean shutdown. This could be a first step into debugging these situations more easily.

I'm also very interested in tracing and eBPF but simply having a function like Loop::state() that returns an array of callbacks, timers and streams with their respective backtrace is a lot simpeler. I can also imagine exposing the state through some socket so you can look into a running application.

As for finding out what's blocking and what not I don't have a particular solution. Try thinking about the hardware implications of everything you add to the event loop and if it's not I/O it might be better to make it non-blocking through forking a process or something similar. We ran into issues while calling json_decode on a large amount of JSON (we don't control the formatting of the source) where we didn't want a large memory spike and chose to use a JSON streaming parser with JSON pointers which is slower and thus was blocking the event loop, YMMV.