regression: crashing journal due to watchdog

[utezduyar]

The for (;;) loop in server_process_datagram might prevent journal
from feeding the watchdog if there is always something to receive in
the syslog socket. Potentially journald is restarted, applications
stall if the syslog socket is staying full....

I thought about fixing it by checking the watchdog on every iteration
of for (;;) by using watchdog_last, watchdog_period and feeding
watchdog if necessary but none of those properties are public.

Current rate limit check is done right before we store the message
(after we receive it, after we forward it to console, wall, kmsg). I
think it is too late.

Maybe the best approach is having a rate limit on sd-event
(sd-event-source) so we can map rate limit options in journald.conf to
journal's sd-event.

Thoughts?

PS: Moving the discussion off the mailing list to get it tracked.

[poettering]

Wouldn't it suffice if we simply bump the priority of the watchdog event source to something very high, so that it is always dispatched before anything else?

[poettering]

Hmm, ignore that, that made no sense, we always dispatch the watchdog event, in every single loop, if it's due, and we do so before anything else.

Not sure I grok the issue then. Can you elaborate how precisely this can happen?

[utezduyar]

Imagine there is always something to fetch at this fd,

systemd/src/journal/journald-server.c

Line 1171 in cde40ac

n = recvmsg(fd, &msghdr, MSG_DONTWAIT|MSG_CMSG_CLOEXEC);

.

In this case, the loop (

systemd/src/journal/journald-server.c

Line 1116 in cde40ac

for (;;) {

) will never quit. We will never return back to event processing where we feed the watchdog.

Guy with bad intention starts multiple processes, while (1) logger "hello" and it should be enough to clog the journal.

[poettering]

Can you check if PR #150 fixes the issue for you please, @utezduyar? Thank you!

[poettering]

@utezduyar if this fixes the issue for you this can be promptly merged!

[utezduyar]

@poettering Verified!