systemd-journal-upload dies due to watchdog

[DSpeichert]

It seems like systemd-journal-upload is not properly notifying watchdog, which is configured in /lib/systemd/system/systemd-journal-upload.service:

WatchdogSec=3min

Nov 23 21:58:48 goweb1 systemd[1]: Started Journal Remote Upload Service.
Nov 23 22:01:48 goweb1 systemd[1]: systemd-journal-upload.service: Watchdog timeout (limit 3min)!
Nov 23 22:01:48 goweb1 systemd[1]: systemd-journal-upload.service: Main process exited, code=dumped, status=6/ABRT
Nov 23 22:01:48 goweb1 systemd[1]: systemd-journal-upload.service: Unit entered failed state.
Nov 23 22:01:48 goweb1 systemd[1]: systemd-journal-upload.service: Failed with result 'core-dump'.
Nov 23 22:57:35 goweb1 systemd[1]: Started Journal Remote Upload Service.
Nov 23 23:00:35 goweb1 systemd[1]: systemd-journal-upload.service: Watchdog timeout (limit 3min)!
Nov 23 23:00:35 goweb1 systemd[1]: systemd-journal-upload.service: Main process exited, code=dumped, status=6/ABRT
Nov 23 23:00:35 goweb1 systemd[1]: systemd-journal-upload.service: Unit entered failed state.
Nov 23 23:00:35 goweb1 systemd[1]: systemd-journal-upload.service: Failed with result 'core-dump'.

[root@goweb1 ~]# /lib/systemd/systemd-journal-upload --version
systemd 227
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN

[evverx]

@DSpeichert , could you attach the result of strace -s500 -p on the systemd-journal-upload pid?

[evverx]

Looks like a starvation of the watchdog timer.
Are there any timeouts here?

[heftig]

Sounds like journal-upload should be ported to curl_multi_perform in order to do non-blocking uploads. Or perhaps use curl_easy_* from another thread.

[DSpeichert]

@evverx Emailed you the link to strace log.

[evverx]

@DSpeichert , thanks.

poll, write, poll, write, ..., poll, write, +++ killed by SIGABRT (core dumped) +++
It's a Curl_poll.
So, you have a starvation of the watchdog timer.

A temporary solution: increase your WatchdogSec (or remove it).

[DSpeichert]

Same thing happens to systemd-journal-remote.service

[chaloulo]

PR #2923 should solve this issue.

[poettering]

Closing this as #2968 has been merged now.

[kaihendry]

This is still a problem for me on v230
http://s.natalian.org.s3.amazonaws.com/2016-06-28/systemd-journal-remote.export