syslog-ng can crash with new ivykis (0.42) after reload #1672
Labels
Comments
|
Does this information help? It was an email I received from the author of
ivykis.
A syslog-ng user tried to use syslog-ng with ivykis 0.41 and ran into
the following issue:
buytenh/ivykis#11
In a nutshell, syslog-ng seems to be registering an ivykis timer
(iv_timer) with timeout tv_sec = 0, tv_nsec = 0, and ivykis >= 0.40
added support for timerfd_create, but due to a bug in ivykis, this
{0, 0} timeout would get passed into timerfd_settime() verbatim, and
timerfd_settime() interprets this zero timeout as a request to disable
the timerfd, which would cause timer processing to stop.
I will release a new ivykis release with a workaround for this, but I
figured that you'd probably want to know about this as well, as I don't
think it makes sense for syslog-ng to be registering timers with an
absolute timeout of zero.
FWIW, I just released ivykis 0.42 with the commit below, which fixes
this issue.
(And sorry, no, I don't have a syslog-ng backtrace, so I have no idea
which code in syslog-ng is registering a timer with an expiry time of
zero. :-( )
commit 580ea0f50c9ea1c537398bed1b1e3a861440c97f
Author: Lennert Buytenhek <buytenh@wantstofly.org>
Date: Wed Apr 26 22:28:20 2017 +0300
Fix ->set_poll_timeout() related bug with timers that expire at time
zero.
If the ivykis core decides to use ->set_poll_timeout() to offload the
handling of a timer, the epoll_create(2) / port_create(3C) poll
methods pass the given timeout directly into the timerfd_settime(2) /
timer_settime(3C) system calls, but if an ivykis user has registered
an iv_timer with an expiration time of (tv_sec, tv_nsec) = (0, 0),
the *_settime() system calls will interpret such a zero timeout value
as a request to disable the timer, and as a result, no timer handlers
will be run.
To work around this, we'll pass a timeout of (tv_sec, tv_nsec) = (0, 1)
into these system calls if the next expiring timer expires at (0, 0).
This bug was reported by and tracked down with the kind help of
Brian De Wolf (github: bldewolf), who ran into this issue with
syslog-ng 3.9.1.
Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
diff --git a/src/iv_fd_epoll.c b/src/iv_fd_epoll.c
index 5517d19..c6b2285 100644
--- a/src/iv_fd_epoll.c
+++ b/src/iv_fd_epoll.c
@@ -388,6 +388,8 @@ iv_fd_epoll_timerfd_set_poll_timeout(struct iv_state
*st,
val.it_interval.tv_sec = 0;
val.it_interval.tv_nsec = 0;
val.it_value = *abs;
+ if (val.it_value.tv_sec == 0 && val.it_value.tv_nsec == 0)
+ val.it_value.tv_nsec = 1;
ret = timerfd_settime(st->u.epoll.timer_fd,
TFD_TIMER_ABSTIME, &val, NULL);
diff --git a/src/iv_fd_port.c b/src/iv_fd_port.c
index 2d427b5..510c948 100644
--- a/src/iv_fd_port.c
+++ b/src/iv_fd_port.c
@@ -296,6 +296,8 @@ iv_fd_port_set_poll_timeout(struct iv_state *st, const
struct timespec *abs)
val.it_interval.tv_sec = 0;
val.it_interval.tv_nsec = 0;
val.it_value = *abs;
+ if (val.it_value.tv_sec == 0 && val.it_value.tv_nsec == 0)
+ val.it_value.tv_nsec = 1;
ret = timer_settime(st->u.port.timer_id, TIMER_ABSTIME, &val, NULL);
if (ret < 0) {
…--
Bazsi
On Thu, Sep 14, 2017 at 2:27 PM, furiel ***@***.***> wrote:
Example configuration:
@Version: 3.11
@include "scl.conf"
source s_local {
};
source s_network {
tcp(port(10514));
};
destination d_local {
file("/tmp/bbb");
};
log {
source(s_network);
destination(d_local);
};
After reload syslog-ng can crash.
One can reproduce by continous reloading, and sending messages with
syslog-ng. Idle syslog-ng does not trigger the crash. I tried with tcp
source and system source as well, the crash happend in both cases. The root
cause might be source independent.
while true; do ./syslog-ng-ctl reload; done;
../bin/loggen -P -s 145 -I 120 -r 10000 localhost 10514 --active-connections=10
The backtrace:
(gdb) bt
#0 0x00007ffd74d6c4c0 in ?? ()
#1 0x00007fcc9875567a in __iv_event_run_pending_events (_st=0x7023e0) at ../../../../syslog-ng/lib/ivykis/src/iv_event.c:56
#2 0x00007fcc9875578b in iv_event_run_pending_events () at ../../../../syslog-ng/lib/ivykis/src/iv_event.c:88
#3 0x00007fcc9875c227 in iv_fd_epoll_timerfd_poll (st=0x7023e0, active=0x7ffd74d6c670, abs=0x0)
at ../../../../syslog-ng/lib/ivykis/src/iv_fd_epoll.c:484
#4 0x00007fcc987582ec in iv_fd_poll_and_run (st=0x7023e0, abs=0x73ccd8) at ../../../../syslog-ng/lib/ivykis/src/iv_fd.c:202
#5 0x00007fcc98759773 in iv_main () at ../../../../syslog-ng/lib/ivykis/src/iv_main_posix.c:112
#6 0x00007fcc9870501f in main_loop_run (self=0x7fcc989a5d00 <main_loop>) at ../syslog-ng/lib/mainloop.c:531
#7 0x0000000000401de4 in main (argc=1, argv=0x7ffd74d6c838) at ../syslog-ng/syslog-ng/main.c:303
(gdb)
If I revert ivykis to the version that was used in syslog-ng previously:
ec9066dcc66b33e37, the crash does not occur.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1672>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AArldj-gpBJ5pmGRVvApaswRCsGw_dSGks5siRuwgaJpZM4PXgBn>
.
|
|
I'm afraid this will be another issue. That issue was fixed in 580ea0f50c9ea1c537398bed1b1e3a861440c97f, and this patch is included in 0.42. |
|
The bisect led to a commit in the code ivykis. I opened an issue to continue the investigation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Example configuration:
After reload syslog-ng can crash.
One can reproduce by continous reloading, and sending messages with syslog-ng. Idle syslog-ng does not trigger the crash. I tried with tcp source and system source as well, the crash happend in both cases. The root cause might be source independent.
The backtrace:
If I revert ivykis to the version that was used in syslog-ng previously: ec9066dcc66b33e37, the crash does not occur.
The text was updated successfully, but these errors were encountered: