syslog-ng crashes with ivykis 0.42.1

[furiel]

I am trying to increase the version of ivykis in syslog-ng, but I ran into a crash. The exact root cause is not known, but I can steadily reproduce it by continuously reloading syslog-ng under load. Idle syslog-ng or without reload seems to work.

I did bisect on the ivykis source, which led me to this commit:
0985782: iv_event: Optimize out one mutex lock/unlock pair per event delivery iteration.

When I revert this commit on the top of 0.42.1, the crash disappears.

The commit (0985782) has two parts:

• an early return when the list of pending events is empty
• a seemingly equivalent transformation on the exit strategy of the while cycle that executes the pending events.

It seems only the second one causes the problem: when I revert only the second part ("int empty_now" related) but leaving the early return in, the crash also disappears.
Attached the diff I used that solves the crash:
revert_empty_now.diff.zip

Probably the empty_now variable is invalidated between the initialization and the usage after the handler, which did not cause problem in the previous version: it happend once in the condition of while(), during holding a lock.

When crash, I have the following backtrace.

(gdb) bt full
#0  0x00007ffeb23f6b10 in ?? ()
No symbol table info available.
#1  0x00007f9c3cc2a67a in __iv_event_run_pending_events (_st=0x1e8f420) at ../../../../syslog-ng/lib/ivykis/src/iv_event.c:56
        ie = 0x7ffeb23f6af8
        empty_now = 1
        st = 0x1e8f420
        events = {next = 0x7ffeb23f6b10, prev = 0x7ffeb23f6b10}
#2  0x00007f9c3cc2a78b in iv_event_run_pending_events () at ../../../../syslog-ng/lib/ivykis/src/iv_event.c:88
No locals.
#3  0x00007f9c3cc31235 in iv_fd_epoll_timerfd_poll (st=0x1e8f420, active=0x7ffeb23f6c70, abs=0x0)
    at ../../../../syslog-ng/lib/ivykis/src/iv_fd_epoll.c:484
        run_timers = 0
        batch = {{events = 4, data = {ptr = 0x1ed6b38, fd = 32336696, u32 = 32336696, u64 = 32336696}}, {events = 1, data = {
              ptr = 0x1e8f420, fd = 32044064, u32 = 32044064, u64 = 32044064}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, 
              u64 = 0}}, {events = 0, data = {ptr = 0x7f9c3cbdaa0d <_request_all_threads_to_exit+76>, fd = 1019062797, 
              u32 = 1019062797, u64 = 140309010688525}}, {events = 2990500768, data = {ptr = 0x3cbdaf3c00007ffe, fd = 32766, 
              u32 = 32766, u64 = 4376847085106921470}}, {events = 32668, data = {ptr = 0x7f9c3ce7ad00 <main_loop>, fd = 1021816064, 
              u32 = 1021816064, u64 = 140309013441792}}, {events = 1019057446, data = {ptr = 0xb23f6bd000007f9c, fd = 32668, 
              u32 = 32668, u64 = 12844103203381346204}}, {events = 32766, data = {
              ptr = 0x7f9c3cbd994d <main_loop_reload_config_initiate+362>, fd = 1019058509, u32 = 1019058509, u64 = 140309010684237}}}
        ret = 2
        run_events = 1
        i = 2
#4  0x00007f9c3cc2d2fa in iv_fd_poll_and_run (st=0x1e8f420, abs=0x1ee6138) at ../../../../syslog-ng/lib/ivykis/src/iv_fd.c:202
        active = {next = 0x1ed6b60, prev = 0x1ed6b60}
        run_timers = 1528
#5  0x00007f9c3cc2e781 in iv_main () at ../../../../syslog-ng/lib/ivykis/src/iv_main_posix.c:112
        _abs = {tv_sec = 140731888921808, tv_nsec = 140309010719332}
        abs = 0x1ee6138
        st = 0x1e8f420
        run_timers = 0
#6  0x00007f9c3cbda01f in main_loop_run (self=0x7f9c3ce7ad00 <main_loop>) at ../syslog-ng/lib/mainloop.c:531
No locals.
#7  0x0000000000401de4 in main (argc=1, argv=0x7ffeb23f6e38) at ../syslog-ng/syslog-ng/main.c:303

Could you help me with this issue?

[furiel]

Fixed with: efafd78
Thanks!