Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

imrelp prevents rsyslog from shutdown #4265

Open
kortemik opened this issue Apr 29, 2020 · 1 comment
Open

imrelp prevents rsyslog from shutdown #4265

kortemik opened this issue Apr 29, 2020 · 1 comment

Comments

@kortemik
Copy link
Contributor

kortemik commented Apr 29, 2020 •
edited

This issue was discovered when debugging #1071, it seems not to be related but still relevant.

Expected behavior

rsyslogd shutsdown after receiving sigterm

Actual behavior

rsyslogd stays up after receving sigterm and hangs

Steps to reproduce the behavior

kill $(pidof rsyslogd)

Environment

  • rsyslog version: 8.1910.0
  • platform: Fedora release 30 (Thirty)
  • for configuration questions/issues, include rsyslog.conf and included config files

[userone@hostnametwo ~]$ /opt/Path-Name/rsyslog/rsyslog/sbin/rsyslogd -v

rsyslogd  8.1910.0 (aka 2019.10) compiled with:
        PLATFORM:                               x86_64-pc-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              No
        FEATURE_DEBUG (debug build, slow code): Yes
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        systemd support:                        No
        Config file:                            /etc/rsyslog.conf
        PID file:                               /var/run/rsyslogd.pid
        Number of Bits in RainerScript integers: 64

See https://www.rsyslog.com for more information.

a.conf

$LocalHostName test
$AbortOnUncleanConfig on
$PreserveFQDN on
module(load="imrelp")
global(
    workDirectory="/home/userone"
    maxMessageSize="256k"
)
main_queue(queue.type="Direct")
input(type="imrelp" port="55601" ruleset="process" MaxDataSize="256k")

lookup_table(name="lookup" file="/home/userone/lookup.json" reloadOnHUP="on")

template(type="list" name="jsonmesg") {
     property(name="jsonmesg")
     constant(value="\n")
}

ruleset(name="process") {
    set $!key = lookup("lookup", $msg);
    action(type="omfile" file="/home/userone/output.log" template="jsonmesg")
}

/home/userone/lookup.json

{
  "version": 1,
  "nomatch": "unknown",
  "type": "string",
  "table": [
    {
      "index": "ping",
      "value": "pong"
    }
  ]
}

[userone@hostnametwo ~]$ /opt/Path-Name/rsyslog/rsyslog/sbin/rsyslogd -f a.conf -iNONE

[userone@hostnametwo ~]$ ps -ef |grep a.conf

userone  9388     1  0 14:24 ?        00:00:00 /opt/Path-Name/rsyslog/rsyslog/sbin/rsyslogd -f a.conf -iNONE

[userone@hostnametwo ~]$ gdb -p 9388

(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#1  0x0000556261fd18a6 in wait_timeout ()
#2  0x0000556261fd1904 in mainloop ()
#3  0x0000556261fd1d94 in main ()

[userone@hostnametwo ~]$ kill 9388

(gdb) step
Single stepping until exit from function select,
which has no line number information.

Thread 1 "rsyslogd" received signal SIGTERM, Terminated.
0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#1  0x0000556261fd18a6 in wait_timeout ()
#2  0x0000556261fd1904 in mainloop ()
#3  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function select,
which has no line number information.
0x0000556261fd1742 in rsyslogdDoDie ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fd1742 in rsyslogdDoDie ()
#1  <signal handler called>
#2  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#3  0x0000556261fd18a6 in wait_timeout ()
#4  0x0000556261fd1904 in mainloop ()
#5  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function rsyslogdDoDie,
which has no line number information.

Thread 1 "rsyslogd" received signal SIGTTIN, Stopped (tty input).
0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0 0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2 0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3 0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4 0x000055626204e367 in thrdStarter ()
#5 0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0 0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
#1 0x0000556261fd1862 in rsyslogdDoDie ()
#2
#3 0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#4 0x0000556261fd18a6 in wait_timeout ()
#5 0x0000556261fd1904 in mainloop ()
#6 0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function pthread_kill,
which has no line number information.
doSIGTTIN (sig=48) at imrelp.c:820
820     imrelp.c: No such file or directory.
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  doSIGTTIN (sig=48) at imrelp.c:820
#1  <signal handler called>
#2  0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
#3  0x0000556261fd1862 in rsyslogdDoDie ()
#4  <signal handler called>
#5  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#6  0x0000556261fd18a6 in wait_timeout ()
#7  0x0000556261fd1904 in mainloop ()
#8  0x0000556261fd1d94 in main ()

(gdb) step
821 in imrelp.c
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0 0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2 0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3 0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4 0x000055626204e367 in thrdStarter ()
#5 0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0 doSIGTTIN (sig=21) at imrelp.c:821
#1
#2 0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
#3 0x0000556261fd1862 in rsyslogdDoDie ()
#4
#5 0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#6 0x0000556261fd18a6 in wait_timeout ()
#7 0x0000556261fd1904 in mainloop ()
#8 0x0000556261fd1d94 in main ()

(gdb) step
822     in imrelp.c
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  doSIGTTIN (sig=21) at imrelp.c:822
#1  <signal handler called>
#2  0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
#3  0x0000556261fd1862 in rsyslogdDoDie ()
#4  <signal handler called>
#5  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#6  0x0000556261fd18a6 in wait_timeout ()
#7  0x0000556261fd1904 in mainloop ()
#8  0x0000556261fd1d94 in main ()

(gdb) step
825 in imrelp.c
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0 0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2 0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3 0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4 0x000055626204e367 in thrdStarter ()
#5 0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0 doSIGTTIN (sig=21) at imrelp.c:825
#1
#2 0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
#3 0x0000556261fd1862 in rsyslogdDoDie ()
#4
#5 0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#6 0x0000556261fd18a6 in wait_timeout ()
#7 0x0000556261fd1904 in mainloop ()
#8 0x0000556261fd1d94 in main ()

(gdb) step
0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00007fc2b94e0938 in pthread_kill () from /usr/lib64/libpthread.so.0
#1  0x0000556261fd1862 in rsyslogdDoDie ()
#2  <signal handler called>
#3  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#4  0x0000556261fd18a6 in wait_timeout ()
#5  0x0000556261fd1904 in mainloop ()
#6  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function pthread_kill,
which has no line number information.
0x0000556261fd1862 in rsyslogdDoDie ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0 0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2 0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3 0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4 0x000055626204e367 in thrdStarter ()
#5 0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0 0x0000556261fd1862 in rsyslogdDoDie ()
#1
#2 0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#3 0x0000556261fd18a6 in wait_timeout ()
#4 0x0000556261fd1904 in mainloop ()
#5 0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function rsyslogdDoDie,
which has no line number information.
0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00007fc2b8e51ffb in select () from /usr/lib64/libc.so.6
#1  0x0000556261fd18a6 in wait_timeout ()
#2  0x0000556261fd1904 in mainloop ()
```#3  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function select,
which has no line number information.
0x0000556261fd18a6 in wait_timeout ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fd18a6 in wait_timeout ()
#1  0x0000556261fd1904 in mainloop ()
```#2  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function wait_timeout,
which has no line number information.
0x0000556261fd1904 in mainloop ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fd1904 in mainloop ()
```#1  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function mainloop,
which has no line number information.
0x0000556261fd1d94 in main ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
```#0  0x0000556261fd1d94 in main ()

(gdb) step
Single stepping until exit from function main,
which has no line number information.

Thread 1 "rsyslogd" received signal SIGTTOU, Stopped (tty output).
0x00007fc2b8d9613b in kill () from /usr/lib64/libc.so.6
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00007fc2b8d9613b in kill () from /usr/lib64/libc.so.6
#1  0x0000556261fe1d90 in iminternalAddMsg ()
#2  0x0000556261fcfc08 in logmsgInternalSubmit ()
#3  0x0000556261fcfce5 in logmsgInternal ()
#4  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#5  0x00005562620184af in doLogMsg ()
#6  0x00005562620186b7 in LogMsg ()
```#7  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function kill,
which has no line number information.
0x0000556261fd05cf in hdlr_sigttin_ou ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fd05cf in hdlr_sigttin_ou ()
#1  <signal handler called>
#2  0x00007fc2b8d9613b in kill () from /usr/lib64/libc.so.6
#3  0x0000556261fe1d90 in iminternalAddMsg ()
#4  0x0000556261fcfc08 in logmsgInternalSubmit ()
#5  0x0000556261fcfce5 in logmsgInternal ()
#6  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#7  0x00005562620184af in doLogMsg ()
#8  0x00005562620186b7 in LogMsg ()
```#9  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function hdlr_sigttin_ou,
which has no line number information.
0x00007fc2b8d9613b in kill () from /usr/lib64/libc.so.6
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00007fc2b8d9613b in kill () from /usr/lib64/libc.so.6
#1  0x0000556261fe1d90 in iminternalAddMsg ()
#2  0x0000556261fcfc08 in logmsgInternalSubmit ()
#3  0x0000556261fcfce5 in logmsgInternal ()
#4  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#5  0x00005562620184af in doLogMsg ()
#6  0x00005562620186b7 in LogMsg ()
```#7  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function kill,
which has no line number information.
0x0000556261fe1d90 in iminternalAddMsg ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fe1d90 in iminternalAddMsg ()
#1  0x0000556261fcfc08 in logmsgInternalSubmit ()
#2  0x0000556261fcfce5 in logmsgInternal ()
#3  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#4  0x00005562620184af in doLogMsg ()
#5  0x00005562620186b7 in LogMsg ()
```#6  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function iminternalAddMsg,
which has no line number information.
0x0000556261fcfc08 in logmsgInternalSubmit ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fcfc08 in logmsgInternalSubmit ()
#1  0x0000556261fcfce5 in logmsgInternal ()
#2  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#3  0x00005562620184af in doLogMsg ()
#4  0x00005562620186b7 in LogMsg ()
```#5  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function logmsgInternalSubmit,
which has no line number information.
0x0000556261fcfce5 in logmsgInternal ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fcfce5 in logmsgInternal ()
#1  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#2  0x00005562620184af in doLogMsg ()
#3  0x00005562620186b7 in LogMsg ()
```#4  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function logmsgInternal,
which has no line number information.
0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fcf9d7 in rsyslogd_submitErrMsg ()
#1  0x00005562620184af in doLogMsg ()
#2  0x00005562620186b7 in LogMsg ()
```#3  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function rsyslogd_submitErrMsg,
which has no line number information.
0x00005562620184af in doLogMsg ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00005562620184af in doLogMsg ()
#1  0x00005562620186b7 in LogMsg ()
```#2  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function doLogMsg,
which has no line number information.
0x00005562620186b7 in LogMsg ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x00005562620186b7 in LogMsg ()
```#1  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function LogMsg,
which has no line number information.
0x0000556261fd1db4 in main ()
(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0  0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2  0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3  0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4  0x000055626204e367 in thrdStarter ()
#5  0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6  0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0  0x0000556261fd1db4 in main ()

(gdb) step
Single stepping until exit from function main,
which has no line number information.

################# STUCK #################

[userone@hostnametwo ~]$ kill -2 9388

Thread 1 "rsyslogd" received signal SIGINT, Interrupt.
0x00007fc2b94da9f8 in __pthread_timedjoin_ex () from /usr/lib64/libpthread.so.0

(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0 0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2 0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3 0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4 0x000055626204e367 in thrdStarter ()
#5 0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0 0x00007fc2b94da9f8 in __pthread_timedjoin_ex () from /usr/lib64/libpthread.so.0
#1 0x000055626204e191 in thrdTerminate ()
#2 0x000055626204dcab in thrdDestruct ()
#3 0x00005562620114c2 in llDestroyElt ()
#4 0x0000556262011585 in llDestroy ()
#5 0x000055626204e1b1 in thrdTerminateAll ()
#6 0x0000556261fd19fb in deinitAll ()
#7 0x0000556261fd1db9 in main ()

(gdb) step

################# STUCK #################

[userone@hostnametwo ~]$ kill -2 9388

Thread 1 "rsyslogd" received signal SIGINT, Interrupt.
0x00007fc2b94da9f8 in __pthread_timedjoin_ex () from /usr/lib64/libpthread.so.0

(gdb) thread apply all bt

Thread 2 (Thread 0x7fc2b895d700 (LWP 9389)):
#0 0x00007fc2b8e5a88e in epoll_wait () from /usr/lib64/libc.so.6
#1 0x00007fc2b94ad642 in engineEventLoopRun (pThis=pThis@entry=0x556263b6efd0) at relp.c:832
#2 0x00007fc2b94adfd2 in relpEngineRun (pThis=0x556263b6efd0) at relp.c:1021
#3 0x00007fc2b89629fd in runInput (pThrd=0x556263b6fd90) at imrelp.c:848
#4 0x000055626204e367 in thrdStarter ()
#5 0x00007fc2b94d94c0 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007fc2b8e5a553 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7fc2b8972dc0 (LWP 9388)):
#0 0x00007fc2b94da9f8 in __pthread_timedjoin_ex () from /usr/lib64/libpthread.so.0
#1 0x000055626204e191 in thrdTerminate ()
#2 0x000055626204dcab in thrdDestruct ()
#3 0x00005562620114c2 in llDestroyElt ()
#4 0x0000556262011585 in llDestroy ()
#5 0x000055626204e1b1 in thrdTerminateAll ()
#6 0x0000556261fd19fb in deinitAll ()
#7 0x0000556261fd1db9 in main ()
@kortemik kortemik changed the title related to #1071 imrelp prevents rsyslog from shutdown Apr 29, 2020
@rgerhards
Copy link
Member

can you re-check with the current version (including librelp). IIRC there was a related fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kortemik @rgerhards