segfault and memory errors

[DyslexicAtheist]

dunst segfaults in latest debian sid packages and also when building from source (git clone) (I checked both the debian source package as well as the latest revision from git.)

The version from Debian is:

/usr/bin/dunst --version
Dunst - A customizable and lightweight notification-daemon 1.4.1 (2019-07-03)

Version from git:

/dunst --version
Dunst - A customizable and lightweight notification-daemon v1.4.1-24-gdfd6e76

I'm starting dunst from the build directory either using the latest git version or from debian tree (apt-get source dunst):

~/src/dunst/dunst -conf ~/.config/dunst/dunstrc

Dunst segfaults periodically (it gets automatically restarted when that happens by my WM), here is the list of core files reported by systemd:

sudo coredumpctl list dunst
TIME                            PID   UID   GID SIG COREFILE  EXE
Tue 2020-02-11 10:44:11 CET   47217  1000  1000   6 missing   /usr/bin/dunst
Tue 2020-02-11 10:44:11 CET   47229  1000  1000   6 missing   /usr/bin/dunst
Wed 2020-02-12 11:42:05 CET   75581  1000  1000   6 missing   /usr/bin/dunst
Wed 2020-02-12 11:42:06 CET   75586  1000  1000   6 missing   /usr/bin/dunst
Sun 2020-02-16 13:59:48 CET  525862  1000  1000   6 missing   /usr/bin/dunst
Sun 2020-02-16 13:59:49 CET  525874  1000  1000   6 missing   /usr/bin/dunst
Tue 2020-02-18 13:09:55 CET  679648  1000  1000   6 missing   /usr/bin/dunst
Tue 2020-02-18 13:09:56 CET  679658  1000  1000   6 missing   /usr/bin/dunst
Sat 2020-02-22 09:05:52 CET   79034  1000  1000   6 missing   /usr/bin/dunst
Sat 2020-02-22 09:05:52 CET   79039  1000  1000   6 missing   /usr/bin/dunst
Mon 2020-02-24 10:54:07 CET   95268  1000  1000   6 missing   /usr/bin/dunst
Mon 2020-02-24 10:54:07 CET   95280  1000  1000   6 missing   /usr/bin/dunst
Thu 2020-02-27 09:48:52 CET  390056  1000  1000   6 missing   /usr/bin/dunst
Thu 2020-02-27 09:48:52 CET  390061  1000  1000   6 missing   /usr/bin/dunst
Sat 2020-02-29 17:19:04 CET  978503  1000  1000   6 missing   /usr/bin/dunst
Sat 2020-02-29 17:19:05 CET  978508  1000  1000   6 missing   /usr/bin/dunst
Mon 2020-03-02 18:03:58 CET  820533  1000  1000   6 missing   /usr/bin/dunst
Mon 2020-03-02 18:03:58 CET  820545  1000  1000   6 missing   /usr/bin/dunst
Tue 2020-03-03 10:22:50 CET  888165  1000  1000   6 missing   /usr/bin/dunst
Tue 2020-03-03 10:22:50 CET  888177  1000  1000   6 missing   /usr/bin/dunst
Thu 2020-03-05 15:21:51 CET  1631565  1000  1000   6 present   /usr/bin/dunst
Thu 2020-03-05 15:21:51 CET  1631577  1000  1000   6 present   /usr/bin/dunst
Sat 2020-03-07 15:05:55 CET  2270537  1000  1000   6 present   /usr/bin/dunst
Sat 2020-03-07 15:05:56 CET  2270542  1000  1000   6 present   /usr/bin/dunst
Sun 2020-03-08 12:43:01 CET  205577  1000  1000   6 present   /usr/bin/dunst
Sun 2020-03-08 12:43:01 CET  205589  1000  1000   6 present   /usr/bin/dunst

I then built dunst with -g CFLAGS and used valgrind to report memory errors.

valgrind --tool=memcheck --vgdb-error=0 --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes --track-origins=yes ~/src/dunst/dunst -conf ~/.config/dunst/dunstrc  | tee ~/dunst-valgrind.log

valgrind log:

dunst-valgrind.log

My .config/dunst/dunstrc:

dunstrc.txt

happy to help patch some of these, just let me know. cheers

[bebehei]

Could you please send us the latest backtrace of the dunst crash?

Thanks for the valgrind log. There are two types of "errors" there:

• Memory leaks: Nothing critical, just a pile of garbage. Would be fixable, but the source of this evil is often another library.
• Open FDs: Yep, that's right. I might have introduced those errors. The backtrace would be super helpful here.

[DyslexicAtheist]

@bebehei the backtrace from the segfault is from the stripped binary from the deb package. There is no -g in that build so the info below is limited. I've installed from a debug build and will report back later with a better backtrace when it crashes again (still trying to reproduce).

coredumpctl dump 205589

       Message: Process 205589 (dunst) of user 1000 dumped core.
                
                Stack trace of thread 205589:
                #0  0x00007f12fde19f61 __GI_raise (libc.so.6 + 0x39f61)
                #1  0x00007f12fde05535 __GI_abort (libc.so.6 + 0x25535)
                #2  0x00007f12fde0540f __assert_fail_base (libc.so.6 + 0x2540f)
                #3  0x00007f12fde12b92 __GI___assert_fail (libc.so.6 + 0x32b92)
                #4  0x000055ba57a2cf51 n/a (dunst + 0xff51)
                #5  0x000055ba57a2d20c n/a (dunst + 0x1020c)
                #6  0x000055ba57a2cafe n/a (dunst + 0xfafe)
                #7  0x000055ba57a25a94 n/a (dunst + 0x8a94)
                #8  0x00007f12fde06bbb __libc_start_main (libc.so.6 + 0x26bbb)
                #9  0x000055ba57a230ca n/a (dunst + 0x60ca)

I see this error in /var/log/user.log just before the segfault:

Mar  8 12:43:01 org.freedesktop.Notifications[205577]: dunst: src/utils.c:29: string_replace_at
: Assertion `repl' failed.
Mar  8 12:43:01 org.freedesktop.Notifications[205589]: dunst: src/utils.c:29: string_replace_at
: Assertion `repl' failed.

[bebehei]

I see this error in /var/log/user.log just before the segfault:

Well, this is basically the segfault. After printing this message, assert() will die with a SIGABRT.

The most valuable info would be the value and the backtrace.

I've checked string_replace_all. Except of two calls of string_replace_at, all calls seem to be safe. The other two ones are too hard to verify. A traceable backtrace would be most helpful.

Thanks in advance.

[DyslexicAtheist]

From the latest crash:

gdb /usr/bin/dunst dunst.core 
...
Reading symbols from /usr/bin/dunst...
[New LWP 550673]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/dunst'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f128763e535 in __GI_abort () at abort.c:79
#2  0x00007f128763e40f in __assert_fail_base
    (fmt=0x7f12877a06e0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5582469fc127 "repl", file=0x5582469fc11b "src/utils.c", line=29, function=<optimized out>) at assert.c:92
#3  0x00007f128764bb92 in __GI___assert_fail
    (assertion=assertion@entry=0x5582469fc127 "repl", file=file@entry=0x5582469fc11b "src/utils.c", line=line@entry=29, function=function@entry=0x5582469fc1d0 <__PRETTY_FUNCTION__.11849> "string_replace_at")
    at assert.c:101
#4  0x00005582469f5c74 in string_replace_at
    (buf=buf@entry=0x558247013870 "~/bin/dunst_espeak", pos=pos@entry=0, len=len@entry=2, repl=repl@entry=0x0) at src/utils.c:29
#5  0x00005582469f5f3e in string_to_path (string=0x558247013870 "~/bin/dunst_espeak") at src/utils.c:143
#6  0x00005582469f29c5 in ini_get_path
    (section=section@entry=0x55824700e230 "espeak-timer", key=key@entry=0x5582469fb1c4 "script", def=def@entry=0x0) at src/option_parser.c:239
#7  0x00005582469f57ce in load_settings (cmdline_config_path=<optimized out>) at src/settings.c:780
#8  0x00005582469ee884 in dunst_main (argc=<optimized out>, argv=0x7fff45f369e8) at src/dunst.c:163
#9  0x00007f128763fbbb in __libc_start_main (main=
    0x5582469ec090 <main>, argc=1, argv=0x7fff45f369e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff45f369d8) at ../csu/libc-start.c:308
#10 0x00005582469ec0ca in _start ()
(gdb) 

[bebehei]

Your HOME-variable isn't set. ... Well I didn't expect this to happen.

I'm preparing a fix.

[bebehei]

@DyslexicAtheist Can you please build the commit from the linked PR #694 and test it? Thank you in advance.

[DyslexicAtheist]

thanks, I just installed from this branch and will report back. I tested env from parent process and the HOME is there, but then I noticed that when I run an apt-get upgrade there are hooks in the postinstall that might interfere with dbus not being available (haven't yet had time to dig in) and my guess is this is what caused the HOME being unset. Obviously a bit of an edge-case :) ... will get back with some results tomorrow.

[DyslexicAtheist]

This didn't solve the issue unfortunately. The crash still happens at the same place:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ff73d7d5535 in __GI_abort () at abort.c:79
#2  0x00007ff73d7d540f in __assert_fail_base
    (fmt=0x7ff73d9376e0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55f0c067e03f "repl", file=0x55f0c067e033 "src/utils.c", line=29, function=<optimized out>) at assert.c:92
#3  0x00007ff73d7e2b92 in __GI___assert_fail
    (assertion=assertion@entry=0x55f0c067e03f "repl", file=file@entry=0x55f0c067e033 "src/utils.c", line=line@entry=29, function=function@entry=0x55f0c067e0e0 <__PRETTY_FUNCTION__.11849> "string_replace_at")
    at assert.c:101
#4  0x000055f0c0677a14 in string_replace_at
    (buf=buf@entry=0x55f0c1ba0870 "~/bin/dunst_espeak", pos=pos@entry=0, len=len@entry=2, repl=repl@entry=0x0) at src/utils.c:29
#5  0x000055f0c0677cde in string_to_path (string=0x55f0c1ba0870 "~/bin/dunst_espeak") at src/utils.c:143
#6  0x000055f0c0674805 in ini_get_path
    (section=section@entry=0x55f0c1b9b230 "espeak-timer", key=key@entry=0x55f0c067d1a1 "script", def=def@entry=0x0) at src/option_parser.c:227
#7  0x000055f0c067756e in load_settings (cmdline_config_path=<optimized out>) at src/settings.c:759
#8  0x000055f0c06708a4 in dunst_main (argc=<optimized out>, argv=0x7ffe9e642a18) at src/dunst.c:163
#9  0x00007ff73d7d6bbb in __libc_start_main (main=
    0x55f0c066e080 <main>, argc=1, argv=0x7ffe9e642a18, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe9e642a08) at ../csu/libc-start.c:308
#10 0x000055f0c066e0ba in _start ()

[bebehei]

Are you sure, you've used the correct branch and you're using the freshly compiled executable?

[DyslexicAtheist]

you're right, I messed up ... still had the old binary on the path. Restarted now with the correct binary from the actual branch and will report back later ...

I can't even facepalm because people telling me to not touch my face.

[DyslexicAtheist]

I can confirm that this branch works. I have also found a way to reproduce. If I install a package such as gnome-books then the post install triggers from the install would cause the crash. It's solved on your branch! thanks for all your help and patience!