2.34-rc1: lsblk: malloc(): unsorted double linked list corrupted

[balducci]

hello

I'm getting this error at run time from lsblk from 2.34-rc1:

 ----8<----
 malloc(): unsorted double linked list corrupted
 Aborted
 ---->8----

The error isn't reproducible: I would say that it shows up ~50% of the
times.
Maybe worth to note: I couldn't trigger the error under gdb, even after
running the command many times (I guess there might be good reasons
for this)

Needless to say: no problem whatsoever when running lsblk from 2.33.2.

My specs follow:

 ----8<----
 bash:3> lsblk --version
 lsblk from util-linux 2.34-rc1
 ----8<----

 ----8<----
 bash:4> uname -sr
 Linux 5.0.10
 ----8<----

 ----8<----
 bash:5> /lib64/libc.so.6 
 GNU C Library (GNU libc) stable release version 2.29.
 Copyright (C) 2019 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.
 There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
 PARTICULAR PURPOSE.
 Compiled by GNU CC version 8.2.0.
 libc ABIs: UNIQUE IFUNC ABSOLUTE
 For bug reporting instructions, please see:
 <http://www.gnu.org/software/libc/bugs.html>.
 ----8<----

 ----8<----
 bash:9> gcc -v
 Using built-in specs.
 COLLECT_GCC=gcc
 COLLECT_LTO_WRAPPER=/opt/stow.d/versions/gcc-8.3.0/usr/lib64/gcc/x86_64-pc-linux-gnu/8.3.0/lto-wrapper
 Target: x86_64-pc-linux-gnu
 Configured with: /home/balducci/tmp/install-us-d/gcc-8.3.0/gcc-8.3.0/configure
 --prefix=/opt/stow.d/versions/gcc-8.3.0/usr
 --libdir=/opt/stow.d/versions/gcc-8.3.0/usr/lib64
 --libexecdir=/opt/stow.d/versions/gcc-8.3.0/usr/lib64
 --enable-shared --disable-bootstrap
 --enable-languages=c,c++,objc,fortran --enable-multilib
 Thread model: posix
 gcc version 8.3.0 (GCC) 
 ----8<----

 ----8<----
 configuration:
 ./configure \
             --prefix=/opt/stow.d/versions/util-linux-2.34-rc1/usr \
             --bindir=/opt/stow.d/versions/util-linux-2.34-rc1/bin \
             --sbindir=/opt/stow.d/versions/util-linux-2.34-rc1/sbin \
             --libdir=/opt/stow.d/versions/util-linux-2.34-rc1/lib64 \
             --disable-nls \
             --enable-partx \
             --enable-wall \
             --enable-write \
             --enable-libuuid \
             --enable-libblkid \
             --disable-login \
             --disable-sulogin \
             --disable-su \
             --disable-runuser \
             --disable-silent-rules
 ----8<----

thanks a lot
ciao
-gabriele

[kerolasa]

I tried to reproduce the problem after enabling address sanitizer:

export CC=clang
export CFLAGS='-fsanitize=address -ggdb'
export ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer

And without atexit() hook.

index 34a6cd9ca..b43392129 100644
--- a/misc-utils/lsblk.c
+++ b/misc-utils/lsblk.c
@@ -1824,7 +1824,7 @@ int main(int argc, char *argv[])
        setlocale(LC_ALL, "");
        bindtextdomain(PACKAGE, LOCALEDIR);
        textdomain(PACKAGE);
-       close_stdout_atexit();
+//     close_stdout_atexit();
 
        lsblk = &_ls;

But unfortunately all I can see is couple memory leaks.

$ ./lsblk >/dev/null 

=================================================================
==2268==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4152 byte(s) in 1 object(s) allocated from:
    #0 0x55ef13c38db1 in calloc (/home/src/util-linux/.libs/lt-lsblk+0x10adb1)
    #1 0x55ef13ca38b7 in ul_new_path /home/src/util-linux/lib/path.c:53:24
    #2 0x55ef13caeb31 in ul_new_sysfs_path /home/src/util-linux/lib/sysfs.c:47:24
    #3 0x55ef13c808e4 in initialize_device /home/src/util-linux/misc-utils/lsblk.c:1140:15
    #4 0x55ef13c7effc in devtree_get_device_or_new /home/src/util-linux/misc-utils/lsblk.c:1194:7
    #5 0x55ef13c7c961 in process_all_devices /home/src/util-linux/misc-utils/lsblk.c:1555:9
    #6 0x55ef13c798aa in main /home/src/util-linux/misc-utils/lsblk.c:2099:4
    #7 0x7fe6a3ac7ce2 in __libc_start_main (/usr/lib/libc.so.6+0x23ce2)

Indirect leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x55ef13c38db1 in calloc (/home/src/util-linux/.libs/lt-lsblk+0x10adb1)
    #1 0x55ef13caef18 in sysfs_blkdev_init_path /home/src/util-linux/lib/sysfs.c:93:9
    #2 0x55ef13caeb80 in ul_new_sysfs_path /home/src/util-linux/lib/sysfs.c:54:6
    #3 0x55ef13c808e4 in initialize_device /home/src/util-linux/misc-utils/lsblk.c:1140:15
    #4 0x55ef13c7effc in devtree_get_device_or_new /home/src/util-linux/misc-utils/lsblk.c:1194:7
    #5 0x55ef13c7c961 in process_all_devices /home/src/util-linux/misc-utils/lsblk.c:1555:9
    #6 0x55ef13c798aa in main /home/src/util-linux/misc-utils/lsblk.c:2099:4
    #7 0x7fe6a3ac7ce2 in __libc_start_main (/usr/lib/libc.so.6+0x23ce2)

Indirect leak of 19 byte(s) in 1 object(s) allocated from:
    #0 0x55ef13b7ea01 in __interceptor_strdup (/home/src/util-linux/.libs/lt-lsblk+0x50a01)
    #1 0x55ef13ca46ee in ul_path_set_dir /home/src/util-linux/lib/path.c:132:7
    #2 0x55ef13caedeb in sysfs_blkdev_init_path /home/src/util-linux/lib/sysfs.c:80:7
    #3 0x55ef13caeb80 in ul_new_sysfs_path /home/src/util-linux/lib/sysfs.c:54:6
    #4 0x55ef13c808e4 in initialize_device /home/src/util-linux/misc-utils/lsblk.c:1140:15
    #5 0x55ef13c7effc in devtree_get_device_or_new /home/src/util-linux/misc-utils/lsblk.c:1194:7
    #6 0x55ef13c7c961 in process_all_devices /home/src/util-linux/misc-utils/lsblk.c:1555:9
    #7 0x55ef13c798aa in main /home/src/util-linux/misc-utils/lsblk.c:2099:4
    #8 0x7fe6a3ac7ce2 in __libc_start_main (/usr/lib/libc.so.6+0x23ce2)

SUMMARY: AddressSanitizer: 4211 byte(s) leaked in 3 allocation(s).

Maybe something with number of block devices, or some other environmental aspect is required to trigger the bug and I simply do not have it. @balducci I wonder if you recompile the same way I did and perhaps corruption is reported easier to understand manner.

[balducci]

export CC=clang
export CFLAGS='-fsanitize=address -ggdb'
export ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer

after rebuilding with clang and CFLAGS='-fsanitize=address -ggdb' I
get the following, which might help in diagnosing the problem:

 util-linux:76> /opt/stow.d/versions/util-linux-2.34-rc1-DEBUG/bin/lsblk
 =================================================================
 ==32442==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000000350 at pc 0x000000537cb1 bp 0x7ffed0310cf0 sp 0x7ffed0310ce8
 READ of size 8 at 0x614000000350 thread T0
     #0 0x537cb0 in process_all_devices /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:1565:19
     #1 0x5347b9 in main /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:2089:4
     #2 0x7f6bb73e3b5a in __libc_start_main /home/balducci/tmp/install-us-d/glibc-2.29.d/glibc-2.29/csu/../csu/libc-start.c:308:16
     #3 0x420a19 in _start /home/balducci/tmp/install-us-d/glibc-2.29.d/glibc-2.29/csu/../sysdeps/x86_64/start.S:120

 0x614000000350 is located 272 bytes inside of 448-byte region [0x614000000240,0x614000000400)
 freed by thread T0 here:
     #0 0x4f3f38 in __interceptor_free /home/balducci/tmp/install-us-d/llvm-8.0.0.d/llvm-8.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:124:3
     #1 0x5497f5 in lsblk_unref_device /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk-devtree.c:120:3
     #2 0x54b775 in lsblk_devtree_remove_device /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk-devtree.c:372:2
     #3 0x537b21 in process_all_devices /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:1552:4
     #4 0x5347b9 in main /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:2089:4
     #5 0x7f6bb73e3b5a in __libc_start_main /home/balducci/tmp/install-us-d/glibc-2.29.d/glibc-2.29/csu/../csu/libc-start.c:308:16

 previously allocated by thread T0 here:
     #0 0x4f454f in calloc /home/balducci/tmp/install-us-d/llvm-8.0.0.d/llvm-8.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:155:3
     #1 0x548a69 in lsblk_new_device /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk-devtree.c:36:8
     #2 0x53a334 in devtree_get_device_or_new /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:1180:9
     #3 0x537984 in process_all_devices /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:1545:9
     #4 0x5347b9 in main /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:2089:4
     #5 0x7f6bb73e3b5a in __libc_start_main /home/balducci/tmp/install-us-d/glibc-2.29.d/glibc-2.29/csu/../csu/libc-start.c:308:16

 SUMMARY: AddressSanitizer: heap-use-after-free /home/balducci/tmp/install-us-d/util-linux-2.34-rc1.d/util-linux-2.34-rc1/misc-utils/lsblk.c:1565:19 in process_all_devices
 Shadow bytes around the buggy address:
   0x0c287fff8010: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
   0x0c287fff8020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
   0x0c287fff8030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
   0x0c287fff8040: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
   0x0c287fff8050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 =>0x0c287fff8060: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
   0x0c287fff8070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
   0x0c287fff8080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x0c287fff8090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x0c287fff80a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
   0x0c287fff80b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 Shadow byte legend (one shadow byte represents 8 application bytes):
   Addressable:           00
   Partially addressable: 01 02 03 04 05 06 07
   Heap left redzone:       fa
   Freed heap region:       fd
   Stack left redzone:      f1
   Stack mid redzone:       f2
   Stack right redzone:     f3
   Stack after return:      f5
   Stack use after scope:   f8
   Global redzone:          f9
   Global init order:       f6
   Poisoned by user:        f7
   Container overflow:      fc
   Array cookie:            ac
   Intra object redzone:    bb
   ASan internal:           fe
   Left alloca redzone:     ca
   Right alloca redzone:    cb
   Shadow gap:              cc
 ==32442==ABORTING

[karelzak]

Please, try also "LSBLK_DEBUG=all lsblk". It will provide more details.

[karelzak]

I probably see the problem...

process_all_devices() {
...
                        lsblk_devtree_remove_device(tr, dev);
                        goto next;
...
next:
                /* Let's be careful with number of open files */
                if (dev && dev->sysfs)
                        ul_path_close_dirfd(dev->sysfs);

We definitely need dev = NULL, after lsblk_devtree_remove_device().

[karelzak]

Fixed (I hope). Please, try the current git tree.

I guess you have a ram disk device, in this case the default exclude filter is applied and the device ignored.

[balducci]

Fixed (I hope). Please, try the current git tree.

AFAICS, that seems to have fixed things: no problems after running
lsblk 100 times

thank you very much

[karelzak]

Thanks for your report and testing!