Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

freeradius 3.0.x crashes with "attempting double-free" / SIGABRT #3188

Open
dafeu opened this issue Dec 9, 2019 · 0 comments
Open

freeradius 3.0.x crashes with "attempting double-free" / SIGABRT #3188

dafeu opened this issue Dec 9, 2019 · 0 comments

Comments

@dafeu
Copy link

@dafeu dafeu commented Dec 9, 2019

Issue type

  • Defect - Crash or memory corruption.

Defect description

Our radius proxy crashes under load after a few minutes with "attempting double-free" (asan enabled) or SIGABRT (without asan). Logging does not show anything different before the crash.

How to reproduce issue

So far I was only able to reproduce the issue with our production load.
I tried to reproduce the issue with -X a while ago but the server drops a lot of requests and crashes much slower and that leads to so many lost requests in our production environment, that it's not an option.
Without -X, the server crashes fairly fast (a few minutes) and seems to work alright to that point.
Side note: we run a pair of radius proxies with freeradius 2.1.1 and they handle the load without crashes.
We reproduced the crashes with 3.0.15, 3.0.17, 3.0.18, 3.0.19 and 3.0.20.
I couldn't reproduce the crashes in single server mode.
commit 36656cc didn't fix the issue.

Output (3.0.20+dfsg-1) of ``[#!/bin/bash

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5
export ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer
export ASAN_OPTIONS="log_path=/tmp/freeradius_asan.log log_to_syslog=true detect_leaks=1 symbolize=1"
freeradius -f
]`` with showing issue occurring

=================================================================
==115708==ERROR: AddressSanitizer: attempting double-free on 0x60b0001bbab0 in thread T0:   
    #0 0x7f7ee2477fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
    #1 0x7f7ee1e825d2 in _tc_free_internal ../talloc.c:1201
    #2 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646
    #3 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163
    #4 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646
    #5 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163
    #6 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646
    #7 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163
    #8 0x7f7ee1e7d347 in _tc_free_children_internal ../talloc.c:1646
    #9 0x7f7ee1e7d347 in _tc_free_internal ../talloc.c:1163
    #10 0x7f7ee1e7d347 in _talloc_free_internal ../talloc.c:1227
    #11 0x7f7ee1e7d347 in _talloc_free ../talloc.c:1769
    #12 0x5594dc01a3cd in request_free src/main/process.c:602
    #13 0x5594dc01a3cd in request_free src/main/process.c:587
    #14 0x5594dc0206fc in request_done src/main/process.c:897
    #15 0x5594dc024ac0 in request_receive src/main/process.c:1765
    #16 0x5594dbfe7c3b in auth_socket_recv src/main/listen.c:1597
    #17 0x5594dc01a775 in event_socket_handler src/main/process.c:4869
    #18 0x7f7ee22b1988 in fr_event_loop src/lib/event.c:649
    #19 0x5594dc031734 in radius_event_process src/main/process.c:5954
    #20 0x5594dbfc7d33 in main src/main/radiusd.c:626
    #21 0x7f7ee197209a in __libc_start_main ../csu/libc-start.c:308
    #22 0x5594dbfc8979 in _start (/usr/sbin/freeradius+0x57979)

0x60b0001bbab0 is located 0 bytes inside of 111-byte region [0x60b0001bbab0,0x60b0001bbb1f)
freed by thread T0 here:
==115708==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_stackdepotbase.h:140 "((id & (((u32)-1) >> kReservedBits))) == ((id))" (0x6a15fa73, 0xea15fa73)
    #0 0x7f7ee2482fa5  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf3fa5)
    #1 0x7f7ee249df39 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ef39)
    #2 0x7f7ee24982af  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x1092af)
    #3 0x7f7ee23bc1ec  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2d1ec)
    #4 0x7f7ee23bd60d  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2e60d)
    #5 0x7f7ee23bdc2f  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2ec2f)
    #6 0x7f7ee247fee3  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf0ee3)
    #7 0x7f7ee23bacc4  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2bcc4)
    #8 0x7f7ee2477f8a in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8f8a)
    #9 0x7f7ee1e825d2 in _tc_free_internal ../talloc.c:1201
    #10 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646
    #11 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163
    #12 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646
    #13 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163
    #14 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646
    #15 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163
    #16 0x7f7ee1e7d347 in _tc_free_children_internal ../talloc.c:1646
    #17 0x7f7ee1e7d347 in _tc_free_internal ../talloc.c:1163
    #18 0x7f7ee1e7d347 in _talloc_free_internal ../talloc.c:1227
    #19 0x7f7ee1e7d347 in _talloc_free ../talloc.c:1769
    #20 0x5594dc01a3cd in request_free src/main/process.c:602
    #21 0x5594dc01a3cd in request_free src/main/process.c:587
    #22 0x5594dc0206fc in request_done src/main/process.c:897
    #23 0x5594dc024ac0 in request_receive src/main/process.c:1765
    #24 0x5594dbfe7c3b in auth_socket_recv src/main/listen.c:1597
    #25 0x5594dc01a775 in event_socket_handler src/main/process.c:4869
    #26 0x7f7ee22b1988 in fr_event_loop src/lib/event.c:649
    #27 0x5594dc031734 in radius_event_process src/main/process.c:5954
    #28 0x5594dbfc7d33 in main src/main/radiusd.c:626
    #29 0x7f7ee197209a in __libc_start_main ../csu/libc-start.c:308
    #30 0x5594dbfc8979 in _start (/usr/sbin/freeradius+0x57979)

Backtrace from coredumpctl info of the latest git code (2019-12-05):

           PID: 89504 (freeradius)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Fri 2019-12-06 12:21:43 CET (2 days ago)
  Command Line: freeradius -f
    Executable: /usr/sbin/freeradius
 Control Group: /user.slice/user-3319007.slice/session-2702.scope
          Unit: session-2702.scope
         Slice: user-3319007.slice
       Session: 2702
     Owner UID:  ***
       Boot ID: 70ef8a36d9f14fefa263dafa1af75ffe
    Machine ID: 441d7afd26e749d6821b9bc63173376a
      Hostname: radius5
       Storage: /var/lib/systemd/coredump/core.freeradius.0.70ef8a36d9f14fefa263dafa1af75ffe.89504.1575631303000000.lz4
       Message: Process 89504 (freeradius) of user 0 dumped core.
                
                Stack trace of thread 89504:
                #0  0x00007f6194f047bb __GI_raise (libc.so.6)
                #1  0x00007f6194eef535 __GI_abort (libc.so.6)
                #2  0x00007f61957e8a57 _fr_talloc_fault_simple (libfreeradius-radius.so)
                #3  0x00007f6195401b4c _tc_free_internal (libtalloc.so.2)
                #4  0x00007f6195401440 _tc_free_children_internal (libtalloc.so.2)
                #5  0x00007f6195401440 _tc_free_children_internal (libtalloc.so.2)
                #6  0x00007f61953fc348 _tc_free_children_internal (libtalloc.so.2)
                #7  0x0000561f683686df request_free (freeradius)
                #8  0x0000561f6836a1ce request_done (freeradius)
                #9  0x0000561f6836f91e request_receive (freeradius)
                #10 0x0000561f68310f57 auth_socket_recv (freeradius)
                #11 0x0000561f683850c8 event_socket_handler (freeradius)
                #12 0x00007f619585104f fr_event_loop (libfreeradius-radius.so)
                #13 0x0000561f683891a0 radius_event_process (freeradius)
                #14 0x0000561f6834f584 main (freeradius)
                #15 0x00007f6194ef109b __libc_start_main (libc.so.6)
                #16 0x0000561f682f864a _start (freeradius)
                
                Stack trace of thread 89508:
                #0  0x00007f6194fbb916 __GI_ppoll (libc.so.6)
                #1  0x00007f6191e955ee n/a (libnss_systemd.so.2)
                #2  0x00007f6191ea037c n/a (libnss_systemd.so.2)
                #3  0x00007f6191ea2baa _nss_systemd_getpwnam_r (libnss_systemd.so.2)
                #4  0x00007f6194f92777 __getpwnam_r (libc.so.6)
                #5  0x00007f6194f92118 getpwnam (libc.so.6)
                #6  0x00007f61959b44e2 __interceptor_getpwnam (libasan.so.5)
                #7  0x00007f6192306d9c mod_authorize (rlm_unix.so)
                #8  0x0000561f683391b0 call_modsingle (freeradius)
                #9  0x0000561f6833a6f6 modcall_recurse (freeradius)
                #10 0x0000561f683396d8 modcall_child (freeradius)
                #11 0x0000561f6833c192 modcall_recurse (freeradius)
                #12 0x0000561f6833ea3e modcall (freeradius)
                #13 0x0000561f6833492a indexed_modcall (freeradius)
                #14 0x0000561f683384a2 process_authorize (freeradius)
                #15 0x0000561f682fb5b0 rad_authenticate (freeradius)
                #16 0x0000561f6836ed55 request_running (freeradius)
                #17 0x0000561f68363392 request_handler_thread (freeradius)
                #18 0x00007f619531cfa3 start_thread (libpthread.so.0)
                #19 0x00007f6194fc64cf __clone (libc.so.6)
                       Stack trace of thread 90013:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 90011:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 89509:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 89511:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 90012:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 90008:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 90009:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 90010:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
                
                Stack trace of thread 89510:
                #0  0x00007f6195325896 futex_abstimed_wait_cancelable (libpthread.so.0)
                #1  0x00007f6195325988 __new_sem_wait_slow (libpthread.so.0)
                #2  0x0000561f68362ed9 request_handler_thread (freeradius)
                #3  0x00007f619531cfa3 start_thread (libpthread.so.0)
                #4  0x00007f6194fc64cf __clone (libc.so.6)
        
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.