[BUG] 3.8.1 and 3.8.0 crashing with signal 11

[hildeb]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Read about bug reporting in general: https://rspamd.com/doc/faq.html#how-to-report-bugs-found-in-rspamd
  • Enabled relevant debugging logs: https://rspamd.com/doc/faq.html#how-to-debug-some-module-in-rspamd
  • Checked the FAQs about Core files in case of fatal crash: https://rspamd.com/doc/faq.html#how-to-figure-out-why-rspamd-process-crashed
  • Tried ASAN package and obtained the ASAN report (if possible): https://rspamd.com/doc/faq.html#asan-builds
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=%E2%9C%93&q=is%3Aissue+user%3Arspamd
  • Checked that there is not already an experimental package or master branch

Describe the bug

rspamd crashes quite frequently with "rspamd_crash_sig_handler: caught fatal signal 11(Segmentation fault)"

# rspamd --version
Rspamd daemon version 3.8.1

CPU architecture x86_64; features: avx, sse2, sse3, ssse3, sse4.1, sse4.2, rdrand
Hyperscan enabled: TRUE
Jemalloc enabled: TRUE
LuaJIT enabled: TRUE (LuaJIT version: LuaJIT 2.1.1700008891)
ASAN enabled: FALSE
BLAS enabled: FALSE
Fasttext enabled: FALSE

This is what I have in mail.log:

Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; task; rspamd_worker_body_handler: accepted connection from 141.42.5.209 port 43222, task ptr: 000074C7E0090898
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; task; rspamd_task_load_message: loaded message from zstd compressed stream; compressed: 9201; uncompressed: 25314
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; task; rspamd_message_parse: loaded message; id: <ADR490000000571560500A6F015001EB1EEEAFCFBE9F307950A6@charite.de>; queue-id: <4TNgws1GNWz1Z2cC>; size: 25314; checksum: <1fdb035d90a8b0465e52218ade419b4e>
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; task; rspamd_mime_part_detect_language: detected part language: de
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; lua; settings.lua:391: <ADR490000000571560500A6F015001EB1EEEAFCFBE9F307950A6@charite.de> apply static settings whitelist_einkauf (id = 1898547127); from matched; priority high
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; lua; greylist.lua:219: skip greylisting for local networks and/or authorized users
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; lua; spf.lua:186: skip SPF checks for local networks and authorized users
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; task; dkim_symbol_callback: skip DKIM checks for local networks and authorized users
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; lua; dmarc.lua:353: skip DMARC checks as either SPF or DKIM were not checked
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; lua; once_received.lua:102: Skipping once_received for authenticated user or local network
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <44f36d>; lua; mime_types.lua:343: using special tables from user settings
Jan 29 09:11:14 mail-cbf-int rspamd[3125357]: <>; ; rspamd_crash_sig_handler: caught fatal signal 11(Segmentation fault), pid: 3125357, trace: 
Jan 29 09:11:17 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #0: [0x74c7e9ba6e59]: ./debian/build/flavor-release/src/./src/libserver/backtrace.cxx:31 in rspamd::log_backtrace()
Jan 29 09:11:18 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #1: [0x74c7e9b758df]: ./debian/build/flavor-release/src/./src/libserver/worker_util.c:1637 in rspamd_crash_sig_handler
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #2: [0x74c7e944251f]: /lib/x86_64-linux-gnu/libc.so.6 in __GI___sigaction
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #3: [0x74c7e9be09f0]: ./debian/build/flavor-release/src/./src/libstat/backends/redis_backend.cxx:155 in redis_stat_runtime<float, true>::process_tokens(_GPtrArray*) const
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #4: [0x74c7e9d8b3e5]: languages.cc:0 in lj_BC_FUNCC
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #5: [0x74c7e9d77758]: /usr/lib/rspamd/librspamd-server.so in lua_pcall
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #6: [0x74c7e9c26206]: ./debian/build/flavor-release/src/./src/lua/lua_redis.c:407 in lua_redis_push_data
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #7: [0x74c7e9cec48c]: ./debian/build/flavor-release/src/../contrib/hiredis/./contrib/hiredis/async.c:269 in __redisRunCallback
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #8: [0x74c7ea28c515]: ./debian/build/flavor-release/contrib/libev/./contrib/libev/ev.c:3809 in ev_invoke_pending
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #9: [0x74c7ea28fdee]: ./debian/build/flavor-release/contrib/libev/./contrib/libev/ev.c:4231 in ev_run
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #10: [0x5fc5e1d71866]: ./debian/build/flavor-release/src/./contrib/libev/ev.h:830 in ev_loop
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #11: [0x74c7e9b74f7f]: ./debian/build/flavor-release/src/./src/libserver/worker_util.c:1179 in rspamd_handle_child_fork
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #12: [0x5fc5e1d6f4ac]: ./debian/build/flavor-release/src/./src/rspamd.c:631 in spawn_worker_type
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #13: [0x5fc5e1d70001]: ./debian/build/flavor-release/src/./src/rspamd.c:734 in spawn_workers
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #14: [0x5fc5e1d613c6]: ./debian/build/flavor-release/src/./src/rspamd.c:1678 in main
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #15: [0x74c7e9429d8f]: ./csu/../sysdeps/nptl/libc_start_call_main.h:58 in __libc_start_call_main
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #16: [0x74c7e9429e3f]: ./csu/../csu/libc-start.c:392 in __libc_start_main_impl
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #17: [0x5fc5e1d61994]: /usr/bin/rspamd in _start
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; log_backtrace: #18: [0xffffffffffffffff]:  in 
Jan 29 09:11:19 mail-cbf-int rspamd[3125357]: <>; ; rspamd_crash_sig_handler: please see Rspamd FAQ to learn how to dump core files and how to fill a bug report
Jan 29 09:11:23 mail-cbf-int rspamd[3125274]: <8fc48d>; main; rspamd_control_broadcast_cmd: cannot write command 9(4080) to the worker 3125357(normal), fd: 255: Connection reset by peer
Jan 29 09:11:23 mail-cbf-int rspamd[3125274]: <8fc48d>; main; rspamd_control_broadcast_cmd: cannot write command 9(4080) to the worker 3125357(normal), fd: 255: Broken pipe
Jan 29 09:11:25 mail-cbf-int rspamd[3125274]: <8fc48d>; main; rspamd_check_termination_clause: normal process 3125357 terminated abnormally by signal: Segmentation fault and created core file; please see Rspamd FAQ to learn how to extract data from core file and fill a bug report
Jan 29 09:11:25 mail-cbf-int rspamd[3125274]: <8fc48d>; main; rspamd_cld_handler: respawn process normal in lieu of terminated process with pid 3125357

This is what the backtrace says:

Core was generated by `rspamd: normal process (141.42.5.208:11333)  '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000074c7e9be09f0 in redis_stat_runtime<float, true>::process_tokens (tokens=<optimized out>, this=0x74c7dfe74740) at ./src/libstat/backends/redis_backend.cxx:155
155	./src/libstat/backends/redis_backend.cxx: No such file or directory.
(gdb) 
#0  0x000074c7e9be09f0 in redis_stat_runtime<float, true>::process_tokens (tokens=<optimized out>, this=0x74c7dfe74740) at ./src/libstat/backends/redis_backend.cxx:155
        idx = <synthetic pointer>
        val = <synthetic pointer>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        tok = 0x74c7e00a0a40
#1  rspamd_redis_classified (L=0x74c7e82c8fb0) at ./src/libstat/backends/redis_backend.cxx:875
        filler_func = <optimized out>
        opposite_rt_maybe = std::optional<redis_stat_runtime<float, true> *> = {[contained value] = 0x74c7dfe74740}
        cookie = 0x74c7de2a2d28 "1e15faeafd7cefe"
        task = 0x74c7e0090898
        rt = 0x74c7e00a90a0
        result = <optimized out>
        __func__ = <optimized out>
        __PRETTY_FUNCTION__ = <optimized out>
#2  0x000074c7e9d8b3e6 in lj_BC_FUNCC () from /usr/lib/rspamd/librspamd-server.so
No symbol table info available.
#3  0x000074c7e9d77759 in lua_pcall () from /usr/lib/rspamd/librspamd-server.so
No symbol table info available.
#4  0x000074c7e9c26207 in lua_redis_push_data (sp_ud=0x74c7e014bf70, ctx=0x74c7e00ab940, r=0x74c7e0058f60) at ./src/lua/lua_redis.c:407
        err_idx = 1
        ret = <optimized out>
        ud = <optimized out>
        cbs = {L = 0x74c7e82c8fb0, my_thread = 0x74c7e5452480, previous_thread = 0x0, thread_pool = 0x74c7e54404e0}
        L = 0x74c7e82c8fb0
        ud = <optimized out>
        cbs = <optimized out>
        L = <optimized out>
        __func__ = <optimized out>
        err_idx = <optimized out>
        ret = <optimized out>
#5  lua_redis_callback (c=<optimized out>, r=0x74c7e0058f60, priv=0x74c7e014bf70) at ./src/lua/lua_redis.c:479
        reply = 0x74c7e0058f60
        sp_ud = 0x74c7e014bf70
        ctx = 0x74c7e00ab940
        ud = 0x74c7e00ab948
        ac = <optimized out>
        __func__ = "lua_redis_callback"
#6  0x000074c7e9cec48d in __redisRunCallback (reply=<optimized out>, cb=0x7ffef4ac87c0, ac=0x74c7df108a80) at ../contrib/hiredis/./contrib/hiredis/async.c:269
        c = 0x74c7df108a80
        c = <optimized out>
#7  redisProcessCallbacks (ac=0x74c7df108a80) at ../contrib/hiredis/./contrib/hiredis/async.c:470
        c = 0x74c7df108a80
        cb = {next = 0x0, fn = 0x74c7e9c26060 <lua_redis_callback(redisAsyncContext*, gpointer, gpointer)>, privdata = 0x74c7e014bf70}
        reply = 0x74c7e0058f60
        status = <optimized out>
#8  0x000074c7ea28c516 in ev_invoke_pending (loop=0x74c7e5498800) at ./contrib/libev/ev.c:3809
        p = <optimized out>
#9  0x000074c7ea28fdef in ev_run (loop=0x74c7e5498800, flags=flags@entry=0) at ./contrib/libev/ev.c:4231
No locals.
#10 0x00005fc5e1d71867 in ev_loop (flags=0, loop=<optimized out>) at ./contrib/libev/ev.h:830
No locals.
#11 start_worker (worker=0x74c7df0f3700) at ./src/worker.c:524
        ctx = 0x74c7e300d110
        is_controller = <optimized out>
        __func__ = "start_worker"
#12 0x000074c7e9b74f80 in rspamd_handle_child_fork (listen_sockets=<optimized out>, cf=0x74c7e54ba980, rspamd_main=0x74c7e545b3c0, wrk=0x74c7df0f3700)
    at ./src/libserver/worker_util.c:1179
        rlim = {rlim_cur = 104857600, rlim_max = 104857600}
        k = 0xda10b567fa5e638c
        rc = <optimized out>
        it = {dummy1 = 0x74c7e5426580, dummy2 = 0x0, dummy3 = 0x4, dummy4 = 8, dummy5 = 29895, dummy6 = 0x6}
        v = 0x74c7e0476b00
        cur = <optimized out>
        rc = <optimized out>
        rlim = <optimized out>
        __func__ = <optimized out>
        it = <optimized out>
        k = <optimized out>
        v = <optimized out>
        cur = <optimized out>
        elt = <optimized out>
        our = <optimized out>
        cur = <optimized out>
        ls = <optimized out>
        ls = <optimized out>
        listen_conf_stringified = <optimized out>
        cur_conf = <optimized out>
#13 rspamd_fork_worker (rspamd_main=rspamd_main@entry=0x74c7e545b3c0, cf=cf@entry=0x74c7e54ba980, index=index@entry=0, ev_base=ev_base@entry=0x74c7ea295060 <default_loop_struct>, 
    term_handler=term_handler@entry=0x5fc5e1d6f5a0 <rspamd_cld_handler>, listen_sockets=<optimized out>) at ./src/libserver/worker_util.c:1296
        wrk = 0x74c7df0f3700
        __func__ = "rspamd_fork_worker"
#14 0x00005fc5e1d6f4ad in spawn_worker_type (rspamd_main=rspamd_main@entry=0x74c7e545b3c0, event_loop=event_loop@entry=0x74c7ea295060 <default_loop_struct>, 
    cf=cf@entry=0x74c7e54ba980) at ./src/rspamd.c:631
        i = 0
        __func__ = "spawn_worker_type"
#15 0x00005fc5e1d70002 in spawn_workers (rspamd_main=0x74c7e545b3c0, ev_base=0x74c7ea295060 <default_loop_struct>) at ./src/rspamd.c:734
        cur = 0x74c7e2a56a60
        ls = <optimized out>
        cf = <optimized out>
        p = <optimized out>
        key = <optimized out>
        bcf = <optimized out>
        listen_ok = <optimized out>
        seen_mandatory_workers = 0x74c7e0476a60
        cw = <optimized out>
        wrk = <optimized out>
        i = <optimized out>
        __func__ = "spawn_workers"
        it = <optimized out>
        k = <optimized out>
        v = <optimized out>
        cur = <optimized out>
        elt = <optimized out>
        our = <optimized out>
        cur = <optimized out>
        ls = <optimized out>
        ls = <optimized out>
        listen_conf_stringified = <optimized out>
        cur_conf = <optimized out>
#13 rspamd_fork_worker (rspamd_main=rspamd_main@entry=0x74c7e545b3c0, cf=cf@entry=0x74c7e54ba980, index=index@entry=0, ev_base=ev_base@entry=0x74c7ea295060 <default_loop_struct>, 
    term_handler=term_handler@entry=0x5fc5e1d6f5a0 <rspamd_cld_handler>, listen_sockets=<optimized out>) at ./src/libserver/worker_util.c:1296
        wrk = 0x74c7df0f3700
        __func__ = "rspamd_fork_worker"
#14 0x00005fc5e1d6f4ad in spawn_worker_type (rspamd_main=rspamd_main@entry=0x74c7e545b3c0, event_loop=event_loop@entry=0x74c7ea295060 <default_loop_struct>, 
    cf=cf@entry=0x74c7e54ba980) at ./src/rspamd.c:631
        i = 0
        __func__ = "spawn_worker_type"
#15 0x00005fc5e1d70002 in spawn_workers (rspamd_main=0x74c7e545b3c0, ev_base=0x74c7ea295060 <default_loop_struct>) at ./src/rspamd.c:734
        cur = 0x74c7e2a56a60
        ls = <optimized out>
        cf = <optimized out>
        p = <optimized out>
        key = <optimized out>
        bcf = <optimized out>
        listen_ok = <optimized out>
        seen_mandatory_workers = 0x74c7e0476a60
        cw = <optimized out>
        wrk = <optimized out>
        i = <optimized out>
        __func__ = "spawn_workers"
--Type <RET> for more, q to quit, c to continue without paging--
#16 0x00005fc5e1d613c7 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at ./src/rspamd.c:1678
        i = <optimized out>
        res = 0
        signals = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
        sigpipe_act = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {4096, 5951023142701029933, 5598183095033931014, 9355839342945317334, 
              5891028331416553666, 0, 128401893860304, 604720007, 128401965882864, 0, 128401965882638, 1048576, 128401893860288, 0, 128401893658816, 128401893860288}}, sa_flags = 0, 
          sa_restorer = 0x2}
        pworker = <optimized out>
        type = <optimized out>
        control_addr = 0x74c7e08c8550
        event_loop = 0x74c7ea295060 <default_loop_struct>
        rspamd_main = 0x74c7e545b3c0
        skip_pid = 1
        control_signals = {__val = {2561, 0, 431415714926, 0, 0, 0, 537455027677187, 4785074604206295, 5396728617009438821, 3240954439389521891, 2271212543720515806, 
            4620546085695470067, 10999170941500810363, 15121281034861957069, 3100079126407513894, 16182935362994651894}}
        __func__ = "main"
        stat_update_time = <optimized out>
(gdb) 

Steps to Reproduce

This happens in 3.8.0 an 3.8.1, 3.7.5-28c86c16jammy works OK!

[pschichtel]

this is a duplicate of #4787, isn't it?

[hildeb]

I tend to agree 👍 Phillip Schichtel ***@***.***> schrieb am So., 11. Feb. 2024, 17:41:

…

this is a duplicate of #4787 <#4787>, isn't it? — Reply to this email directly, view it on GitHub <#4799 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASQLWS3AMAXKKZX523X5ULYTDYCRAVCNFSM6AAAAABCO75JGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXHAYDIMZYGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[vstakhov]

And we still cannot reproduce it, so I don't know how to fix unreproducible issues tbh.

[hildeb]

Is there any particular place in my config I could look at? Or maybe deleting all of Reddit storage might help.? Vsevolod Stakhov ***@***.***> schrieb am Mi., 14. Feb. 2024, 18:13:

…

And we still cannot reproduce it, so I don't know how to fix unreproducible issues tbh. — Reply to this email directly, view it on GitHub <#4799 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASQLWQ2FJSFWFKTHCUXNULYTTWEBAVCNFSM6AAAAABCO75JGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBUGI3DCNBRGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>