Intermittent SIGSEGV during hot reload via SIGHUP(Fluent-bit v4.0.12)

[wangshuai809]

When sending a SIGHUP signal to trigger a hot reload of Fluent Bit (flb v4.0.12), an intermittent SIGSEGV (segmentation fault) occurs and fluent-bit process stop.
The issue appears sporadically during hot reload. Sometimes hot reload is success via SIGHUP.

cfg：

[INPUT]
    Name tail
    Path /export/kubelet/pods/xxx/volumes/kubernetes.io~lvm/export/Logs/xxxx/log-info.log
    Tag output_xxx_xxx.caf1afdf5e2feceb1e3730c8a729bd99
    Buffer_Chunk_Size 512k
    Buffer_Max_Size 1MB
    DB /opt/export/log-agent-pro/offset/xxx/xxx.db
    Mem_Buf_Limit 10MB
    Exclude_Path *.tar,*.gz,*.tgz,*.bz2,*.bz,*.Z,*.zip,*.rar
    multiline.parser multiline-regex-test
    Read_from_Head Off 
    Rotate_Wait 300
    Path_key path
    Key log
    Skip_Long_Lines On

Debug Log

[2025/11/19 14:21:46] [debug] [input:tail:tail.629] inode=131308 removing file name /export/xxx/common-error.log
[2025/11/19 14:21:46] [ info] [input:tail:tail.629] inotify_fs_remove(): inode=131308 watch_fd=1
[2025/11/19 14:21:46] [debug] [input:tail:tail.624] inode=1441801 removing file name /export/xxx/log-info.log
[2025/11/19 14:21:46] [engine] caught signal (SIGSEGV)
#0  0x48a3cc            in  c() at src/flb_input_chunk.c:1767
#1  0x48a9fa            in  flb_input_chunk_append_raw() at src/flb_input_chunk.c:2016
#2  0x4de7d0            in  input_log_append() at src/flb_input_log.c:71
#3  0x4de85f            in  flb_input_log_append() at src/flb_input_log.c:90
#4  0x52956d            in  ml_stream_buffer_flush() at plugins/in_tail/tail_file.c:414
#5  0x529610            in  ml_flush_callback() at plugins/in_tail/tail_file.c:992
#6  0x4bfc5e            in  flb_ml_flush_stream_group() at src/multiline/flb_ml.c:1519
#7  0x4c03c5            in  flb_ml_flush_parser_instance() at src/multiline/flb_ml.c:117
#8  0x4e9178            in  flb_ml_stream_id_destroy_all() at src/multiline/flb_ml_stream.c:316
#9  0x52a5dc            in  flb_tail_file_remove() at plugins/in_tail/tail_file.c:1329
#10 0x52a8e4            in  flb_tail_file_remove_all() at plugins/in_tail/tail_file.c:1401
#11 0x524448            in  in_tail_exit() at plugins/in_tail/tail.c:514
#12 0x486f19            in  flb_input_exit_all() at src/flb_input.c:1484
#13 0x4a2454            in  flb_engine_shutdown() at src/flb_engine.c:1227
#14 0x4a4793            in  flb_engine_start() at src/flb_engine.c:1107
#15 0x47b0f3            in  flb_lib_worker() at src/flb_lib.c:835
#16 0x7f28e28c3ea4      in  ???() at ???:0
#17 0x7f28e0f73b0c      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0

Do you have any suggestions for me？
Thanks

[patrick-stephens]

Does it happen with a more recent version?

The 4.0 series is EOL in December: https://github.com/fluent/fluent-bit/security/policy

[wangshuai809]

更新的版本也会出现这种情况吗？

4.0 系列将于 12 月停止维护：https://github.com/fluent/fluent-bit/security/policy

@patrick-stephens Thank you for your reply.

I attempted to upgrade flb to 4.2.0, but SIGSEGV still occurred occasionally during hot reload.

debug log as:

[2025/11/21 12:36:39.536960961] [debug] [task] task_id=18 reached retry-attempts limit 1/1
[2025/11/21 12:36:39.536969239] [error] [engine] chunk '90-1763699793.170768157.flb' cannot be retried: task_id=18, input=tail.552 > output=logger.0
[2025/11/21 12:36:39.538805934] [debug] [task] task_id=22 reached retry-attempts limit 1/1
[2025/11/21 12:36:39.538813328] [error] [engine] chunk '90-1763699793.856566165.flb' cannot be retried: task_id=22, input=tail.552 > output=logger.0


xxxxxx(other log)


[2025/11/21 12:36:39.864406040] [debug] [input:tail:tail.657] inode=4170 removing file name /export/kubelet/pods/xxxxxx/dal/xxx-error.log
[2025/11/21 12:36:39.864447024] [ info] [input:tail:tail.657] inotify_fs_remove(): inode=4170 watch_fd=1

[2025/11/21 12:36:39.864806717] [debug] [input:tail:tail.656] inode=92 removing file name /export/kubelet/pods/1xxxxx/logs/catxxxnaxx.out
[2025/11/21 12:36:39.864828124] [ info] [input:tail:tail.656] inotify_fs_remove(): inode=92 watch_fd=1

[2025/11/21 12:36:39.880687142] [debug] [input:tail:tail.645] inode=101 removing file name /export/kubelet/pods/40486xxxx7be43005d513/xxxxxx.log
[2025/11/21 12:36:39.880742756] [ info] [input:tail:tail.645] inotify_fs_remove(): inode=101 watch_fd=1

[2025/11/21 12:36:39.884668105] [debug] [input:tail:tail.642] inode=133780 removing file name /export/kubelet/pods/3xxxxx-sxxxk-b-task/common-error.log
[2025/11/21 12:36:39] 
[engine] caught signal (SIGSEGV)

#0  0x4956ec            in  input_chunk_append_raw() at src/flb_input_chunk.c:2852
#1  0x495d3d            in  flb_input_chunk_append_raw() at src/flb_input_chunk.c:3115
#2  0x4f264f            in  input_log_append() at src/flb_input_log.c:1390
#3  0x4f418f            in  flb_input_log_append() at src/flb_input_log.c:1421
#4  0x5404ed            in  ml_stream_buffer_flush() at plugins/in_tail/tail_file.c:415
#5  0x540590            in  ml_flush_callback() at plugins/in_tail/tail_file.c:1124
#6  0x4d3328            in  flb_ml_flush_stream_group() at src/multiline/flb_ml.c:1468
#7  0x4d3b65            in  flb_ml_flush_parser_instance() at src/multiline/flb_ml.c:119
#8  0x4fc758            in  flb_ml_stream_id_destroy_all() at src/multiline/flb_ml_stream.c:320
#9  0x541e8c            in  flb_tail_file_remove() at plugins/in_tail/tail_file.c:1461
#10 0x542194            in  flb_tail_file_remove_all() at plugins/in_tail/tail_file.c:1533
#11 0x53b3b8            in  in_tail_exit() at plugins/in_tail/tail.c:514
#12 0x4908e9            in  flb_input_exit_all() at src/flb_input.c:1555
#13 0x4ae133            in  flb_engine_shutdown() at src/flb_engine.c:1320
#14 0x4b07af            in  flb_engine_start() at src/flb_engine.c:1200
#15 0x483c93            in  flb_lib_worker() at src/flb_lib.c:904
#16 0x7fd2bde07ea4      in  ???() at ???:0
#17 0x7fd2bc4b7b0c      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0

[2025/11/21 12:36:40.61863141] [debug] [input:tail:tail.8] inode=655369, /export/xxx.log, events: IN_MODIFY 
[2025/11/21 12:36:40.68198290] [debug] [input:tail:tail.8] inode=1310752, /export/kubelet/pods/xxMP-Monitp.log, events: IN_MODIFY 
[2025/11/21 12:36:40.88172489] [debug] [input:tail:tail.8] inode=262161, /export/kubelet/xxx97_tp.log, events: IN_MODIFY 
[2025/11/21 12:36:40.304693229] [debug] [input:tail:tail.10] inode=567264663, /export/home/xxx_tp.log, events: IN_MODIFY 
[2025/11/21 12:36:40.709391566] [debug] [task] created task=0x7fc7682267c0 id=0 OK
[2025/11/21 12:36:40.711020571] [debug] [out flush] cb_destroy coro_id=617
[2025/11/21 12:36:40.711046272] [debug] [task] destroy task=0x7fc7682267c0 (task_id=0)

......

the result is that fluent-bit process stopped.

My daemon will restart fluent-bit after a few seconds.

Fluent Bit v4.2.0
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF graduated project under the Fluent organization
* https://fluentbit.io

______ _                  _    ______ _ _             ___   _____ 
|  ___| |                | |   | ___ (_) |           /   | / __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| | `' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| |   / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |_./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)_____/
                                                                  
             Fluent Bit v4.2 – Direct Routes Ahead
         Celebrating 10 Years of Open, Fluent Innovation!


[2025/11/21 12:37:09.707212641] [ info] Configuration:
[2025/11/21 12:37:09.707290423] [ info]  flush time     | 1.000000 seconds
[2025/11/21 12:37:09.707295946] [ info]  grace          | 5 seconds
[2025/11/21 12:37:09.707298099] [ info]  daemon         | 0
[2025/11/21 12:37:09.707300150] [ info] ___________
[2025/11/21 12:37:09.707301914] [ info]  inputs:
[2025/11/21 12:37:09.707304167] [ info]      tail
[2025/11/21 12:37:09.707306278] [ info]      tail
[2025/11/21 12:37:09.707308261] [ info]      tail
[2025/11/21 12:37:09.707310277] [ info]      tail
[2025/11/21 12:37:09.707312251] [ info]      tail
[2025/11/21 12:37:09.707314071] [ info]      tail
[2025/11/21 12:37:09.707315993] [ info]      tail

[patrick-stephens]

OK, thanks for confirming.

Can you follow the issue template to provide the details it includes, e.g. OS and target?