Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

worker crashes (and dump backtrace) after fork #100

Closed
kyontan opened this issue Mar 12, 2024 · 9 comments
Closed

worker crashes (and dump backtrace) after fork #100

kyontan opened this issue Mar 12, 2024 · 9 comments

Comments

@kyontan
Copy link

kyontan commented Mar 12, 2024 •
edited

Hi again,
sorry for creating issues repeatedly 馃檹

At the same time of #99, we also see the errors below:
I'm not used to read this kinda traces, but it seems that the handler around fork for redis-client or concurrent_ruby is somehow weird?

Environment:

  • uname -a: Linux 5.10.210-201.852.amzn2.x86_64 ...
  • Distribution: Amazon Linux 2
  • Architecture: x86_64
  • Ruby: Ruby 3.2.2 (2023-03-30 revision e51014f9c0) + YJIT
  • gems
    • pitchfork: 0.11.0
    • redis-client: 0.19.0 (latest: 0.21.1)
    • concurrent_ruby: 1.2.2 (latest: 1.2.3)
    • (we are not sure why we are not using the latest gem, so we may try upgrading them if it relates)
Stacktrace

NOTE that the log is ordered by timestamp desc.

@timestamp,log
2024-03-12 01:02:14.835,
2024-03-12 01:02:14.835,
2024-03-12 01:02:14.834,
2024-03-12 01:02:14.834,* Process memory map:
2024-03-12 01:02:14.834,
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(rb_f_load) /usr/src/ruby/load.c:833
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x117) [0x7fe11041eaa7] /usr/src/ruby/vm_insnhelper.c:3268
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(vm_sendish+0xa3) [0x7fe11042eac4] /usr/src/ruby/vm_insnhelper.c:5080
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(vm_exec_core) /usr/src/ruby/insns.def:820
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(rb_vm_exec+0xb3) [0x7fe110434233] /usr/src/ruby/vm.c:2374
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(rb_ec_exec_node+0xa5) [0x7fe110249395] /usr/src/ruby/eval.c:289
2024-03-12 01:02:14.824,/usr/local/lib/libruby.so.3.2(ruby_run_node+0x8b) [0x7fe11024f85b] /usr/src/ruby/eval.c:330
2024-03-12 01:02:14.824,/usr/local/bin/ruby(rb_main+0x21) [0x5568d0437112] ./main.c:38
2024-03-12 01:02:14.824,/usr/local/bin/ruby(main) ./main.c:57
2024-03-12 01:02:14.824,/lib/x86_64-linux-gnu/libc.so.6(0x7fe10fd7524a) [0x7fe10fd7524a]
2024-03-12 01:02:14.824,/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fe10fd75305]
2024-03-12 01:02:14.824,[0x5568d0437151]
2024-03-12 01:02:14.824,
2024-03-12 01:02:14.824,-- Other runtime information -----------------------------------------------
2024-03-12 01:02:14.824,
2024-03-12 01:02:14.824,* Loaded script: /vendor/bundle/ruby/3.2.0/bin/pitchfork
2024-03-12 01:02:14.824,
2024-03-12 01:02:14.824,* Loaded features:
2024-03-12 01:02:14.824,
2024-03-12 01:02:14.824,    0 enumerator.so
2024-03-12 01:02:14.824,    1 thread.rb
2024-03-12 01:02:14.824,    2 fiber.so
2024-03-12 01:02:14.824,    3 rational.so
2024-03-12 01:02:14.824,    4 complex.so
2024-03-12 01:02:14.824,    5 ruby2_keywords.rb
...
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x117) [0x7fe11041eaa7] /usr/src/ruby/vm_insnhelper.c:3268
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_sendish+0x154) [0x7fe110424c54] /usr/src/ruby/vm_insnhelper.c:5080
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_exec_core+0xc80) [0x7fe11042f630] /usr/src/ruby/insns.def:801
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_vm_exec+0xb3) [0x7fe110434233] /usr/src/ruby/vm.c:2374
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(raise_load_if_failed+0x0) [0x7fe1102b4519] /usr/src/ruby/load.c:697
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_load_internal) /usr/src/ruby/load.c:760
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_f_load) /usr/src/ruby/load.c:833
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x117) [0x7fe11041eaa7] /usr/src/ruby/vm_insnhelper.c:3268
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_sendish+0xa3) [0x7fe11042eac4] /usr/src/ruby/vm_insnhelper.c:5080
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_exec_core) /usr/src/ruby/insns.def:820
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_vm_exec+0xb3) [0x7fe110434233] /usr/src/ruby/vm.c:2374
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(raise_load_if_failed+0x0) [0x7fe1102b4519] /usr/src/ruby/load.c:697
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_load_internal) /usr/src/ruby/load.c:760
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_f_load) /usr/src/ruby/load.c:833
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x117) [0x7fe11041eaa7] /usr/src/ruby/vm_insnhelper.c:3268
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_sendish+0xa3) [0x7fe11042eac4] /usr/src/ruby/vm_insnhelper.c:5080
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(vm_exec_core) /usr/src/ruby/insns.def:820
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_vm_exec+0x51c) [0x7fe11043469c] /usr/src/ruby/vm.c:2383
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(raise_load_if_failed+0x0) [0x7fe1102b4519] /usr/src/ruby/load.c:697
2024-03-12 01:02:14.823,/usr/local/lib/libruby.so.3.2(rb_load_internal) /usr/src/ruby/load.c:760
2024-03-12 01:02:14.822,"/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::vec::into_iter::IntoIter<usize, alloc::alloc::Global>>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497"
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(gen_entry_prologue) yjit/src/codegen.rs:631
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(gen_entry_point+0x7d) [0x7fe11045e27d] yjit/src/core.rs:1573
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_yjit_compile_iseq+0x3c) [0x7fe11045164c] /usr/src/ruby/yjit.c:1017
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(jit_exec+0x13) [0x7fe110434b59] /usr/src/ruby/vm.c:433
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_vm_exec) /usr/src/ruby/vm.c:2373
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_yield+0x1de) [0x7fe110438b2e] /usr/src/ruby/vm.c:1398
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(RB_FL_TEST_RAW+0x0) [0x7fe1101b0f8c] /usr/src/ruby/array.c:2740
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(RB_FL_ANY_RAW) ./include/ruby/internal/fl_type.h:552
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_array_len) ./include/ruby/internal/core/rarray.h:321
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_ary_each) /usr/src/ruby/array.c:2739
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x117) [0x7fe11041eaa7] /usr/src/ruby/vm_insnhelper.c:3268
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(vm_sendish+0x154) [0x7fe110424c54] /usr/src/ruby/vm_insnhelper.c:5080
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(vm_exec_core+0xc80) [0x7fe11042f630] /usr/src/ruby/insns.def:801
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_vm_exec+0xb3) [0x7fe110434233] /usr/src/ruby/vm.c:2374
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(catch_i+0x24b) [0x7fe110436adb] /usr/src/ruby/vm.c:1398
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(vm_catch_protect+0xe2) [0x7fe110427422] /usr/src/ruby/vm_eval.c:2393
2024-03-12 01:02:14.822,/usr/local/lib/libruby.so.3.2(rb_catch_obj+0x4c) [0x7fe1104275bc] /usr/src/ruby/vm_eval.c:2419
2024-03-12 01:02:14.821,/usr/local/lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x7fe110449c3d] /usr/src/ruby/vm_dump.c:785
2024-03-12 01:02:14.821,/usr/local/lib/libruby.so.3.2(rb_vm_bugreport) /usr/src/ruby/vm_dump.c:1080
2024-03-12 01:02:14.821,/usr/local/lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf0) [0x7fe110244140] /usr/src/ruby/error.c:813
2024-03-12 01:02:14.821,/usr/local/lib/libruby.so.3.2(sigsegv+0x4b) [0x7fe11039bf7b] /usr/src/ruby/signal.c:964
2024-03-12 01:02:14.821,/lib/x86_64-linux-gnu/libc.so.6(0x7fe10fd8a050) [0x7fe10fd8a050]
2024-03-12 01:02:14.821,/usr/lib/libjemalloc.so.2(0x7fe1107796b3) [0x7fe1107796b3]
2024-03-12 01:02:14.821,/usr/lib/libjemalloc.so.2(0x7fe11076a737) [0x7fe11076a737]
2024-03-12 01:02:14.821,/usr/local/lib/libruby.so.3.2(0x7fe11053f7c1) [0x7fe11053f7c1]
2024-03-12 01:02:14.783,
2024-03-12 01:02:14.783,
2024-03-12 01:02:14.782,
2024-03-12 01:02:14.782,* Process memory map:
2024-03-12 01:02:14.782,
2024-03-12 01:02:14.772,
2024-03-12 01:02:14.772,-- Other runtime information -----------------------------------------------
2024-03-12 01:02:14.772,
2024-03-12 01:02:14.772,* Loaded script: /vendor/bundle/ruby/3.2.0/bin/pitchfork
2024-03-12 01:02:14.772,
2024-03-12 01:02:14.772,* Loaded features:
2024-03-12 01:02:14.772,
2024-03-12 01:02:14.772,    0 enumerator.so
2024-03-12 01:02:14.772,    1 thread.rb
2024-03-12 01:02:14.772,    2 fiber.so
2024-03-12 01:02:14.772,    3 rational.so
2024-03-12 01:02:14.772,    4 complex.so
2024-03-12 01:02:14.772,    5 ruby2_keywords.rb
...
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x7fe110449c3d] /usr/src/ruby/vm_dump.c:785
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(rb_vm_bugreport) /usr/src/ruby/vm_dump.c:1080
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf0) [0x7fe110244140] /usr/src/ruby/error.c:813
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(sigsegv+0x4b) [0x7fe11039bf7b] /usr/src/ruby/signal.c:964
2024-03-12 01:02:14.771,/lib/x86_64-linux-gnu/libc.so.6(0x7fe10fd8a050) [0x7fe10fd8a050]
2024-03-12 01:02:14.771,/usr/lib/libjemalloc.so.2(0x7fe1107796b3) [0x7fe1107796b3]
2024-03-12 01:02:14.771,/usr/lib/libjemalloc.so.2(0x7fe11076a737) [0x7fe11076a737]
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(0x7fe110540239) [0x7fe110540239]
2024-03-12 01:02:14.771,"/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::raw_vec::RawVec<u8, alloc::alloc::Global>>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497"
2024-03-12 01:02:14.771,"/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::vec::Vec<u8, alloc::alloc::Global>>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497"
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::string::String>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(gen_single_block) yjit/src/codegen.rs:772
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(gen_block_series_body+0x4e) [0x7fe11045d9be] yjit/src/core.rs:1464
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(gen_block_series) yjit/src/core.rs:1442
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(branch_stub_hit_body+0x3b3) [0x7fe11045eee3] yjit/src/core.rs:1791
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(0x7fe110479c48) [0x7fe110479c48]
2024-03-12 01:02:14.771,"/usr/local/lib/libruby.so.3.2(with_vm_lock<yjit::core::branch_stub_hit::{closure_env#0}, *const u8>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:468"
2024-03-12 01:02:14.771,/usr/local/lib/libruby.so.3.2(branch_stub_hit+0x71) [0x7fe110462df1] yjit/src/core.rs:1693
2024-03-12 01:02:14.771,[0x7fe110c5201f]
2024-03-12 01:02:14.761,[Pitchfork] worker=1 pid=4908 gen=8 reaped (#<Process::Status: pid 4908 exit 0>)
2024-03-12 01:02:14.752,
2024-03-12 01:02:14.752,
2024-03-12 01:02:14.751,
2024-03-12 01:02:14.751,* Process memory map:
2024-03-12 01:02:14.751,
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::string::String>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(gen_single_block) yjit/src/codegen.rs:772
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(gen_block_series_body+0x4e) [0x7fe11045d9be] yjit/src/core.rs:1464
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(gen_block_series) yjit/src/core.rs:1442
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(branch_stub_hit_body+0x3b3) [0x7fe11045eee3] yjit/src/core.rs:1791
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(0x7fe110479c48) [0x7fe110479c48]
2024-03-12 01:02:14.741,"/usr/local/lib/libruby.so.3.2(with_vm_lock<yjit::core::branch_stub_hit::{closure_env#0}, *const u8>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:468"
2024-03-12 01:02:14.741,/usr/local/lib/libruby.so.3.2(branch_stub_hit+0x71) [0x7fe110462df1] yjit/src/core.rs:1693
2024-03-12 01:02:14.741,[0x7fe110c5201f]
2024-03-12 01:02:14.741,
2024-03-12 01:02:14.741,-- Other runtime information -----------------------------------------------
2024-03-12 01:02:14.741,
2024-03-12 01:02:14.741,* Loaded script: /vendor/bundle/ruby/3.2.0/bin/pitchfork
2024-03-12 01:02:14.741,
2024-03-12 01:02:14.741,* Loaded features:
2024-03-12 01:02:14.741,
2024-03-12 01:02:14.741,    0 enumerator.so
2024-03-12 01:02:14.741,    1 thread.rb
2024-03-12 01:02:14.741,    2 fiber.so
2024-03-12 01:02:14.741,    3 rational.so
2024-03-12 01:02:14.741,    4 complex.so
2024-03-12 01:02:14.741,    5 ruby2_keywords.rb
...
2024-03-12 01:02:14.740,"/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::vec::Vec<u8, alloc::alloc::Global>>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497"
2024-03-12 01:02:14.739,/usr/local/lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x7fe110449c3d] /usr/src/ruby/vm_dump.c:785
2024-03-12 01:02:14.739,/usr/local/lib/libruby.so.3.2(rb_vm_bugreport) /usr/src/ruby/vm_dump.c:1080
2024-03-12 01:02:14.739,/usr/local/lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf0) [0x7fe110244140] /usr/src/ruby/error.c:813
2024-03-12 01:02:14.739,/usr/local/lib/libruby.so.3.2(sigsegv+0x4b) [0x7fe11039bf7b] /usr/src/ruby/signal.c:964
2024-03-12 01:02:14.739,/lib/x86_64-linux-gnu/libc.so.6(0x7fe10fd8a050) [0x7fe10fd8a050]
2024-03-12 01:02:14.739,/usr/lib/libjemalloc.so.2(0x7fe1107796b3) [0x7fe1107796b3]
2024-03-12 01:02:14.739,/usr/lib/libjemalloc.so.2(0x7fe11076a737) [0x7fe11076a737]
2024-03-12 01:02:14.739,/usr/local/lib/libruby.so.3.2(0x7fe110540239) [0x7fe110540239]
2024-03-12 01:02:14.739,"/usr/local/lib/libruby.so.3.2(drop_in_place<alloc::raw_vec::RawVec<u8, alloc::alloc::Global>>) /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ptr/mod.rs:497"
2024-03-12 01:02:14.629,c:0001 p:0000 s:0003 E:000a10 DUMMY  [FINISH]
2024-03-12 01:02:14.629,
2024-03-12 01:02:14.629,-- Ruby level backtrace information ----------------------------------------
2024-03-12 01:02:14.629,/usr/local/bundle/bin/bundle:25:in `<main>'
2024-03-12 01:02:14.629,/usr/local/bundle/bin/bundle:25:in `load'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
...
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `load'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/bin/pitchfork:25:in `<top (required)>'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/bin/pitchfork:25:in `load'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/exe/pitchfork:109:in `<top (required)>'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:196:in `start'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:593:in `spawn_initial_mold'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186:in `clean_fork'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186:in `catch'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:188:in `block in clean_fork'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1045:in `block in fork_sibling'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:583:in `block in spawn_worker'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:832:in `worker_loop'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:786:in `init_worker_process'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:786:in `each'
2024-03-12 01:02:14.629,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:786:in `block in init_worker_process'
2024-03-12 01:02:14.629,
2024-03-12 01:02:14.629,-- Machine register context ------------------------------------------------
2024-03-12 01:02:14.629, RIP: 0x00007fe1107796b3 RBP: 0x00007fe10fb0f7f8 RSP: 0x00007ffc875b7650
2024-03-12 01:02:14.629, RAX: 0x00007fe0f7ba1bf8 RBX: 0x0000000000000000 RCX: 0x0000000000000024
2024-03-12 01:02:14.629, RDX: 0x00007fe10fb0fb50 RDI: 0x00007fe10fb0f7f8 RSI: 0x00007fe110818aa0
2024-03-12 01:02:14.629,  R8: 0x0000000000000000  R9: 0x00007fe110a2eac0 R10: 0x00007fe0c0000000
2024-03-12 01:02:14.629, R11: 0x00007fe10fb0fb50 R12: 0x00007fe0f4367068 R13: 0x00007fe10fb0fb50
2024-03-12 01:02:14.629, R14: 0x0000000000000000 R15: 0x0000000000000008 EFL: 0x0000000000010246
2024-03-12 01:02:14.629,
2024-03-12 01:02:14.629,-- C level backtrace information -------------------------------------------
2024-03-12 01:02:14.629,[Pitchfork] worker=1 pid=4908 gen=8 exiting
2024-03-12 01:02:14.628,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:786: [BUG] Segmentation fault at 0x0000000000000000[Pitchfork] worker=0 pid=5510 gen=9 registered
2024-03-12 01:02:14.628,
2024-03-12 01:02:14.628,ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [x86_64-linux]
2024-03-12 01:02:14.628,
2024-03-12 01:02:14.628,-- Control frame information -----------------------------------------------
2024-03-12 01:02:14.628,c:0030 p:0000 s:0172 e:000171 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:786 [FINISH]
2024-03-12 01:02:14.628,c:0029 p:---- s:0168 e:000167 CFUNC  :each
2024-03-12 01:02:14.628,c:0028 p:0055 s:0164 E:000fd8 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:786
2024-03-12 01:02:14.628,c:0027 p:0005 s:0157 e:000156 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:832
2024-03-12 01:02:14.628,c:0026 p:0018 s:0145 e:000144 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:583
2024-03-12 01:02:14.628,c:0025 p:0007 s:0142 e:000141 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1045
2024-03-12 01:02:14.628,c:0024 p:0016 s:0139 e:000138 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:188 [FINISH]
2024-03-12 01:02:14.628,c:0023 p:---- [Pitchfork] Sent SIGTERM to worker=1 pid=4908 gen=8
2024-03-12 01:02:14.628,s:0136 e:000135 CFUNC  :catch
2024-03-12 01:02:14.628,c:0022 p:0057 s:0131 e:000130 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186
2024-03-12 01:02:14.628,c:0021 p:0019 s:0123 E:002658 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:593
2024-03-12 01:02:14.628,c:0020 p:0053 s:0117 E:0020e8 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:196
2024-03-12 01:02:14.628,c:0019 p:0117 s:0112 E:000e20 TOP    /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/exe/pitchfork:109 [FINISH]
2024-03-12 01:02:14.628,c:0018 p:---- s:0104 e:000103 CFUNC  :load
2024-03-12 01:02:14.628,c:0017 p:0078 s:0099 e:000098 TOP    /vendor/bundle/ruby/3.2.0/bin/pitchfork:25 [FINISH]
2024-03-12 01:02:14.628,c:0016 p:---- s:0094 e:000093 CFUNC  :load
2024-03-12 01:02:14.628,c:0015 p:0064 s:0089 e:000088 METHOD /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58
...
2024-03-12 01:02:14.628,c:0005 p:0007 s:0022 e:000021 METHOD /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117
2024-03-12 01:02:14.628,c:0004 p:0108 s:0017 E:000258 TOP    /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/exe/bundle:29 [FINISH]
2024-03-12 01:02:14.628,c:0003 p:---- s:0013 e:000012 CFUNC  :load
2024-03-12 01:02:14.628,c:0002 p:0081 s:0008 E:000090 EVAL   /usr/local/bundle/bin/bundle:25 [FINISH]
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74: [BUG] Segmentation fault at 0x0000000000000000
2024-03-12 01:02:14.622,ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [x86_64-linux]
2024-03-12 01:02:14.622,
2024-03-12 01:02:14.622,-- Control frame information -----------------------------------------------
2024-03-12 01:02:14.622,c:0037 p:0018 s:0216 e:000215 METHOD /vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74
2024-03-12 01:02:14.622,c:0036 p:0004 s:0211 e:000210 METHOD /vendor/bundle/ruby/3.2.0/gems/dalli-3.2.6/lib/dalli/pid_cache.rb:26
2024-03-12 01:02:14.622,c:0035 p:0004 s:0206 e:000205 METHOD /vendor/bundle/ruby/3.2.0/gems/redis-client-0.19.0/lib/redis_client/pid_cache.rb:20
2024-03-12 01:02:14.622,c:0034 p:0004 s:0201 e:000200 METHOD /vendor/bundle/ruby/3.2.0/gems/activesupport-7.1.3.2/lib/active_support/fork_tracker.rb:7 [FINISH]
2024-03-12 01:02:14.622,c:0033 p:---- s:0196 e:000195 CFUNC  :fork
2024-03-12 01:02:14.622,c:0032 p:0004 s:0192 e:000191 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:159 [FINISH]
2024-03-12 01:02:14.622,c:0031 p:---- s:0189 e:000188 CFUNC  :synchronize
2024-03-12 01:02:14.622,c:0030 p:0005 s:0185 e:000184 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:159
2024-03-12 01:02:14.622,c:0029 p:0108 s:0177 e:000176 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1042
2024-03-12 01:02:14.622,c:0028 p:0054 s:0166 e:000165 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:579
2024-03-12 01:02:14.622,c:0027 p:0128 s:0159 e:000158 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:934
2024-03-12 01:02:14.622,c:0026 p:0039 s:0146 e:000145 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:900
2024-03-12 01:02:14.622,c:0025 p:0007 s:0142 e:000141 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1045
2024-03-12 01:02:14.622,c:0024 p:0016 s:0139 e:000138 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:188 [FINISH]
2024-03-12 01:02:14.622,c:0023 p:---- s:0136 e:000135 CFUNC  :catch
2024-03-12 01:02:14.622,c:0022 p:0057 s:0131 e:000130 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186
2024-03-12 01:02:14.622,c:0021 p:0019 s:0123 E:002658 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:593
2024-03-12 01:02:14.622,c:0020 p:0053 s:0117 E:0020e8 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:196
2024-03-12 01:02:14.622,c:0019 p:0117 s:0112 E:000e20 TOP    /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/exe/pitchfork:109 [FINISH]
2024-03-12 01:02:14.622,c:0018 p:---- s:0104 e:000103 CFUNC  :load
2024-03-12 01:02:14.622,c:0017 p:0078 s:0099 e:000098 TOP    /vendor/bundle/ruby/3.2.0/bin/pitchfork:25 [FINISH]
2024-03-12 01:02:14.622,c:0016 p:---- s:0094 e:000093 CFUNC  :load
2024-03-12 01:02:14.622,c:0015 p:0064 s:0089 e:000088 METHOD /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58
...
2024-03-12 01:02:14.622,c:0005 p:0007 s:0022 e:000021 METHOD /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117
2024-03-12 01:02:14.622,c:0004 p:0108 s:0017 E:000258 TOP    /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/exe/bundle:29 [FINISH]
2024-03-12 01:02:14.622,c:0003 p:---- s:0013 e:000012 CFUNC  :load
2024-03-12 01:02:14.622,c:0002 p:0081 s:0008 E:000090 EVAL   /usr/local/bundle/bin/bundle:25 [FINISH]
2024-03-12 01:02:14.622,c:0001 p:0000 s:0003 E:000a10 DUMMY  [FINISH]
2024-03-12 01:02:14.622,
2024-03-12 01:02:14.622,-- Ruby level backtrace information ----------------------------------------
2024-03-12 01:02:14.622,/usr/local/bundle/bin/bundle:25:in `<main>'
2024-03-12 01:02:14.622,/usr/local/bundle/bin/bundle:25:in `load'
...
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `load'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/bin/pitchfork:25:in `<top (required)>'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/bin/pitchfork:25:in `load'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/exe/pitchfork:109:in `<top (required)>'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:196:in `start'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:593:in `spawn_initial_mold'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186:in `clean_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186:in `catch'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:188:in `block in clean_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1045:in `block in fork_sibling'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:900:in `block in spawn_mold'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:934:in `mold_loop'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:579:in `spawn_worker'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1042:in `fork_sibling'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:159:in `clean_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:159:in `synchronize'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:159:in `block in clean_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:159:in `fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/activesupport-7.1.3.2/lib/active_support/fork_tracker.rb:7:in `_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/redis-client-0.19.0/lib/redis_client/pid_cache.rb:20:in `_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/dalli-3.2.6/lib/dalli/pid_cache.rb:26:in `_fork'
2024-03-12 01:02:14.622,/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74:in `_fork'
2024-03-12 01:02:14.622,
2024-03-12 01:02:14.622,-- Machine register context ------------------------------------------------
2024-03-12 01:02:14.622, RIP: 0x00007fe1107796b3 RBP: 0x00007fe10fb0f7f8 RSP: 0x00007ffc875b6e10
2024-03-12 01:02:14.622, RAX: 0x00007fe0f7ba1bf8 RBX: 0x0000000000000000 RCX: 0x0000000000000024
2024-03-12 01:02:14.622, RDX: 0x00007fe10fb0fb50 RDI: 0x00007fe10fb0f7f8 RSI: 0x00007fe110818aa0
2024-03-12 01:02:14.622,  R8: 0x0000000000000000  R9: 0x00007fe110a2eac0 R10: 0x00007fe0c0000000
2024-03-12 01:02:14.622, R11: 0x00007fe10fb0fb50 R12: 0x00007fe0f4367068 R13: 0x00007fe10fb0fb50
2024-03-12 01:02:14.622, R14: 0x0000000000000000 R15: 0x0000000000000008 EFL: 0x0000000000010246
2024-03-12 01:02:14.622,
2024-03-12 01:02:14.622,-- C level backtrace information -------------------------------------------
2024-03-12 01:02:14.590,c:0035 p:0004 s:0203 e:000202 METHOD /vendor/bundle/ruby/3.2.0/gems/dalli-3.2.6/lib/dalli/pid_cache.rb:26
2024-03-12 01:02:14.590,c:0034 p:0004 s:0198 e:000197 METHOD /vendor/bundle/ruby/3.2.0/gems/redis-client-0.19.0/lib/redis_client/pid_cache.rb:20
2024-03-12 01:02:14.590,c:0033 p:0004 s:0193 e:000192 METHOD /vendor/bundle/ruby/3.2.0/gems/activesupport-7.1.3.2/lib/active_support/fork_tracker.rb:7 [FINISH]
2024-03-12 01:02:14.590,c:0032 p:---- s:0188 e:000187 CFUNC  :fork
2024-03-12 01:02:14.590,c:0031 p:0004 s:0184 e:000183 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1022 [FINISH]
2024-03-12 01:02:14.590,c:0030 p:---- s:0181 e:000180 CFUNC  :synchronize
2024-03-12 01:02:14.590,c:0029 p:0024 s:0177 e:000176 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1022
2024-03-12 01:02:14.590,c:0028 p:0054 s:0166 e:000165 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:579
2024-03-12 01:02:14.590,c:0027 p:0128 s:0159 e:000158 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:934
2024-03-12 01:02:14.590,c:0026 p:0039 s:0146 e:000145 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:900
2024-03-12 01:02:14.590,c:0025 p:0007 s:0142 e:000141 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1045
2024-03-12 01:02:14.590,c:0024 p:0016 s:0139 e:000138 BLOCK  /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:188 [FINISH]
2024-03-12 01:02:14.590,c:0023 p:---- s:0136 e:000135 CFUNC  :catch
2024-03-12 01:02:14.590,c:0022 p:0057 s:0131 e:000130 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186
2024-03-12 01:02:14.590,c:0021 p:0019 s:0123 E:002658 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:593
2024-03-12 01:02:14.590,c:0020 p:0053 s:0117 E:0020e8 METHOD /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:196
2024-03-12 01:02:14.590,c:0019 p:0117 s:0112 E:000e20 TOP    /vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/exe/pitchfork:109 [FINISH]
2024-03-12 01:02:14.590,c:0018 p:---- s:0104 e:000103 CFUNC  :load
2024-03-12 01:02:14.590,c:0017 p:0078 s:0099 e:000098 TOP    /vendor/bundle/ruby/3.2.0/bin/pitchfork:25 [FINISH]
2024-03-12 01:02:14.590,c:0016 p:---- s:0094 e:000093 CFUNC  :load
2024-03-12 01:02:14.590,c:0015 p:0064 s:0089 e:000088 METHOD /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58
...
2024-03-12 01:02:14.590,c:0005 p:0007 s:0022 e:000021 METHOD /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117
2024-03-12 01:02:14.590,c:0004 p:0108 s:0017 E:000258 TOP    /vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/exe/bundle:29 [FINISH]
2024-03-12 01:02:14.590,c:0003 p:---- s:0013 e:000012 CFUNC  :load
2024-03-12 01:02:14.590,c:0002 p:0081 s:0008 E:000090 EVAL   /usr/local/bundle/bin/bundle:25 [FINISH]
2024-03-12 01:02:14.590,c:0001 p:0000 s:0003 E:000a10 DUMMY  [FINISH]
2024-03-12 01:02:14.590,
2024-03-12 01:02:14.590,-- Ruby level backtrace information ----------------------------------------
2024-03-12 01:02:14.590,/usr/local/bundle/bin/bundle:25:in `<main>'
2024-03-12 01:02:14.590,/usr/local/bundle/bin/bundle:25:in `load'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
...
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `load'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/bin/pitchfork:25:in `<top (required)>'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/bin/pitchfork:25:in `load'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/exe/pitchfork:109:in `<top (required)>'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:196:in `start'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:593:in `spawn_initial_mold'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186:in `clean_fork'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:186:in `catch'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork.rb:188:in `block in clean_fork'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1045:in `block in fork_sibling'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:900:in `block in spawn_mold'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:934:in `mold_loop'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:579:in `spawn_worker'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1022:in `fork_sibling'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1022:in `synchronize'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1022:in `block in fork_sibling'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/pitchfork-0.11.0/lib/pitchfork/http_server.rb:1022:in `fork'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/activesupport-7.1.3.2/lib/active_support/fork_tracker.rb:7:in `_fork'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/redis-client-0.19.0/lib/redis_client/pid_cache.rb:20:in `_fork'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/dalli-3.2.6/lib/dalli/pid_cache.rb:26:in `_fork'
2024-03-12 01:02:14.590,/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74:in `_fork'
2024-03-12 01:02:14.590,
2024-03-12 01:02:14.590,-- Machine register context ------------------------------------------------
2024-03-12 01:02:14.590, RIP: 0x00007fe1107796b3 RBP: 0x00007fe10fb0f7f8 RSP: 0x00007ffc875b6e10
2024-03-12 01:02:14.590, RAX: 0x00007fe0f7ba1bf8 RBX: 0x0000000000000000 RCX: 0x0000000000000024
2024-03-12 01:02:14.590, RDX: 0x00007fe10fb0fb50 RDI: 0x00007fe10fb0f7f8 RSI: 0x00007fe110818aa0
2024-03-12 01:02:14.590,  R8: 0x0000000000000000  R9: 0x00007fe110a2eac0 R10: 0x00007fe0c0000000
2024-03-12 01:02:14.590, R11: 0x00007fe10fb0fb50 R12: 0x00007fe0f4367068 R13: 0x00007fe10fb0fb50
2024-03-12 01:02:14.590, R14: 0x0000000000000000 R15: 0x0000000000000008 EFL: 0x0000000000010246
2024-03-12 01:02:14.590,
2024-03-12 01:02:14.590,-- C level backtrace information -------------------------------------------
2024-03-12 01:02:14.589,/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74: [BUG] Segmentation fault at 0x0000000000000000
2024-03-12 01:02:14.589,ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [x86_64-linux]
2024-03-12 01:02:14.589,
2024-03-12 01:02:14.589,-- Control frame information -----------------------------------------------
2024-03-12 01:02:14.589,c:0036 p:0018 s:0208 e:000207 METHOD /vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:74
2024-03-12 01:02:14.575,[Pitchfork] worker=0 pid=4867 gen=8 reaped (#<Process::Status: pid 4867 exit 0>)
2024-03-12 01:02:14.575,[Pitchfork] worker=0 gen=9 spawning...
2024-03-12 01:02:14.575,"[Pitchfork] before_fork hook called"
2024-03-12 01:02:14.551,[Pitchfork] mold pid=4830 gen=8 reaped (#<Process::Status: pid 4830 exit 0>)
2024-03-12 01:02:14.447,[Pitchfork] mold pid=5474 gen=9 ready
2024-03-12 01:02:14.447,[Pitchfork] Terminating old mold pid=4830 gen=8
2024-03-12 01:02:14.447,[Pitchfork] Sent SIGTERM to worker=0 pid=4867 gen=8
2024-03-12 01:02:14.447,[Pitchfork] worker=0 pid=4867 gen=8 exiting
2024-03-12 01:02:12.844,"[Pitchfork] worker=0 gen=8 Refork condition met, promoting ourselves"
2024-03-12 01:02:12.683,[Pitchfork] mold pid=5474 gen=9 spawned
2024-03-12 01:02:12.682,[Pitchfork] after_mold_fork hook called
2024-03-12 01:02:12.623,"[Pitchfork] worker=0 pid=4867 processed 1000 requests, triggering a refork"
2024-03-12 01:02:12.623,"[Pitchfork] before_fork hook called"
@kyontan kyontan changed the title [WIP] worker crashes (and dump backtrace) after fork worker crashes (and dump backtrace) after fork Mar 12, 2024
@kyontan
Copy link
Author

kyontan commented Mar 12, 2024

let me try to update gems first

@casperisfine
Copy link
Contributor

Alright, so I can't be certain, but there are some evidence that this may be one of the two weakmap bugs:

Likely the first one given you are not compacting?

What suggest this is:

I'd suggest upgrading to Ruby 3.2.3 which should contain these fixes, or to 3.3.0.

@casperisfine
Copy link
Contributor

Also note to self: I should detect these broken versions, issue a warning and disable Pitchfork::Info.keep_io.

@kyontan
Copy link
Author

kyontan commented Mar 12, 2024

wow, thanks for the quick and helpful suggestion! Let me also try to upgrade Ruby and report then!

@casperisfine casperisfine mentioned this issue Mar 12, 2024
Merged
@casperisfine
Copy link
Contributor

Also note to self: I should detect these broken versions, issue a warning and disable Pitchfork::Info.keep_io.

Done in #101

@kyontan
Copy link
Author

kyontan commented Mar 13, 2024

UPDATE: We have upgraded our deployments to Ruby 3.2.3 and we see no crash so far 馃憤 (I'll keep eyes on it for another 1-2 days)

@casperisfine
Copy link
Contributor

That's great to hear! Closing for now then, but let me know if you run into another issue.

Is #99 solved too? Perhaps it was the same root cause but with different symptoms?

@kyontan
Copy link
Author

kyontan commented Mar 13, 2024 •
edited

@casperisfine
Yes, I guess #99 also had the same root cause, we too didn't observe the error logs after upgrading Ruby.
So let me close it as of now.

Thanks for your great support every time.

@kyontan kyontan mentioned this issue Mar 13, 2024
Closed
@casperisfine
Copy link
Contributor

馃檱

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kyontan @casperisfine