HELM causing segmentation fault on Spacemacs inside ssh connection

[cvcore]

Description

This problem happens to me roughly once or twice a week. When this happens, emacs will crash immediately after I invoke some helm buffer. (in this bug report HELM Woman was being invoked when I pressed K multiple times. HELM Mini SPC b-b and many other HELM buffers has also caused the crash multiple times before)

Reproduction guide 🪲

• Start Emacs
• Using Spacemacs for some time
• Invoke some HELM buffer e.g. HELM Find Files, HELM Mini, HELM Woman
• Emacs will crash by chance

Observed behaviour: 👀 💔
As stated above
Plus every time backtrace will include something involving libpthread.so.0 (see below)

Expected behaviour: ❤️ 😄
HELM Buffer invocation.

(%LAST_KEYS%)
KKKKKK

%SYSTEM_INFO%
Ubuntu 16.04 LTS
Kernel 4.4.0-59-generic
Emacs 24.5.1
Spacemacs v.0.200.7

Backtrace 🐾

Fatal error 11: Segmentation faultKKK
Backtrace:
emacs[0x5036d3]
emacs[0x4e9d6e]
emacs[0x50249e]
emacs[0x5026c3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113e0)[0x7f8e748af3e0]
emacs[0x4aa060]
emacs[0x4ad4ca]
emacs[0x41cde9]
emacs[0x41fb9e]
emacs[0x421796]
emacs[0x4504f9]
emacs[0x4528e5]
emacs[0x4225c0]
emacs[0x4f5214]
emacs[0x4f63ed]
emacs[0x4f8150]
emacs[0x55bba7]
emacs[0x4ea13e]
emacs[0x55ba8b]
emacs[0x4ee7a6]
emacs[0x51756f]
emacs[0x517b7f]
emacs[0x55d8b7]
emacs[0x592b23]
emacs[0x55d74b]
emacs[0x592b23]
emacs[0x55d74b]
emacs[0x55ce5c]
emacs[0x56026e]
emacs[0x593f2f]
emacs[0x55d74b]
emacs[0x55ec6c]
emacs[0x55d82a]
emacs[0x592b23]
emacs[0x55d74b]
emacs[0x55ec6c]
emacs[0x55d82a]
emacs[0x592b23]
emacs[0x55d74b]
emacs[0x592b23]
emacs[0x55d74b]

[sdwolfz]

This might be related to my comment for #6093. Do you have a custom linum-format inside your .spacemacs? This seems to be the cause for my segfaults but I have no idea how to debug it.

[cvcore]

Yes. I have
(eval-after-load "linum" '(set-face-attribute 'linum nil :height 100))
Inside my dotspacemacs/user-config
But I'm not sure if this is the cause. I will remove this line and see if the crash still persists.

edit 21-01-17:
the above settings is a workaround for #7261. now it's no longer needed.

[GregorySchwartz]

Also see #7572 and #6093 (as previously mentioned).

[cvcore]

Update:
The problem seems to be resolved after I removed linum related settings. My editor did not crash in the past 3 days.

Thanks @SDWolf 😃

[cvcore]

Darn. Just trying to invoke HELM Mini Spc b b and crashed again 😭

Backtrace:
emacs[0x557cae]
emacs[0x5313f9]
emacs[0x5568e8]
/lib64/libpthread.so.0(+0xf370)[0x7f0f07104370]
emacs[0x4de2e8]
emacs[0x4e3279]
emacs[0x41f148]
emacs[0x422a08]
emacs[0x424f5c]
emacs[0x46fdec]
emacs[0x53f051]
emacs[0x541640]
emacs[0x543786]
emacs[0x5cd2f1]
emacs[0x5317fc]
emacs[0x5cd1dc]
emacs[0x531760]
emacs[0x537e07]
emacs[0x57353f]
emacs[0x573fbc]
emacs[0x5d071c]
emacs[0x6176d3]
emacs[0x5d056f]
emacs[0x6176d3]
emacs[0x5d056f]
emacs[0x5d14e0]
emacs[0x5d06a9]
emacs[0x6176d3]
emacs[0x5d056f]
emacs[0x5d14e0]
emacs[0x5d06a9]
emacs[0x6176d3]
emacs[0x5d056f]
emacs[0x6176d3]
emacs[0x5d056f]
emacs[0x5ca66a]
emacs[0x5d06a9]
emacs[0x5d162c]
emacs[0x5ca840]
emacs[0x5d0798]
emacs[0x6176d3]

[robdaemon]

I've been having this a lot with emacs 25.1.1 inside tmux over an SSH connection. Based on the comment about linum overrides, I'm going to give it a shot with removing (global-linum-mode t) from my .spacemacs.

[sdwolfz]

@robdaemon now that you mentioned it, I don't remember having segfaults until I started using it inside tmux, even with the changes to my linum-format. I'l try to trigger a crash and open it again outside tmux to see if I can reproduce it. But I still don't know much about how to debug it.

[thatnerdjosh]

I keep getting these segfaults too :(, I have linum-mode enabled with relative line numbers, I use tmux, but I am not using it when I get the segfaults... also it is in an SSH connection

[thatnerdjosh]

╭─nerdsville@dev.baseball.nerdsville.net ~/baseball  ‹develop*›
╰─➤  sed -n 's/.*\[\(.*\)]$/\1/p' backtrace |
  addr2line -C -f -i -p -e `which emacs`
emacs_backtrace at /root/emacs-25.1/src/sysdep.c:2234
terminate_due_to_signal at /root/emacs-25.1/src/emacs.c:375
handle_fatal_signal at /root/emacs-25.1/src/sysdep.c:1601
deliver_thread_signal.constprop.6 at /root/emacs-25.1/src/sysdep.c:1575
stack_overflow at /root/emacs-25.1/src/sysdep.c:1648
 (inlined by) handle_sigsegv at /root/emacs-25.1/src/sysdep.c:1691
?? ??:0
turn_on_face at /root/emacs-25.1/src/term.c:1895 (discriminator 3)
tty_write_glyphs at /root/emacs-25.1/src/term.c:769
update_frame_line at /root/emacs-25.1/src/dispnew.c:4838
update_frame_1 at /root/emacs-25.1/src/dispnew.c:4540
update_frame at /root/emacs-25.1/src/dispnew.c:3123
redisplay_internal at /root/emacs-25.1/src/xdisp.c:14044 (discriminator 2)
read_char at /root/emacs-25.1/src/keyboard.c:2479
read_key_sequence at /root/emacs-25.1/src/keyboard.c:9063
command_loop_1 at /root/emacs-25.1/src/keyboard.c:1365
internal_condition_case at /root/emacs-25.1/src/eval.c:1311
command_loop_2 at /root/emacs-25.1/src/keyboard.c:1108 (discriminator 1)
internal_catch at /root/emacs-25.1/src/eval.c:1076
command_loop at /root/emacs-25.1/src/keyboard.c:1079
recursive_edit_1 at /root/emacs-25.1/src/keyboard.c:693
XUNTAG at /root/emacs-25.1/src/lisp.h:866
 (inlined by) XWINDOW at /root/emacs-25.1/src/lisp.h:1043
 (inlined by) read_minibuf at /root/emacs-25.1/src/minibuf.c:664
Fread_from_minibuffer at /root/emacs-25.1/src/minibuf.c:946
Ffuncall at /root/emacs-25.1/src/eval.c:2723
exec_byte_code at /root/emacs-25.1/src/bytecode.c:880
backtrace_debug_on_exit at /root/emacs-25.1/src/eval.c:154
 (inlined by) Ffuncall at /root/emacs-25.1/src/eval.c:2766
exec_byte_code at /root/emacs-25.1/src/bytecode.c:880
backtrace_debug_on_exit at /root/emacs-25.1/src/eval.c:154
 (inlined by) Ffuncall at /root/emacs-25.1/src/eval.c:2766
Fapply at /root/emacs-25.1/src/eval.c:2323
Ffuncall at /root/emacs-25.1/src/eval.c:2673
exec_byte_code at /root/emacs-25.1/src/bytecode.c:880
backtrace_debug_on_exit at /root/emacs-25.1/src/eval.c:154
 (inlined by) Ffuncall at /root/emacs-25.1/src/eval.c:2766
Fapply at /root/emacs-25.1/src/eval.c:2323
Ffuncall at /root/emacs-25.1/src/eval.c:2673
exec_byte_code at /root/emacs-25.1/src/bytecode.c:880
backtrace_debug_on_exit at /root/emacs-25.1/src/eval.c:154
 (inlined by) Ffuncall at /root/emacs-25.1/src/eval.c:2766
exec_byte_code at /root/emacs-25.1/src/bytecode.c:880
apply_lambda at /root/emacs-25.1/src/eval.c:2794
eval_sub at /root/emacs-25.1/src/eval.c:2253
XCDR at /root/emacs-25.1/src/lisp.h:1230
 (inlined by) Fprogn at /root/emacs-25.1/src/eval.c:427
FletX at /root/emacs-25.1/src/eval.c:884
eval_sub at /root/emacs-25.1/src/eval.c:2119

[syl20bnr]

Marking it as duplicate of #7572

Anyone succeeded in reproducing it without line numbers activated ?

Someone could try to reproduce it with stock Emacs with line number activated ?

[syl20bnr]

I found this, not sure it is relevant: https://bugs.launchpad.net/ubuntu/+bug/1570950

[thatnerdjosh]

I will try to reproduce this without line numbers next, I tried using ivy instead of helm and same thing, I feel like we are getting closer to nailing this down

[thatnerdjosh]

@syl20bnr AFAICT, turning off line numbers resolves the issue, this may be where to look, will let you know if I see another crash, but so far so good

[cvcore]

@NerdsvilleCEO @syl20bnr For me I am still keeping linum but without any customization. The crash rate is significantly lower than before.

[scotttrinh]

I was having this problem pretty consistently on 25.1.2. Just built 25.1.91.1 from source and it seems to have helped. I'll report back after a few more days of working with it and see if it's fixed for me.

[thatnerdjosh]

@scotttrinh Interesting! I haven't had any crashes with linum turned off entirely, maybe it was a bug in emacs itself

[scotttrinh]

@NerdsvilleCEO I found this thread on the emacs mailing list and while I didn't go through the trouble of seeing if the backtrace of the discussed issue was related to my own problem, I just took the "Did you turn it off and back on again?" approach and re-built from source. We shall see!

[thatnerdjosh]

@scotttrinh How has everything been going with this? Any crashes since?

[scotttrinh]

So far, so good! I'm only about 1 full work day into using the new build, but I've not had a segfault yet, which is promising!

[scotttrinh]

Just had a segfault with M-x. 😿

[thatnerdjosh]

@scotttrinh Please provide the stack trace... it may be a different issue, let's make sure it is the same

[scotttrinh]

Yeah, it's so intermittent and I wasn't ready for it and didn't save the stack trace. I'll make sure my system is setup to give good stack traces so I'm ready for it next time it happens.

[scotttrinh]

Fatal error 11: Segmentation fault
Backtrace:
emacs[0x4a3ff2]
emacs[0x48c9a9]
emacs[0x4a2f3e]
emacs[0x4a3143]
emacs[0x4a317a]
/lib64/libpthread.so.0(+0xf100)[0x7fa6ba816100]
emacs[0x47bd0a]
emacs[0x47fc6c]
emacs[0x40a56b]
emacs[0x40c3ae]
emacs[0x40db20]
emacs[0x432fc4]
emacs[0x495cfb]
emacs[0x49825f]
emacs[0x499d66]
emacs[0x4fb09d]
emacs[0x48cebc]
emacs[0x4fb04b]
emacs[0x48ce77]
emacs[0x490f6f]
emacs[0x491295]
emacs[0x4066f4]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa6b9f4eb15]
emacs[0x407171]
Segmentation fault

Not sure how helpful that is. Any pointers on how to get more meaningful backtraces from emacs?

[GregorySchwartz]

@scotttrinh Using gdb would be your best bet.

[scotttrinh]

I tried to look up how to use gdb but quickly found myself lost. I'm SSHing into an EC2 instance, and couldn't quickly figure out how to get core dump turned on, so I followed the advice at https://www.gnu.org/software/emacs/manual/html_node/emacs/Crashing.html about using addr2line, and here is the output:

emacs_backtrace at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/sysdep.c:2242
terminate_due_to_signal at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/emacs.c:375
handle_fatal_signal at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/sysdep.c:1601
deliver_thread_signal.constprop.6 at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/sysdep.c:1575
stack_overflow at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/sysdep.c:1648
 (inlined by) handle_sigsegv at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/sysdep.c:1691
?? ??:0
turn_on_face at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/term.c:1895 (discriminator 3)
tty_write_glyphs at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/term.c:769
update_frame_line at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/dispnew.c:4838
update_frame_1 at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/dispnew.c:4540
update_frame at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/dispnew.c:3123
redisplay_internal at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/xdisp.c:14085 (discriminator 2)
read_char at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:2484
read_key_sequence at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:9068
command_loop_1 at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:1370
internal_condition_case at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/eval.c:1317
command_loop_2 at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:1113 (discriminator 1)
internal_catch at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/eval.c:1082
command_loop at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:1095
recursive_edit_1 at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:698
SPECPDL_INDEX at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/lisp.h:3143
 (inlined by) Frecursive_edit at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/keyboard.c:740
main at /persistent/home/shootproof_dev/src/emacs-emacs-25.1.91/src/emacs.c:1632
?? ??:0
_start at ??:?

Note, that this particular crash was launching a terminal, not M-x. Let me know if I should open a separate ticket.

[scotttrinh]

And then just got another crash (same backtrace) when opening NeoTree with an ansi-shell buffer open. Seems like shell buffer is the common denominator for my crashes.

[sdwolfz]

I've managed to find the problem in emacs 25.1.1 using gdb but i'm to tired now and I have to take a break. Will post the details over the weekend. You can find a temporary fix below, but I don't believe this is the real solution and it might cause more trouble in unexpected places in the future.

If you want to try it out just search where the FACE_FROM_ID macro is defined and replace it with this:

EDIT: use this commit instead: 7b60bce571dfc3fcdd461511751744eac74eb463

#define FACE_FROM_ID(F, ID)                                                  \
     (UNSIGNED_CMP (ID, <, FRAME_FACE_CACHE (F)->used)      \
      ? FRAME_FACE_CACHE (F)->faces_by_id[ID]                               \
      : FRAME_FACE_CACHE (F)->faces_by_id[0])

Then recompile emacs with:

./autogen.sh
./configure
make
sudo make install

And let me know if this helps you as well.

[thatnerdjosh]

@SDWolf which issue does this solve? I think we have two issues in this thread, comparing the stack traces

[sdwolfz]

@NerdsvilleCEO this should solve both, the stacktraces are identical to mine and I've produced the stacktrace while replicating the linum-format related segfault. I will detail it more tomorrow after I replicate it on my home machine as well.

In short, the segfault happens because the turn_on_face function has a call to FACE_FROM_ID which returns NULL instead of a valid struct because ID is equal to (instead of less than) (F)->used. But I have not yet found the reason for why this is.

[scotttrinh]

@SDWolf Just quickly tried out some risky commands that sometimes segfault, and seems to be working for me! I think I owe you a slice of 🍕. Maybe a whole pie.

[sdwolfz]

@scotttrinh I've pushed a commit that contains an even better fix on my fork: 7b60bce571dfc3fcdd461511751744eac74eb463

I still do not know the reason why this happens.
Here is a screenshot of my editor right when the segfault happens and gdb pauses execution:

[scotttrinh]

Interesting: emacs-mirror/emacs@374f6a5

Looks like they fixed (?) this issue in this commit by returning non-NULL. Not sure what happens in the case when ID is equal to (F)->used like you pointed out. Have we brought this issue up to the emacs team yet?

[sdwolfz]

Here is the promissed explanation (what I have so far):

Backtrace in gdb:

#0  0x00000000004ab2e0 in turn_on_face (f=0xc1c680, face_id=33) at term.c:1895
#1  0x00000000004af432 in tty_write_glyphs (f=<optimized out>, string=<optimized out>, len=<optimized out>) at term.c:767
#2  0x000000000041c2ea in update_frame_line (f=f@entry=0xc1c680, vpos=vpos@entry=29) at dispnew.c:4838
#3  0x000000000041f9e6 in update_frame_1 (f=f@entry=0xc1c680, force_p=force_p@entry=true, inhibit_id_p=<optimized out>,
    inhibit_id_p@entry=false, set_cursor_p=set_cursor_p@entry=true) at dispnew.c:4540
#4  0x00000000004210fe in update_frame (f=f@entry=0xc1c680, force_p=<optimized out>, force_p@entry=false, inhibit_hairy_id_p=inhibit_hairy_id_p@entry=false)
    at dispnew.c:3122
#5  0x0000000000452ebc in redisplay_internal () at xdisp.c:14085
#6  0x0000000000454ad5 in redisplay () at xdisp.c:13255
#7  0x00000000004f3d1b in read_char (commandflag=commandflag@entry=1, map=map@entry=28177075, prev_event=0, used_mouse_menu=used_mouse_menu@entry=0x7fffe9ed04db, end_time=end_time@entry=0x0) at keyboard.c:2482
#8  0x00000000004f66f0 in read_key_sequence (keybuf=keybuf@entry=0x7fffe9ed05b0, prompt=prompt@entry=0, dont_downcase_last=dont_downcase_last@entry=false, can_return_switch_frame=can_return_switch_frame@entry=true, fix_current_buffer=fix_current_buffer@entry=true, prevent_redisplay=prevent_redisplay@entry=false, bufsize=30)
    at keyboard.c:9068
#9  0x00000000004f8216 in command_loop_1 () at keyboard.c:1370
#10 0x0000000000559742 in internal_condition_case (bfun=bfun@entry=0x4f8020 <command_loop_1>, handlers=handlers@entry=19104, hfun=hfun@entry=0x4eece0 <cmd_error>)
    at eval.c:1315
#11 0x00000000004ea3fc in command_loop_2 (ignore=ignore@entry=0) at keyboard.c:1112
#12 0x00000000005596e3 in internal_catch (tag=tag@entry=45936, func=func@entry=0x4ea3e0 <command_loop_2>, arg=arg@entry=0) at eval.c:1080
#13 0x00000000004ea3b9 in command_loop () at keyboard.c:1091
#14 0x00000000004ee8e7 in recursive_edit_1 () at keyboard.c:697
#15 0x00000000004eec28 in Frecursive_edit () at keyboard.c:768
#16 0x00000000004182bb in main (argc=3, argv=0x7fffe9ed0908) at emacs.c:1629

Notice face_id=33 on frame 0 args. That is equal to f->face_cache->used which means FACE_FROM_ID returns NULL and causes the segfault in turn_on_face.
Also notice on frame 2 vpos=29. That is the index of the row currently being written. 29 corresponds to the row HELM's [emacs] Find file: text is displayed. Each one of the first 19 glyphs in the text (final space included) has face_id=33, the rest have face_id=5.

So it looks like HELM's text is at fault here. I will dig deeper tomorrow to see how this can be fixed without needing to patch emacs.

How to debug (in case anyone else wants to take a spin at it):

1. start spacemacs in a terminal: emacs -nw .
2. find emacs's PID: ps aux | grep emacs
3. start gdb by attaching it to the pid: sudo gdb attach 12345 (replace 12345 with the PID)
4. execute continue inside gdb.
5. go to spacemacs and open a relatively large file using HELM in the hope of triggering the segfault. Wait a few seconds to be sure spacemacs is unresponsive and the segfault was triggered.
6. go to gdb and start digging

@scotttrinh I tried building master and the segfault still hapens, that commit does nothing to help. I haven't notified upstream about this since I want to dig deeper and make a better case before submitting any fix. IMO this should be fixed in both places, emacs (null pointer derefference ) and HELM (font?).

[sdwolfz]

I've looked into this a bit more and I can not pin it down to anything specific, seems to be that helm's and linum's involvement in this is just a coincidence and the problem lies entirely in emacs.

I'l notify upstream with all the details that I have.

[sdwolfz]

Here's a link to the emacs bug report: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26097

[sdwolfz]

A fix for this landed on emacs master: emacs-mirror/emacs@8275687 thanks to Eli Zaretskii's effort.

Here is what you need to do in order to install emacs master from source:

git clone https://github.com/emacs-mirror/emacs.git
cd emacs
./autogen.sh
./autogen.sh git
./configure
make
sudo make install

Any error you might get in these commands are just a web search away, usually it's just a sudo apt-get install <missing-package> or a git clean -fdx and repeat.

This might be the solution for the following issues as well: #7776 #7572 #6093

[cvcore]

Closing the issue since it has been resolved in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26097

[sdwolfz]

Note that the fix is only present on master and most likely will only be available on emacs 26. I would not really consider this issue as "Closed" until then...