--gdbjit segfaults in semver

[BenLubar]

• Version: node 4.4.7, configured with --gdb
• Platform: Linux australium 4.4.0-28-generic V8 upgrades and what they mean for versioning #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem: crash only happens when --gdbjit is active

NodeJS segfaults when --gdbjit is active. The crash looks like memory corruption:
https://gist.github.com/BenLubar/93c545491a4db3d1d6bdd75090474da3#file-gistfile1-txt

I've narrowed the code that causes the crash down to this reproduceable test case:
https://gist.github.com/BenLubar/064b65170ae1f01ea00328b537eaefba

[MylesBorins]

@BenLubar would you be able to test this on some earlier versions of 4.x and see if this is a regression or a problem that has always existed?

Maybe try v4.2.0 to start as it was the first LTS.

[bnoordhuis]

@BenLubar Can you post the output of disassemble $pc,$pc+32 and info registers in gdb?

[BenLubar]

@bnoordhuis

(gdb) disassemble $pc,$pc+32
Dump of assembler code from 0xa03420 to 0xa03440:
=> 0x0000000000a03420 <_ZN2v88internal15GDBJITInterface12EventHandlerEPKNS_12JitCodeEventE+496>:        mov    (%rax),%rax
   0x0000000000a03423 <_ZN2v88internal15GDBJITInterface12EventHandlerEPKNS_12JitCodeEventE+499>:        mov    %rax,-0x248(%rbp)
   0x0000000000a0342a <_ZN2v88internal15GDBJITInterface12EventHandlerEPKNS_12JitCodeEventE+506>:        lea    -0x230(%rbp),%rdi
   0x0000000000a03431 <_ZN2v88internal15GDBJITInterface12EventHandlerEPKNS_12JitCodeEventE+513>:        callq  0xca8150 <_ZN2v88internal19SimpleStringBuilder8FinalizeEv>
   0x0000000000a03436 <_ZN2v88internal15GDBJITInterface12EventHandlerEPKNS_12JitCodeEventE+518>:        mov    0xaae803(%rip),%rbx        # 0x14b1c40 <_ZZN2v88internal15GDBJITInterfaceL10GetCodeMapEvE8code_map>
   0x0000000000a0343d <_ZN2v88internal15GDBJITInterface12EventHandlerEPKNS_12JitCodeEventE+525>:        mov    %rax,-0x258(%rbp)
End of assembler dump.
(gdb) info registers
rax            0x7c303a3f282e5c3a       8948716502516653114
rbx            0x7fffffff9bc0   140737488329664
rcx            0x7fffffff9c00   140737488329728
rdx            0x7fffffff9c00   140737488329728
rsi            0x14c6b28        21785384
rdi            0x7fffffff9a6c   140737488329324
rbp            0x7fffffff9ba0   0x7fffffff9ba0
rsp            0x7fffffff9940   0x7fffffff9940
r8             0xffff8000014cd0c8       -140737466543928
r9             0xffff8000014cd0b8       -140737466543944
r10            0xffff8000014cd0a8       -140737466543960
r11            0xfb3f308f71e    17265551013662
r12            0x2d83a1f75480   50043381306496
r13            0x2d83a1f75421   50043381306401
r14            0x14c6b14        21785364
r15            0x2d83a1f75421   50043381306401
rip            0xa03420 0xa03420 <v8::internal::GDBJITInterface::EventHandler(v8::JitCodeEvent const*)+496>
eflags         0x10206  [ PF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

@thealphanerd I tried it on 6.3.0 earlier and the same crash happened. I'll compile 4.2.0 now and try that.

[BenLubar]

@thealphanerd 4.2.0 does not crash.

[BenLubar]

4.3.0 also does not crash.

[BenLubar]

The crash appeared somewhere between 4.4.2 and 4.4.3

[MylesBorins]

I'm guessing it is aadf356aa2

/cc @bnoordhuis

[MylesBorins]

@BenLubar can you do a run with that commit reverted?

[MylesBorins]

/cc @nodejs/v8 @ofrobots

(sorry for the spam)

[ofrobots]

@BenLubar what are you using --gdbjit for and was that working for you in versions 4.4.2 and earlier?

[BenLubar]

@ofrobots we are trying to debug something that's causing our forum software to hang, so I made this script: https://github.com/boomzillawtf/tdwtf/blob/04633e03478ceffe5779706c40e408312b48c729/watchdog.bash

However, it only has symbols for the threads that are waiting on locks and semaphores, not the thread that's running the forum code.

We haven't had this running successfully yet on any version of NodeJS.

[BenLubar]

@thealphanerd it compiles and runs, but the symbols are not generated.

[ofrobots]

Okay, if you're interested in JS stack traces you might also want to look into the gdbinit file that ships with the node distribution (check share/doc/node/gdbinit). Check the jst macro defined in there. It can be finicky getting this macro working too as it depends on your precise gdb version and whether it can deal with the dwarf symbols and overloads.

The crash you reported needs investigation, but hopefully the gdbinit trick works for you for now. Very few people are using --gdbinit with node or V8, and it is not clear to me if it has gotten any testing in the past few years. It was been accidentally disabled for > 1yr without anyone noticing.

[BenLubar]

https://gist.github.com/BenLubar/e39e13628496661eb5818a0a0bf2f43e

(gdb) run

Starting program: /usr/src/v8/out/x64.debug/d8 --gdbjit-full --gdbjit-dump foo.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff623c700 (LWP 7755)]


#
# Fatal error in ../src/utils.cc, line 30
# Check failed: !is_finalized() && position_ + n <= buffer_.length().
#

==== C stack trace ===============================

 1: V8_Fatal
 2: v8::internal::SimpleStringBuilder::AddSubstring(char const*, int)
 3: v8::internal::GDBJITInterface::EventHandler(v8::JitCodeEvent const*)
 4: v8::internal::JitLogger::LogRecordedBuffer(v8::internal::AbstractCode*, v8::internal::SharedFunctionInfo*, char const*, int)
 5: v8::internal::CodeEventLogger::RegExpCodeCreateEvent(v8::internal::AbstractCode*, v8::internal::String*)
 6: v8::internal::CodeEventDispatcher::RegExpCodeCreateEvent(v8::internal::AbstractCode*, v8::internal::String*)
 7: v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>)
 8: v8::internal::RegExpCompiler::Assemble(v8::internal::RegExpMacroAssembler*, v8::internal::RegExpNode*, int, v8::internal::Handle<v8::internal::String>)
 9: v8::internal::RegExpEngine::Compile(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::RegExpCompileData*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, bool)
10: v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, bool)
11: v8::internal::RegExpImpl::EnsureCompiledIrregexp(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, bool)
12: v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>)
13: v8::internal::RegExpImpl::IrregexpExec(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::JSArray>)
14: v8::internal::RegExpImpl::Exec(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::JSArray>)
15: 0x1a9c54e
16: v8::internal::Runtime_RegExpExec(int, v8::internal::Object**, v8::internal::Isolate*)
17: 0x25d3ade063a7

Program received signal SIGILL, Illegal instruction.
v8::base::OS::Abort () at ../src/base/platform/platform-posix.cc:240
240         V8_IMMEDIATE_CRASH();
(gdb) bt
#0  v8::base::OS::Abort () at ../src/base/platform/platform-posix.cc:240
#1  0x0000000001bb56e8 in V8_Fatal (file=0x1e1b117 "../src/utils.cc", line=30, 
    format=0x1da99b8 "Check failed: %s.") at ../src/base/logging.cc:116
#2  0x000000000166d373 in v8::internal::SimpleStringBuilder::AddSubstring (
    this=0x7fffffffbff0, 
    s=0x22ef454 "RegExp:^\\s*([v=\\s]*(0|[1-9]\\d*|x|X|\\*)(?:\\.(0|[1-9]\\d*|x|X|\\*)(?:\\.(0|[1-9]\\d*|x|X|\\*)(?:(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][a-zA-Z0-9-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][a-zA-Z0-9-]*))*)))?(?:\\+([0-9A-Za-"..., 
    n=464) at ../src/utils.cc:30
#3  0x00000000012fb238 in v8::internal::GDBJITInterface::EventHandler (
    event=0x7fffffffc188) at ../src/gdb-jit.cc:2183
#4  0x000000000143f5fe in v8::internal::JitLogger::LogRecordedBuffer (
    this=0x22e3f00, code=0x25d3adf0aaa1, shared=0x0, 
    name=0x22ef454 "RegExp:^\\s*([v=\\s]*(0|[1-9]\\d*|x|X|\\*)(?:\\.(0|[1-9]\\d*|x|X|\\*)(?:\\.(0|[1-9]\\d*|x|X|\\*)(?:(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][a-zA-Z0-9-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][a-zA-Z0-9-]*))*)))?(?:\\+([0-9A-Za-"..., length=464) at ../src/log.cc:463
#5  0x000000000143ecb0 in v8::internal::CodeEventLogger::RegExpCodeCreateEvent
    (this=0x22e3f00, code=0x25d3adf0aaa1, source=0x330901825621)
    at ../src/log.cc:208
#6  0x000000000179e581 in v8::internal::CodeEventDispatcher::RegExpCodeCreateEvent (this=0x22dc360, code=0x25d3adf0aaa1, source=0x330901825621)
    at .././src/code-events.h:157
#7  0x000000000179dbfb in v8::internal::RegExpMacroAssemblerX64::GetCode (
---Type <return> to continue, or q <return> to quit---
    this=0x7fffffffd070, source=...)
    at ../src/regexp/x64/regexp-macro-assembler-x64.cc:1010
#8  0x00000000015bec1a in v8::internal::RegExpCompiler::Assemble (
    this=0x7fffffffd288, macro_assembler=0x7fffffffd070, start=0x2409800, 
    capture_count=12, pattern=...) at ../src/regexp/jsregexp.cc:1140
#9  0x00000000015bd189 in v8::internal::RegExpEngine::Compile (
    isolate=0x22dd3c0, zone=0x7fffffffd8f0, data=0x7fffffffd858, flags=..., 
    pattern=..., sample_subject=..., is_one_byte=true)
    at ../src/regexp/jsregexp.cc:6766
#10 0x00000000015bc727 in v8::internal::RegExpImpl::CompileIrregexp (re=..., 
    sample_subject=..., is_one_byte=true) at ../src/regexp/jsregexp.cc:388
#11 0x00000000015bd5ba in v8::internal::RegExpImpl::EnsureCompiledIrregexp (
    re=..., sample_subject=..., is_one_byte=true)
    at ../src/regexp/jsregexp.cc:340
#12 0x00000000015bd41c in v8::internal::RegExpImpl::IrregexpPrepare (
    regexp=..., subject=...) at ../src/regexp/jsregexp.cc:468
#13 0x00000000015bb742 in v8::internal::RegExpImpl::IrregexpExec (regexp=..., 
    subject=..., previous_index=0, last_match_info=...)
    at ../src/regexp/jsregexp.cc:588
#14 0x00000000015bb495 in v8::internal::RegExpImpl::Exec (regexp=..., 
    subject=..., index=0, last_match_info=...) at ../src/regexp/jsregexp.cc:203
#15 0x0000000001a9c54e in v8::internal::__RT_impl_Runtime_RegExpExec (
    args=..., isolate=0x22dd3c0) at ../src/runtime/runtime-regexp.cc:780
#16 0x0000000001a9c016 in v8::internal::Runtime_RegExpExec (args_length=4, 
---Type <return> to continue, or q <return> to quit---
    args_object=0x7fffffffdcd8, isolate=0x22dd3c0)
    at ../src/runtime/runtime-regexp.cc:768
#17 0x000025d3ade063a7 in Stub:CEntryStub ()
#18 0x000025d3adf078a8 in LazyCompile:~DoRegExpExec native regexp.js:86 ()
#19 0x000025d3adf0a6ea in LazyCompile:~StringReplaceNonGlobalRegExpWithFunction native regexp.js:502 ()
#20 0x000025d3adf0a60a in LazyCompile:~RegExpReplace native regexp.js:528 ()
#21 0x000025d3adf0969a in LazyCompile:~[Symbol.replace] native regexp.js:647 ()
#22 0x000025d3adf09013 in LazyCompile:~replace native string.js:102 ()
#23 0x000025d3adf087f2 in LazyCompile:~Range.parseRange foo.js:745 ()
#24 0x000025d3adf082dc in LazyCompile:~ foo.js:720 ()
#25 0x000025d3ade07e55 in Builtin:ArgumentsAdaptorTrampoline ()
#26 0x000025d3adf07fa1 in LazyCompile:~map native array.js:861 ()
#27 0x000025d3ade795b2 in LazyCompile:~Range foo.js:709 ()
#28 0x000025d3ade07afb in Builtin:JSConstructStubGeneric ()
#29 0x000025d3ade78fb2 in LazyCompile:~validRange foo.js:1107 ()
#30 0x000025d3ade07e55 in Builtin:ArgumentsAdaptorTrampoline ()
#31 0x000025d3ade738fc in Script:~foo.js ()
#32 0x000025d3ade46983 in Builtin:JSEntryTrampoline ()
#33 0x000025d3ade2620f in Stub:JSEntryStub ()
#34 0x00000000012a61da in v8::internal::(anonymous namespace)::Invoke (
    isolate=0x22dd3c0, is_construct=false, target=..., receiver=..., argc=0, 
    args=0x0, new_target=...) at ../src/execution.cc:111
#35 0x00000000012a5eb7 in v8::internal::Execution::Call (isolate=0x22dd3c0, 
---Type <return> to continue, or q <return> to quit---
    callable=..., receiver=..., argc=0, argv=0x0) at ../src/execution.cc:168
#36 0x0000000000e834ac in v8::Script::Run (this=0x232a068, context=...)
    at ../src/api.cc:1909
#37 0x0000000000e3c580 in v8::Shell::ExecuteString (isolate=0x22dd3c0, 
    source=..., name=..., print_result=false, report_exceptions=true, 
    source_type=v8::Shell::SCRIPT) at ../src/d8.cc:392
#38 0x0000000000e4624b in v8::SourceGroup::Execute (this=0x22db018, 
    isolate=0x22dd3c0) at ../src/d8.cc:1549
#39 0x0000000000e49079 in v8::Shell::RunMain (isolate=0x22dd3c0, argc=4, 
    argv=0x7fffffffecf8, last_run=true) at ../src/d8.cc:2051
#40 0x0000000000e4a244 in v8::Shell::Main (argc=4, argv=0x7fffffffecf8)
    at ../src/d8.cc:2515
#41 0x0000000000e51052 in main (argc=4, argv=0x7fffffffecf8)
    at ../src/d8.cc:2560
(gdb) up
#1  0x0000000001bb56e8 in V8_Fatal (file=0x1e1b117 "../src/utils.cc", line=30, 
    format=0x1da99b8 "Check failed: %s.") at ../src/base/logging.cc:116
116       v8::base::OS::Abort();
(gdb) 
#2  0x000000000166d373 in v8::internal::SimpleStringBuilder::AddSubstring (
    this=0x7fffffffbff0, 
    s=0x22ef454 "RegExp:^\\s*([v=\\s]*(0|[1-9]\\d*|x|X|\\*)(?:\\.(0|[1-9]\\d*|x|X|\\*)(?:\\.(0|[1-9]\\d*|x|X|\\*)(?:(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][a-zA-Z0-9-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][a-zA-Z0-9-]*))*)))?(?:\\+([0-9A-Za-"..., 
    n=464) at ../src/utils.cc:30
30        DCHECK(!is_finalized() && position_ + n <= buffer_.length());
(gdb) p is_finalized()
$1 = false
(gdb) p position_ + n
$2 = 464
(gdb) p buffer_.length()
$3 = 256
(gdb) p position_
$4 = 0

[Trott]

Does anyone know if this is still an issue or if it got fixed?

[bnoordhuis]

Unfixed, most likely. gdb-jit.cc has seen no real changes upstream since this issue was raised.

[MylesBorins]

Closing due to lack of activity. Let me know if we should re open