VM crash when profiler thread-unsafely accesses stub code entry

[kodandersson]

Starting program: /usr/local/google/home/koda/migrate2/sdk/out/DebugIA32/dart --ignore-unrecognized-flags --enable_asserts --enable_type_checks --package-root=out/DebugIA32/packages/ /usr/local/google/home/koda/migrate2/sdk/tests/standalone/io/http_advanced_test.dart
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/grte/v4/lib64/libthread_db.so.1".
[New Thread 0xf65ffb40 (LWP 22405)]
[New Thread 0xf6b3fb40 (LWP 22404)]
[New Thread 0xf7052b40 (LWP 22403)]
[New Thread 0xf7267b40 (LWP 22402)]
[New Thread 0xf7c85b40 (LWP 22401)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf7052b40 (LWP 22403)]
0x084cbf13 in dart::StubCode::InInvocationStubForIsolate (isolate=0xf6e061f0, pc=4130123616) at runtime/vm/stub_code.h:174
174 STUB_CODE_LIST(STUB_CODE_ACCESSOR);
(gdb) bt
#0 0x084cbf13 in dart::StubCode::InInvocationStubForIsolate (isolate=0xf6e061f0, pc=4130123616) at runtime/vm/stub_code.h:174
#1 0x084c441d in HasNext (this=0xf704fb94) at runtime/vm/stack_frame.h:209
#2 dart::StackFrameIterator::NextFrame (this=this@entry=0xf704fb68) at runtime/vm/stack_frame.cc:386
#3 0x084558ad in NextFrame (this=0xf704fb64) at runtime/vm/stack_frame.h:276
#4 dart::ProfilerDartExitStackWalker::walk (this=this@entry=0xf704fb60) at runtime/vm/profiler.cc:514
#5 0x0845573c in CollectSample (isolate=0xf6e061f0, pc=4158348346, dart_stack_walker=0xf704fb44, dart_exit_stack_walker=0xf704fb60,

native_stack_walker=0xf704fb2c, sample=0xf7273db0, in_dart_code=false, exited_dart_code=true) at runtime/vm/profiler.cc:882

#6 dart::Profiler::RecordSampleInterruptCallback (state=..., data=0xf6e061f0) at runtime/vm/profiler.cc:1078
#7 0x084e5fd3 in dart::ThreadInterrupterLinux::ThreadInterruptSignalHandler (signal=27, info=0xf704fc8c, context_=0xf704fd0c)

at runtime/vm/thread_interrupter_linux.cc:41

#8
#9 0xf7db5c3a in ?? () from /lib/i386-linux-gnu/libc.so.6
#10 0x084f3285 in memset (__len=2097160, __ch=171, __dest=0xf5f3f008) at /usr/include/bits/string3.h:84
#11 dart::Zone::Segment::New (size=size@entry=2097160, next=0x0) at runtime/vm/zone.cc:70
#12 0x084f34c3 in dart::Zone::AllocateLargeSegment (this=this@entry=0xf7050378, size=2097160, size@entry=2097152)

at runtime/vm/zone.cc:166

#13 0x084f367c in dart::Zone::AllocateExpand (this=this@entry=0xf7050378, size=size@entry=2097152) at runtime/vm/zone.cc:138
#14 0x081b72ec in dart::Zone::AllocUnsafe (this=0xf7050378, size=) at runtime/vm/zone.h:214
#15 0x084b9014 in Allocdart::BackRefNode (len=, this=0xf7050378) at runtime/vm/zone.h:236
#16 BaseGrowableArray (allocator=, initial_capacity=163840, this=0xf6e1bae0) at runtime/vm/growable_array.h:30
#17 ZoneGrowableArray (initial_capacity=163840, this=0xf6e1bae0) at runtime/vm/growable_array.h:180
#18 dart::IsolateSnapshotReader::IsolateSnapshotReader (this=0xf7050308,

buffer=0x8813df0 <dart::bin::isolate_snapshot_buffer_+16> "9056854342863f495b219e2308aac128ݵ", size=256616, isolate=0xf6e061f0, 
zone=0xf7050378) at runtime/vm/snapshot.cc:1213

#19 0x08228ca7 in dart::Dart::InitializeIsolate (

snapshot_buffer=snapshot_buffer@entry=0x8813de0 <dart::bin::isolate_snapshot_buffer_> "h\352\003", data=data@entry=0xf6e635a0)
at runtime/vm/dart.cc:281

#20 0x081b42ba in dart::Dart_CreateIsolate (

script_uri=0x8c31440 "/usr/local/google/home/koda/migrate2/sdk/tests/standalone/io/http_advanced_test.dart", 
main=0xf6e63580 "startIsolatedHttpServer", snapshot=0x8813de0 <dart::bin::isolate_snapshot_buffer_> "h\352\003", flags=0xf705115c, 
callback_data=0xf6e635a0, error=0xf7051158) at runtime/vm/dart_api_impl.cc:1363

#21 0x0818ae35 in dart::bin::CreateIsolateAndSetupHelper (

script_uri=script_uri@entry=0x8c31440 "/usr/local/google/home/koda/migrate2/sdk/tests/standalone/io/http_advanced_test.dart", 
main=main@entry=0xf6e63580 "startIsolatedHttpServer", package_root=package_root@entry=0x8c314a0 "out/DebugIA32/packages/", 
flags=flags@entry=0xf705115c, error=error@entry=0xf7051158, exit_code=exit_code@entry=0xf70510f4) at runtime/bin/main.cc:594

#22 0x0818b19b in dart::bin::CreateIsolateAndSetup (

script_uri=0x8c31440 "/usr/local/google/home/koda/migrate2/sdk/tests/standalone/io/http_advanced_test.dart", 
main=0xf6e63580 "startIsolatedHttpServer", package_root=0x8c314a0 "out/DebugIA32/packages/", flags=flags@entry=0xf705115c, 
data=0x8c31428, error=error@entry=0xf7051158) at runtime/bin/main.cc:708

#23 0x087b073a in CreateIsolate (error=0xf7051158, state=0xf6e83f88, parent_isolate=0x8c343d8) at runtime/lib/isolate.cc:184
#24 dart::Spawn (parent_isolate=parent_isolate@entry=0x8c343d8, state=state@entry=0xf6e83f88) at runtime/lib/isolate.cc:202
#25 0x087b5c5b in DN_HelperIsolate_spawnFunction (thread=, arguments=0xf7051854, zone=0xf70511f8, isolate=0x8c343d8)

at runtime/lib/isolate.cc:239

#26 dart::BootstrapNatives::DN_Isolate_spawnFunction (args=0xf7051854) at runtime/lib/isolate.cc:220
#27 0xf6f92160 in ?? ()
#28 0xf62caf60 in ?? ()
#29 0xf62ca9b8 in ?? ()
#30 0xf62ca309 in ?? ()
#31 0xf62c5555 in ?? ()
#32 0xf62c4fdf in ?? ()
#33 0xf62c4df2 in ?? ()
#34 0xf6fca3b3 in ?? ()
#35 0xf62c4c62 in ?? ()
#36 0xf6fa2e16 in ?? ()
#37 0xf6fc8471 in ?? ()
#38 0xf6f93c93 in ?? ()
#39 0x08233edb in dart::DartEntry::InvokeFunction (function=..., arguments=..., arguments_descriptor=...) at runtime/vm/dart_entry.cc:121
#40 0x08235dcf in dart::DartEntry::InvokeFunction (function=..., arguments=...) at runtime/vm/dart_entry.cc:27
#41 0x08239f36 in dart::DartLibraryCalls::HandleMessage (handler=..., message=...) at runtime/vm/dart_entry.cc:501
#42 0x0836a0d7 in dart::IsolateMessageHandler::HandleMessage (this=0x8c31578, message=0x8c31400) at runtime/vm/isolate.cc:475

---Type to continue, or q to quit---q

[kodandersson]

@johnmccutchan

[kodandersson]

I've only observed this in a modified client, but maybe @johnmccutchan can confirm this is likely an issue in the existing code?

[johnmccutchan]

This may be related to your change because we rely on top_exit_frame_info to decide which stack walker to use.

[kodandersson]

Yes, top_exit_frame_info was indeed being incorrectly updated.

Still, we might want to make sure that there is no race condition here by having explicit synchronization or reordering the initialization steps, but this is low prio.