Flaky crash hitting cls.is_finalized() assertion

[mkustermann]

From this log:

/===============================================================================\
| standalone_2/out_of_memory_recovery_test broke (Pass -> Crash, expected Pass) |
\===============================================================================/

--- Command "vm" (took 34.000779s):
DART_CONFIGURATION=DebugX64 out/DebugX64/dart --old_gen_heap_size=20 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/tests/standalone_2/out_of_memory_recovery_test.dart

exit code:
-6

stdout:
>> [SendPort, 1]
<< [1, Okay]
>> [SendPort, 2]

stderr:
Exhausted heap space, trying to allocate 32 bytes.
Exhausted heap space, trying to allocate 128 bytes.
../../runtime/vm/resolver.cc: 72: error: expected: cls.is_finalized()
version=2.13.0-edge.5bfeed21c1af642d97d0ff5f8d9ba47d3af1181e (be) (Fri Feb 5 11:59:42 2021 +0000) on "linux_x64"
pid=23780, thread=23821, isolate_group=main(0x55cdb442f000), isolate=main(0x55cdb4518000)
isolate_instructions=55cdb20faa40, vm_instructions=55cdb20faa40
  pc 0x000055cdb24c6bec fp 0x00007fa6860fdfd0 dart::Profiler::DumpStackTrace(void*)+0x7c
  pc 0x000055cdb20fabf2 fp 0x00007fa6860fe0b0 dart::Assert::Fail(char const*, ...)+0x82
  pc 0x000055cdb252f69c fp 0x00007fa6860fe170 out/DebugX64/dart+0x1f9569c
  pc 0x000055cdb252ef8e fp 0x00007fa6860fe1e0 out/DebugX64/dart+0x1f94f8e
  pc 0x000055cdb252eefa fp 0x00007fa6860fe210 dart::Resolver::ResolveDynamicForReceiverClass(dart::Class const&, dart::String const&, dart::ArgumentsDescriptor const&, bool)+0x3a
  pc 0x000055cdb2536477 fp 0x00007fa6860fe280 dart::InlineCacheMissHelper(dart::Class const&, dart::Array const&, dart::String const&)+0xb7
  pc 0x000055cdb25388e2 fp 0x00007fa6860fe2e0 dart::PatchableCallHandler::ResolveTargetFunction(dart::Object const&)+0x372
  pc 0x000055cdb2538b00 fp 0x00007fa6860fe350 dart::PatchableCallHandler::ResolveSwitchAndReturn(dart::Object const&)+0x20
  pc 0x000055cdb2539126 fp 0x00007fa6860fe570 dart::DRT_InlineCacheMissHandlerOneArg(dart::NativeArguments)+0x2b6
  pc 0x00007fa68da823e2 fp 0x00007fa6860fe5b0 Unknown symbol
  pc 0x00007fa68da83340 fp 0x00007fa6860fe600 Unknown symbol
  pc 0x00007fa687e2b8d1 fp 0x00007fa6860fe668 Unknown symbol
  pc 0x00007fa687e2b6e0 fp 0x00007fa6860fe6e8 Unknown symbol
  pc 0x00007fa687e2b42c fp 0x00007fa6860fe718 Unknown symbol
  pc 0x00007fa687e256af fp 0x00007fa6860fe778 Unknown symbol
  pc 0x00007fa687e25088 fp 0x00007fa6860fe7b0 Unknown symbol
  pc 0x00007fa68da8286d fp 0x00007fa6860fe828 Unknown symbol
  pc 0x000055cdb234be76 fp 0x00007fa6860fe8c0 dart::DartEntry::InvokeCode(dart::Code const&, dart::Array const&, dart::Array const&, dart::Thread*)+0x116
  pc 0x000055cdb234ba65 fp 0x00007fa6860fe930 dart::DartEntry::InvokeFunction(dart::Function const&, dart::Array const&, dart::Array const&, unsigned long)+0x135
  pc 0x000055cdb234ec74 fp 0x00007fa6860fe990 dart::DartLibraryCalls::HandleMessage(dart::Object const&, dart::Instance const&)+0x1d4
  pc 0x000055cdb238b4ae fp 0x00007fa6860feb80 dart::IsolateMessageHandler::HandleMessage(std::__2::unique_ptr<dart::Message, std::__2::default_delete<dart::Message> >)+0x54e
  pc 0x000055cdb23cdda1 fp 0x00007fa6860fec10 dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool)+0x221
  pc 0x000055cdb23cecc9 fp 0x00007fa6860fec90 dart::MessageHandler::TaskCallback()+0x359
  pc 0x000055cdb2586e8b fp 0x00007fa6860fed30 dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)+0x1cb
  pc 0x000055cdb25878c9 fp 0x00007fa6860fed80 dart::ThreadPool::Worker::Main(unsigned long)+0x179
  pc 0x000055cdb24bf6b9 fp 0x00007fa6860feed0 out/DebugX64/dart+0x1f256b9
-- End of DumpStackTrace
[exit     : sp(0) fp(0x7fa6860fe5b0) pc(0)]
[stub     : sp(0x7fa6860fe5c0) fp(0x7fa6860fe600) pc(0x7fa68da83340)]
[dart     : sp(0x7fa6860fe610) fp(0x7fa6860fe668) pc(0x7fa687e2b8d1) dart:core__StringBase@0150898__interpolate@0150898 ]
[dart     : sp(0x7fa6860fe678) fp(0x7fa6860fe6e8) pc(0x7fa687e2b6e0) file:///b/s/w/ir/tests/standalone_2/out_of_memory_recovery_test.dart_::_handleMessage ]
[dart     : sp(0x7fa6860fe6f8) fp(0x7fa6860fe718) pc(0x7fa687e2b42c) file:///b/s/w/ir/tests/standalone_2/out_of_memory_recovery_test.dart_::_handleMessage_handleMessage ]
[dart     : sp(0x7fa6860fe728) fp(0x7fa6860fe778) pc(0x7fa687e256af) dart:core__Closure@0150898_dyn_call ]
[dart     : sp(0x7fa6860fe788) fp(0x7fa6860fe7b0) pc(0x7fa687e25088) dart:isolate__RawReceivePortImpl@1026248__handleMessage@1026248 ]
[entry    : sp(0x7fa6860fe7c0) fp(0x7fa6860fe828) pc(0x7fa68da8286d)]

--- Re-run this test:
python tools/test.py -n dartk-linux-debug-x64 standalone_2/out_of_memory_recovery_test

/cc @aam It seems related to splitting class state into allocate-finalized and finalized. Could you look into it?

[aam]

I believe this is stale now.

[aam]

Looking at call stack we do EnsureClassIsFinalized in Resolve called from ResolveTargetFunction https://github.com/dart-lang/sdk/blob/master/runtime/vm/runtime_entry.cc#L2084 before InlineCacheMissHelper runs.

So this looks like a race of some sort which potentially got addressed since this flaky crash happened.

[mkustermann]

So this looks like a race of some sort which potentially got addressed since this flaky crash happened.

Hmm, could we then bisect to a commit that fixed it?

Looking at call stack we do EnsureClassIsFinalized in Resolve called from ResolveTargetFunction https://github.com/dart-lang/sdk/blob/master/runtime/vm/runtime_entry.cc#L2084 before InlineCacheMissHelper runs.

That check has been there for a long time. So the crash above has happened with this check.

Could it possibly be a usage of set_state() without holding a lock (e.g. T1 is finalizing while holding lock, T2 reads state bits (finalized: false), T1 sets finalized bit, T2 sets state bits (finalized: false))?

[aam]

Hmm, could we then bisect to a commit that fixed it?

I will see if I can reproduce the crash.

[aam]

I was able to reproduce the crash on ToT, so yeah, it's not fixed. It's a good guess regarding set_state being racy.

https://dart-review.googlesource.com/c/sdk/+/188362 should protect updating of the state via set_is_fields_marked_nullable in AllocateObject.