Dart VM Crash: Trying to evaluate expression on optimized out variable crashes the VM

[jensjoha]

Reproduction:

bool debug = false;

main() {
  for(int i = 0; i < 50000; i++) {
    foo();
  }
  debug = true;
  foo();
}

foo() {
  List<int> data = [1, 2, 3];
  for(int i in data) {
    if (bar(i)) {
      break;
    }
  }
}

bool bar(int i) {
  if (i == 2) {
    if (debug) {
      print("set a breakpoint here!");
      print("woke up");
    }
    return true;
  }
  return false;
}

Start it, e.g. like this

$ out/ReleaseX64/dart --pause_isolates_on_start --enable-vm-service --serve-observatory --disable-service-auth-codes t.dart

open e.g. observatory and set a breakpoint on the print("set a breakpoint here!"); line and continue execution.
Go to the "foo" frame (frame 1 in observatory; "foo" in the "call stack" in devtools) and evaluate data.length:

../../runtime/vm/object.h: 8309: error: Handle check failed: saw <optimized out> expected Instance
version=3.3.0-edge.0be8cb24f20b4335a91c364cac27de9b7a57b635 (be) (Thu Nov 9 01:15:50 2023 +0000) on "linux_x64"
pid=1569475, thread=1569492, isolate_group=main(0x55cae705c570), isolate=main(0x55cae705ed40)
os=linux, arch=x64, comp=no, sim=no
isolate_instructions=55cae5e56e80, vm_instructions=55cae5e56e80
fp=7f143f57bce0, sp=7f143f57bba8, pc=55cae605f2dc
  pc 0x000055cae605f2dc fp 0x00007f143f57bce0 dart::Profiler::DumpStackTrace+0x7c
  pc 0x000055cae5e57074 fp 0x00007f143f57bdc0 dart::Assert::Fail+0x84
  pc 0x000055cae5f3d709 fp 0x00007f143f57bdf0 dart::Instance::CheckedHandle+0xa9
  pc 0x000055cae60a23c4 fp 0x00007f143f57c370 dart::DRT_InlineCacheMissHandlerOneArg+0xa4
  pc 0x00007f1467782a03 fp 0x00007f143f57c3b8 Unknown symbol
  pc 0x00007f14677839c7 fp 0x00007f143f57c408 Unknown symbol
  pc 0x00007f1467124b58 fp 0x00007f143f57c438 Unknown symbol
  pc 0x00007f1467782e46 fp 0x00007f143f57c4b0 Unknown symbol
  pc 0x000055cae5f94fd3 fp 0x00007f143f57c510 dart::DartEntry::InvokeFunction+0x163
  pc 0x000055cae5fff08f fp 0x00007f143f57c8a0 dart::Instance::EvaluateCompiledExpression+0x3cf
  pc 0x000055cae5f99f7d fp 0x00007f143f57c920 dart::ActivationFrame::EvaluateCompiledExpression+0x12d
  pc 0x000055cae60b5b99 fp 0x00007f143f57c9b0 dart::EvaluateCompiledExpression+0x349
  pc 0x000055cae60abebd fp 0x00007f143f57cfc0 dart::Service::InvokeMethod+0x30d
  pc 0x000055cae60ac401 fp 0x00007f143f57cfe0 dart::Service::HandleIsolateMessage+0x11
  pc 0x000055cae5fb2eb2 fp 0x00007f143f57d570 dart::IsolateMessageHandler::HandleMessage+0x1f2
  pc 0x000055cae5fd52f6 fp 0x00007f143f57d5e0 dart::MessageHandler::HandleMessages+0x116
  pc 0x000055cae5fd5683 fp 0x00007f143f57d620 dart::MessageHandler::HandleOOBMessages+0x43
  pc 0x000055cae62af74b fp 0x00007f143f57d6b0 Dart_HandleServiceMessages+0x11b
  pc 0x000055cae5fb72ad fp 0x00007f143f57d6f0 dart::Isolate::PauseEventHandler+0x9d
  pc 0x000055cae5f98021 fp 0x00007f143f57d8a0 dart::Debugger::Pause+0x101
  pc 0x000055cae5fa0350 fp 0x00007f143f57dd60 dart::Debugger::PauseBreakpoint+0x410
  pc 0x000055cae60a11fd fp 0x00007f143f57e350 dart::DRT_BreakpointRuntimeHandler+0x12d
  pc 0x00007f1467782a03 fp 0x00007f143f57e390 Unknown symbol
  pc 0x00007f1467783841 fp 0x00007f143f57e3c0 Unknown symbol
  pc 0x00007f1467123da2 fp 0x00007f143f57e3f0 Unknown symbol
  pc 0x00007f146712431e fp 0x00007f143f57e440 Unknown symbol
  pc 0x00007f1467124180 fp 0x00007f143f57e470 Unknown symbol
  pc 0x00007f1467122ae6 fp 0x00007f143f57e498 Unknown symbol
  pc 0x00007f1467122a13 fp 0x00007f143f57e4f8 Unknown symbol
  pc 0x00007f14671218d3 fp 0x00007f143f57e538 Unknown symbol
  pc 0x00007f1467121616 fp 0x00007f143f57e5a0 Unknown symbol
  pc 0x00007f1467120ebc fp 0x00007f143f57e5f8 Unknown symbol
  pc 0x00007f1467782e46 fp 0x00007f143f57e670 Unknown symbol
  pc 0x000055cae5f94fd3 fp 0x00007f143f57e6d0 dart::DartEntry::InvokeFunction+0x163
  pc 0x000055cae5f96a06 fp 0x00007f143f57e710 dart::DartLibraryCalls::HandleMessage+0x126
  pc 0x000055cae5fb2f7f fp 0x00007f143f57eca0 dart::IsolateMessageHandler::HandleMessage+0x2bf
  pc 0x000055cae5fd52f6 fp 0x00007f143f57ed10 dart::MessageHandler::HandleMessages+0x116
  pc 0x000055cae5fd58e8 fp 0x00007f143f57ed60 dart::MessageHandler::TaskCallback+0x1e8
  pc 0x000055cae60d2c4b fp 0x00007f143f57ede0 dart::ThreadPool::WorkerLoop+0x14b
  pc 0x000055cae60d2ee2 fp 0x00007f143f57ee10 dart::ThreadPool::Worker::Main+0x72
  pc 0x000055cae605c326 fp 0x00007f143f57eed0 dart::ThreadStart+0xd6
-- End of DumpStackTrace
  pc 0x0000000000000000 fp 0x00007f143f57c3b8 sp 0x0000000000000000 [Stub] CallToRuntime
  pc 0x00007f14677839c7 fp 0x00007f143f57c408 sp 0x00007f143f57c3c8 [Stub] OneArgCheckInlineCache
  pc 0x00007f1467124b58 fp 0x00007f143f57c438 sp 0x00007f143f57c418 [Unoptimized] :Eval
  pc 0x00007f1467782e46 fp 0x00007f143f57c4b0 sp 0x00007f143f57c448 [Stub] InvokeDartCode
  pc 0x0000000000000000 fp 0x00007f143f57e390 sp 0x0000000000000000 [Stub] CallToRuntime
  pc 0x00007f1467783841 fp 0x00007f143f57e3c0 sp 0x00007f143f57e3a0 [Stub] UnoptStaticCallBreakpoint
  pc 0x00007f1467123da2 fp 0x00007f143f57e3f0 sp 0x00007f143f57e3d0 [Unoptimized] bar
  pc 0x00007f146712431e fp 0x00007f143f57e440 sp 0x00007f143f57e400 [Optimized] foo
  pc 0x00007f1467124180 fp 0x00007f143f57e470 sp 0x00007f143f57e450 [Optimized] main
  pc 0x00007f1467122ae6 fp 0x00007f143f57e498 sp 0x00007f143f57e480 [Unoptimized] main
  pc 0x00007f1467122a13 fp 0x00007f143f57e4f8 sp 0x00007f143f57e4a8 [Unoptimized] _Closure@0150898.dyn:call
  pc 0x00007f14671218d3 fp 0x00007f143f57e538 sp 0x00007f143f57e508 [Unoptimized] _delayEntrypointInvocation@1026248.<anonymous closure>
  pc 0x00007f1467121616 fp 0x00007f143f57e5a0 sp 0x00007f143f57e548 [Unoptimized] _Closure@0150898.dyn:call
  pc 0x00007f1467120ebc fp 0x00007f143f57e5f8 sp 0x00007f143f57e5b0 [Unoptimized] _RawReceivePort@1026248._handleMessage@1026248
  pc 0x00007f1467782e46 fp 0x00007f143f57e670 sp 0x00007f143f57e608 [Stub] InvokeDartCode
Aborted

I'm on 0be8cb2 but it also happens in e.g. Dart 3.0.0.

[mkustermann]

/cc @aam could you look at this?

[derekxu16]

This crash is caused by Object::optimized_out() being passed as an argument to an eval_function that calls a getter on that argument. One way to fix it would be to return Object::optimized_out() instead of running eval_function in cases where Object::optimized_out() will be passed as an argument to eval_function, and we know that eval_function will try to use that argument. I know that @mkustermann added is_receiver_used() to be able to detect when an eval_function uses the receiver, would we have to add something similar to be able to check whether an arugment is used? Is that even feasible? @aam, @alexmarkov, @mkustermann, please let me know what you think.

[derekxu16]

Oh also, implementing #53687 is an alternative way to fix this specific crash, but I imagine we'll still need the checks described above to handle cases unrelated to pruning dead locals.

[alexmarkov]

In order to support expression evaluation involving extension types, CFE will use static types of variables at given source position instead of runtime types of variables. That would make dynamic (runtime) types of variables optional. After that, we can change the VM to provide only a subset of runtime types of variables, filtering out all optimized out. If subset of available runtime variables would not be enough, front-end would fail compiling the expression and issue a nice readable error. If certain optimized out variable is not used in the expression, then it would be compiled and executed successfully. We can also change the Procedure which is created by the front-end for the evaluated expression to receive all variable values as named parameters, so VM can match their names against available non-optimized variables and issue an error if used variable is not available/optimized out.

@johnniwinther WDYT?

[jensjoha]

Our plan (at least currently) is to (somehow) make use of both the runtime type and the static type.
It also currently looks like we're going to end in a state where we, when finding the static type of variables, will likely have a superset, e.g. we won't know for certain if there's a variable called 'foo', but if there is we'll know it has type 'SomeStaticType' --- and we'll thus still need the information about which variables are actually available.

After that, we can change the VM to provide only a subset of runtime types of variables, filtering out all optimized out.

Is anything stopping us from doing that now? If a variable is not actually available (for whatever reason) it seems to me the compiler shouldn't be told it exists.

[alexmarkov]

@jensjoha Yes, VM could filter optimized out variables even now. At least, we would avoid crash in this case. However, that would not make expression evaluation correct - we would get either 1) a compile-time error about not found variable, which is somewhat misleading; 2) a silent incorrect behavior when missing local variable shadows another variable/getter from outer scope, which is silently used if variable is missing. Currently this already happens inside closures for variables which are not captured, but it would be nice to fix that too.

Related issues: #53087, #45913, #53688.

[jensjoha]

So it should communicate that these variables do exist in scope but is unavailable --- say have a special "_OptimizedOut" type, or be in a separate list so the expression compiler could define them as constants of a certain type or something.

Was that basically what you suggested but I didn't understand in #53996 (comment)?

[alexmarkov]

So it should communicate that these variables do exist in scope but is unavailable --- say have a special "_OptimizedOut" type, or be in a separate list so the expression compiler could define them as constants of a certain type or something.

No. VM currently doesn't have information about variables inside closures which were not captured.

If needed, CFE should be able to figure out the complete set of variables in scope by given source position, in order to get their static types (and maybe even promoted types) and correctly distinguish between references to local variables and getters from outer scope.

VM would provide runtime types for a subset of variables which are actually available at run time (which were not optimized out and not omitted from closure context). CFE can issue an error if variable value is not available at run time (or generate a Procedure which would take those values as named parameters, and VM can issue such error).

[derekxu16]

The commit above filters out optimized out variables when building the scope in ActivationFrame::BuildParameters. After making that change, running the example in the first comment of this issue no longer makes the compiler crash. Instead, it makes the compiler return an error that says:

> data.length
  evaluateInFrame: (113) Expression compilation error
  org-dartlang-debug:synthetic_debug_expression:1:1: Error: Undefined name 'data'.
  data.length
  ^^^^

I spoke with @alexmarkov and we think that the error should be made more specialized and user-readable.

[a-siva]

I guess the remaining work is in the front end so updated issue accordingly.

//cc @johnniwinther

[johnniwinther]

@jensjoha Can you take a look at this?