lldb-mi freezes on MacOS when breaking near uninitialised variables

[andrewfindon]

Type: Debugger

Describe the bug

• OS and Version: MacOS 11.2.3 (Big Sur)
• VS Code Version: 1.5.4.3
• C/C++ Extension Version: 1.2.2
• I have recreated on both Intel and Apple Silicon (Rosetta) MacBook Pros.

During debugging, if stopping on (or stepping in to) a frame containing certain types of uninitialised variables, lldb-mi gets stuck at 100% cpu and starts to consume multiple GBs of memory.

I've encountered this a a few times in different code, I've condensed the latest example in to a tiny snippet (below), using a std::initialiser_list<int>.

I couldn't reproduce using the lldb command line. Xcode is fine too, but I notice it uses lldb-rpc-server to communicate with debugserver instead of lldb-mi.

I also tried using the CodeLLDB extension (https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) which doesn't suffer with the same problem (and doesn't use lldb-mi) - but interestingly, it does try to inspect the values of the initializer_list but times out after a short time.

If debugging is stopped while this is going on, the lldb-mi process continue to spin with high cpu usage in the background.

This may be related to the following:

• LLDB debugger hangs (using 100% cpu) at launch if std::map is uninitialised  #5805
• c++ lldb debugger seems hangs #860
• lldb-mi becomes unresponsive, uses lots of memory #1899

The issue may not be directly related to cpptools (could be an lldb-mi bug?), but the extension suffers as a result - in that case, maybe there is something between the interaction with lldb-mi and cpptools that could avoid triggering the bug.

To Reproduce

(I've also prepared a tiny repo for easily reproducing the problem here: https://github.com/andrewfindon/lldbmi-freeze)

To trigger the problem, step in to the freeze_up() function in the example below. The lldb-mi process should spike in cpu/memory usage and vscode will appear to freeze waiting for local variables to resolve.

main.cpp

#include <initializer_list>

void freeze_up()
{
	auto bad_var = {1, 2, 3};		// A breakpoint here will trigger the issue.
}						// A breakpoint here is fine (bad_var is initialised).

int main() {
	freeze_up();
	
	return 0;
}

launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/freeze",
            "args": [],
            "stopAtEntry": false,
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "lldb",
            "preLaunchTask": "clang++ build",

            "logging": {
                "engineLogging": true,
                "trace": true,
                "traceResponse": true
            }
        }
    ]
}

tasks.json

{
    "version": "2.0.0",
    "tasks": [
        {
            "type": "shell",
            "label": "clang++ build",
            "command": "/usr/bin/clang++",
            "args": [
                "main.cpp",
                "-arch",
                "x86_64",
                "-std=c++17",
                "-Wall",
                "-g",
                "-o",
                "freeze",
            ],
            "problemMatcher": [
                "$gcc"
            ],
            "group": "build",
        }
    ]
}

The tail of my engine logging output at the point where the debugger has frozen on a breakpoint, looks like:

--> E (stopped): {"type":"event","event":"stopped","body":{"reason":"step","threadId":1,"allThreadsStopped":true,"source":{"name":"main.cpp","path":"/Users/andrewfindon/Documents/lldbmi-crash/main.cpp","sources":[],"checksums":[]},"line":6,"column":1},"seq":641}
<--   C (threads-12): {"command":"threads","type":"request","seq":12}
--> R (threads-12): {"type":"response","request_seq":12,"success":true,"command":"threads","body":{"threads":[{"id":1,"name":"Thread #1"}]},"seq":644}
<--   C (stackTrace-13): {"command":"stackTrace","arguments":{"threadId":1,"startFrame":0,"levels":20},"type":"request","seq":13}
--> R (stackTrace-13): {"type":"response","request_seq":13,"success":true,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"freeze!freeze_up()","source":{"name":"main.cpp","path":"/Users/andrewfindon/Documents/lldbmi-crash/main.cpp","sources":[],"checksums":[]},"line":6,"column":1,"moduleId":1},{"id":1001,"name":"freeze!main","source":{"name":"main.cpp","path":"/Users/andrewfindon/Documents/lldbmi-crash/main.cpp","sources":[],"checksums":[]},"line":12,"column":1,"moduleId":1},{"id":1002,"name":"libdyld.dylib!start","line":0,"column":0,"moduleId":12},{"id":1003,"name":"libdyld.dylib!start","line":0,"column":0,"moduleId":12}],"totalFrames":4},"seq":647}
<--   C (scopes-14): {"command":"scopes","arguments":{"frameId":1000},"type":"request","seq":14}
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33618) <-1018-stack-list-variables 0 --thread 1 --frame 0\n"},"seq":650}
1: (33618) <-1018-stack-list-variables 0 --thread 1 --frame 0
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33631) ->1018^done,variables=[{name=\"bad_var\"}]\n"},"seq":652}
1: (33631) ->1018^done,variables=[{name="bad_var"}]
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33631) ->(gdb)\n"},"seq":654}
1: (33631) ->(gdb)
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33632) 1018: elapsed time 13\n"},"seq":656}
1: (33632) 1018: elapsed time 13
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33657) <-1019-var-create - - \"bad_var\" --thread 1 --frame 0\n"},"seq":658}
1: (33657) <-1019-var-create - - "bad_var" --thread 1 --frame 0

Here's a trace of lldb-mi in Instruments - it's clear on the timeline when I stepped in to the freeze_up function:

Example debugging session:

[WardenGnaw]

lldb-mi seems to already have some issues being slow debugging as per #1899 (comment).

For Apple Silicon, it will be slow since both the Debug Adapter (MIEngine) and lldb-mi will both be running in emulation.

To fix the latter, it is dependent on #7035

[andrewfindon]

Please note, this issue applies equally to Intel machines (I mention Apple Silicon only to be thorough).

Also, I have left It in this high cpu, high memory state for "some time" (multiple minutes) - it doesn't seem to recover (so less about it being slow, and more about it getting stuck).

cpptools no longer responds, except for cancelling debugging (but this leaves lldb-mi running in the background eating cpu and rapidly draining battery).

This ~6 line sample is enough to break the debugging experience entirely, but personally I've encountered these symptoms a couple of times in actual code, although then I think it was an uninitialised std::vector that was the trigger.

Anecdotally it seems like the debugger is reaching in to the uninitiated memory forever to resolve values for the local variables view.

[ni4]

+1 for this issue, macOS Catalina 10.15.7, Intel CPU.
However:

• all variables in the function I'm trying to stop in seems to be initialized
• when put breakpoint in called function, I'm able to navigate, and when returning to the original function see this:

• at the moment of stop all of those variables are initialized
• if it helps, I'm building with -fsanitize=address,undefined. Removing this doesn't change the behaviour.

[Wdong04]

Same problem here. Would like to know how to address it.

[shiretu]

I have recompiled my lldb-mi with this patch:

diff --git a/src/MICmdCmdVar.cpp b/src/MICmdCmdVar.cpp
index 6712a48..e9ceab7 100644
--- a/src/MICmdCmdVar.cpp
+++ b/src/MICmdCmdVar.cpp
@@ -304,7 +304,8 @@ void CMICmdCmdVarCreate::CompleteSBValue(lldb::SBValue &vrwValue) {
   // And update its children
   lldb::SBType valueType = vrwValue.GetType();
   if (!valueType.IsPointerType() && !valueType.IsReferenceType()) {
-    const MIuint nChildren = vrwValue.GetNumChildren();
+    const auto temp = vrwValue.GetNumChildren();
+    const MIuint nChildren = temp > 64 ? 64 : temp;
     for (MIuint i = 0; i < nChildren; ++i) {
       lldb::SBValue member = vrwValue.GetChildAtIndex(i);
       if (member.IsValid())

[Ayasra]

Same problem here

[PenceGL]

I just experienced this a few minutes ago. I am on MacOS Monterey, v12.3.1, Intel CPU.
I was able to hit breakpoints, but then realized my debugger controls (Continue, Step Over, Step In) were completely unresponsive. I was able to press stop to quit debugging as @andrewfindon mentioned. But then my laptop tried to enter orbit... fans on full blast, CPU utilization pegged at 100%. Activity Monitor showed several instances (I guess because I had hit stop and rerun the debugger a few times) of lldb-mi using lots of memory and 100% CPU. Stopping them manually through Activity Monitor resolved the issue and the fan/utilization went back to normal levels.

A few quick Google searches led me here. I'm glad to see that it's an issue that's still getting attention.

[abhishekag0812]

This helped me to avoid the problem for now.
Looking forward for a permanent fix though.

[lp35]

Hey guys, do you have a solution for Linux users?
Cheers!

[vlebourl]

Hey, any news on this?

[shiretu]

Hey, any news on this?

I do not think this is a VSCode issue. This is truly a problem with the lldb. See my patch above. lldb is out-of-bounds-ing on walking uninitialized variables with garbage values on things like internal counters and sizes. My quick yet brutal fix is to limit these things to 64.

Unfortunately, VSCode tries to render all local variables in the current scope. And when it reaches one of those not-yet-initialized vars, it behaves like that.

[jpcofr]

As a workaround, I installed CodeLLDB. It fixed the issue. Still, vscode's own debugger got lldb to use all the memory in my machine, which in turn made it unresponsive and I had to restart it.

[garfieldnate]

I ran into this today. Since @shiretu says this is an lldb-mi issue, I guess the place to track it is here?

[heafox]

CodeLLDB is fine

[H-G-Hristov]

CodeLLDB is fine

I have a bad feeling that CodeLLDB doesn't work now too.
vadimcn/codelldb#987
vadimcn/codelldb#998

[H-G-Hristov]

@bobbrow @WardenGnaw Is there any change of this getting fixed? It seems this issue is specific to Apple Silicon. What about Windows on arm?

[mlangiu]

It seems this issue is specific to Apple Silicon.

Nope this happens also on older Intel macs

[ni4]

+1, I'm on Intel Mac and have the same issue. It hurts so bad during debug that in a new code I always initialize all local vars :)

[kkHAIKE]

I have recompiled my lldb-mi with this patch:

diff --git a/src/MICmdCmdVar.cpp b/src/MICmdCmdVar.cpp
index 6712a48..e9ceab7 100644
--- a/src/MICmdCmdVar.cpp
+++ b/src/MICmdCmdVar.cpp
@@ -304,7 +304,8 @@ void CMICmdCmdVarCreate::CompleteSBValue(lldb::SBValue &vrwValue) {
   // And update its children
   lldb::SBType valueType = vrwValue.GetType();
   if (!valueType.IsPointerType() && !valueType.IsReferenceType()) {
-    const MIuint nChildren = vrwValue.GetNumChildren();
+    const auto temp = vrwValue.GetNumChildren();
+    const MIuint nChildren = temp > 64 ? 64 : temp;
     for (MIuint i = 0; i < nChildren; ++i) {
       lldb::SBValue member = vrwValue.GetChildAtIndex(i);
       if (member.IsValid())

Is it working normally?

[sunshaoce]

I have recompiled my lldb-mi with this patch:

diff --git a/src/MICmdCmdVar.cpp b/src/MICmdCmdVar.cpp
index 6712a48..e9ceab7 100644
--- a/src/MICmdCmdVar.cpp
+++ b/src/MICmdCmdVar.cpp
@@ -304,7 +304,8 @@ void CMICmdCmdVarCreate::CompleteSBValue(lldb::SBValue &vrwValue) {
   // And update its children
   lldb::SBType valueType = vrwValue.GetType();
   if (!valueType.IsPointerType() && !valueType.IsReferenceType()) {
-    const MIuint nChildren = vrwValue.GetNumChildren();
+    const auto temp = vrwValue.GetNumChildren();
+    const MIuint nChildren = temp > 64 ? 64 : temp;
     for (MIuint i = 0; i < nChildren; ++i) {
       lldb::SBValue member = vrwValue.GetChildAtIndex(i);
       if (member.IsValid())

@shiretu Would you consider improving it and then submit it to lldb-mi?

[robUx4]

This is still happening with cpptools v1.18.5. I disabled the "Variables" view of the debugger so I can get some work done. Mouse hover on a variable still shows the variables when needed.