Debugging lsass with x64dbg

[dirkjanm]

Hey,

I'm trying to understand some Windows internals and looking at logon packages and stuff, which are part of the lsass process.
I'd like to do this with x64dbg and actually have done so in the past, but I can't get it to work on recent Windows 10 builds. As you may be aware lsass is responsible for a lot of the security things taking place in Windows, so debugging it is quite tricky.
At the moment I attach to the lsass process, x64dbg freezes and stops doing whatever it normally does, at which point a hard reboot is the only option since the whole OS freezes. Likely this is because attaching to a process suspends the threads of this process, but the attach process itself also needs a functioning lsass process for whatever reason, thus causing a deadlock. I've attached x64dbg to x64dbg to figure out where the issue is taking place, and it seems to be the DebugActiveProcess call in kernel32 that hangs the whole thing.

I tried this with various builds of x64dbg ranging from a build from january to the latest snapshot (snapshot_2020-07-29_04-15) and also with various Windows 10 versions (for example 1909) and they all show the same behavior. Interesting enough I did some debugging successfully on lsass with Server 2016 last year (but not sure which build), so it did work at some point.

Is there a way to work around this? I'd love to be able to use x64dbg instead of having to go into kernel debug mode with other tools.

[mrexodia]

You could try something like https://github.com/brock7/xdbg or cheat engine's SEH debugging. I don't really know how to solve a deadlock in the kernel, so if you find anything let me know.

[dirkjanm]

I couldn't get it working with xdbg, same result. However, I found out the way to do it:

• Run x64dbg as SYSTEM (using for example psexec)
• Make sure initial breakpoints are disabled (just to be sure, may work without as well)
• Make sure debug privilege is not enabled in the settings (having it enabled will make x64dbg crash even when SYSTEM). Not sure if it actively removes the privilege after this setting is disabled, so I restarted x64dbg to be sure.

Apart from that I don't think you can fix it from x64dbg as it seems related to the kernel indeed.

[Mattiwatti]

I looked into this because I was highly skeptical of @dirkjanm's claim that disabling the debug privilege option somehow fixed this problem. Why is this? Because a process started as SYSTEM (at least, by psexec) will already have SE_DEBUG_PRIVILEGE enabled by default, so this makes no difference. Furthermore, it simply doesn't make sense that enabling a privilege in your token would cause something as drastic as a system-wide deadlock. However, I tested it and it is actually completely true!

Fortunately it was quite easy to find the cause (highlighted in red - ignore the top part of the stack as my symbols for this version of ntoskrnl seem to be missing):

EngineSetDebugPrivilege, the function that enables or disables SE_DEBUG_PRIVILEGE in TitanEngine, is called to enable the privilege. It starts by using LookupPrivilegeValueW to convert the name of the privilege to a LUID. Higher up in the stack you see advapi32.dll making an ALPC RPC call to another process to look up this value. Three guesses which process is responsible for handling this call... ding ding, it's lsass.exe, which now can't respond.

I can't really blame whoever wrote this function, since this is exactly how the MSDN sample code does it, but it's a little-known secret that LookupPrivilegeValueW is a completely pointless function. All SE_XXX_PRIVILEGE values are simple numerical constants that have never been changed. Fixed in this TitanEngine PR: x64dbg/TitanEngine#6.

With the fixed TitanEngine, you can now attach to lsass.exe with the debug privilege option enabled, both as SYSTEM and as a regular administrator. However, be warned that detaching from the process kills it, which will cause Windows to display a 'reboot in 1 minute' message since lsass is a critical process. This only happens when running as administrator, not as SYSTEM. I may look into this some other time as it's really a separate issue.

[dirkjanm]

Ha nice find! looking forward to be able to use it without jumping through hoops to run it as SYSTEM. Appreciate you looking into it :)

[Mattiwatti]

Created another TitanEngine PR with a tentative fix for lsass dying on detach: x64dbg/TitanEngine#7. I say tentative because I can't really explain why this fix works or why there was a difference between admin and SYSTEM users re: this. But I tested detaching multiple times with this patch and lsass no longer seems to die.

Note for @dirkjanm: the undocumented SafeAttach option must be enabled in x64dbg.ini for this change to have an effect. Example INI snippet:

[Engine]
SafeAttach=1