[REQUEST]: Document Elastic Agent Upgrade Fails on Windows with status 0xc0000142

[gabriellandau]

Description

We should publicly document this so customers can find it via Google and self-service without having to contact support.

Title

Elastic Agent Upgrade Fails on Windows with status 0xc0000142

Description

During an Elastic Agent upgrade, Agent spawns a "watcher" process that monitors the upgrade process. As with all detached console processes, Windows will attempt to create a temporary console for this process. If Windows cannot create this console, then the watcher process initialization will fail with a return code of 0xc0000142 (STATUS_DLL_INIT_FAILED), resulting in an upgrade failure. Agent will log this error at info level:

Affected Versions

All versions of Elastic Agent are affected. This is not a bug in Agent.

Root Cause

A process on the system running as a service is allocating but failing to release windowing resources, exhausing the Windows Desktop Heap. Desktop heap exhaustion can occur when a service repeatedly calls windowing APIs such as CreateWindowEx or RegisterClassExW without releasing the resources afterwards. Elastic Agent runs as a Windows service and uses the service desktop; it shares the desktop heap with all other services. A windowing resource leak in any of those services can affect Elastic Agent.

Note

Interactively-run instances of elastic-agent.exe are not subject to this limitation. Only instances which run as services are potentially affected.

Workarounds

Update Agent Immediately After A System Reboot

A system reboot will destroy and recreate each desktop heap, resolving any prior exhaustion. Since many memory leaks are gradual, updating Agent immediately after a system reboot might allow Agent to update before the leaking application exhausts the desktop heap.

Note

A cold startup will reset kernel memory, but wake-from-hibernation and fast startups will not. A regular reboot (e.g. shutdown /r /t 0) will result in a cold startup and reset the desktop heap.

Update Third-Party Service Applications

Unfortunately, standard Windows tools such as Task Manager and Process Explorer do not attribute desktop heap usage by application, so users troubleshooting this issue must consider all third-party processes running as services. You can use the following Powershell to list these applications:

PS C:\> Get-Process | Where {$_.SI -eq 0} | Where {$_.MainModule.FileVersionInfo.ProductName -and (-not (($_.MainModule.FileVersionInfo.CompanyName -eq "Microsoft Corporation") -and ($_.MainModule.FileVersionInfo.ProductName -like "*Windows*"))) } | ForEach-Object { $_.MainModule.FileVersionInfo.ProductName + ' - ' + $_.Path }

Microsoft Edge Update - C:\Program Files (x86)\Microsoft\EdgeUpdate\MicrosoftEdgeUpdate.exe
VMware Guest Authentication - C:\Program Files\VMware\VMware Tools\VMware VGAuth\VGAuthService.exe
VMware SVGA 3D - C:\WINDOWS\system32\vm3dservice.exe
VMware Tools - C:\Program Files\VMware\VMware Tools\vmtoolsd.exe

Install any updates from these applications' manufacturers.

Terminate or Uninstall Third-Party Service Applications

Terminating a process will release its desktop heap resources. Users can try terminating or uninstalling non-critical third-party service applications before applying the update.

Note

The Agent update process does not require a significant amount of desktop heap resources. Therefore, a successful Agent upgrade after terminating or uninstalling an application does not necessary mean that the application was exhausting the desktop heap.

Resize the Desktop Heap

Microsoft documented a method to increase the size of the desktop heap:

The third SharedSection value is the size of the desktop heap for each desktop that is associated with a non-interactive window station. The value is in kilobytes (KB).

If the root cause is an application memory leak, then increasing the size of the desktop heap may only postpone heap exhaustion, so we do not consider this a long-term solution.

Resources

https://learn.microsoft.com/en-us/troubleshoot/windows-server/performance/desktop-heap-limitation-out-of-memory
https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/distinguishing-fast-startup-from-wake-from-hibernation
https://knowledge.broadcom.com/external/article/257078/faulting-application-name-scheduleproces.html#:~:text=The%20only%20remaining%20c0000142%20failure,if%20the%20problem%20is%20reproducible

Collaboration

Other (please describe below in the Point of contact section).

Point of contact.

Main contact: @gabriellandau

[kilfoyle]

Thanks a lot @gabriellandau! We'll get to this very soon.