Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Package Manager Server runs on high CPU #3667

Open
MarvinNorway opened this issue Sep 25, 2023 · 70 comments
Open

Windows Package Manager Server runs on high CPU #3667

MarvinNorway opened this issue Sep 25, 2023 · 70 comments
Labels
Area-Performance Issue related to CPU or memory performance Issue-Bug It either shouldn't be doing this or needs an investigation.
Milestone

Comments

@MarvinNorway
Copy link

Brief description of your issue

I'm not sure if this is the right place to report this, but the "WinGET COM Server" with a child process of "WindowsPackageManagerServer.exe" is using up a lot of CPU on my system.
When killed, it just respawns immediately.

I'm not the only one with this issue:
https://www.reddit.com/r/techsupport/comments/162dj42/what_is_winget_com_server/

Steps to reproduce

Not sure how to reproduce.

Expected behavior

It shouldn't run on high CPU for extended periods of time.

Actual behavior

It runs on super high CPU all the time.

Environment

Microsoft.DesktopAppInstaller_1.20.2201.0_x64__8wekyb3d8bbwe
WindowsPackageManagerServer.exe version 1.20.2308.8001-v1.20WPMv1.5

Hope that helps...
@microsoft-github-policy-service microsoft-github-policy-service bot added the Needs-Triage Issue need to be triaged label Sep 25, 2023
@sba923
Copy link

sba923 commented Sep 25, 2023

Same here. The process keeps running after I close its parent (a pwsh.exe instance). I can't copy the log from %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir 'cos it's locked.

@denelon denelon added Issue-Bug It either shouldn't be doing this or needs an investigation. Area-Performance Issue related to CPU or memory performance and removed Needs-Triage Issue need to be triaged labels Sep 25, 2023
@MozPri
Copy link

MozPri commented Sep 28, 2023

Same here. WindowsPackageManagerServer.exe is using 25% of the CPU and can't be killed.
Microsoft.DesktopAppInstaller_1.21.2701.0

@PartedLine
Copy link

Same here, rather not have my OS frying my CPU

@saggitar
Copy link

Same here ...

@HyP3r-
Copy link

HyP3r- commented Sep 29, 2023

It is the same with me... when I end the process it is restarted immediately.

Environment

Microsoft.DesktopAppInstaller_1.21.2701.0_x64__8wekyb3d8bbwe

@MarvinNorway
Copy link
Author

MarvinNorway commented Sep 29, 2023

Not sure if it helps anyone, but I left it running over night, and then it was quiet the next day.
It happened one more time 1-2 days later, during the day, but I went out for a bit, and when I came back, it was over, as well.
Since then, it has been quiet. Would still be nice to know what the process does and if there's a way to stop this behavior...

@chylex
Copy link

chylex commented Sep 30, 2023

Mine has started crashing constantly, trying to stop it just restarts it and crashes again... how do I get rid of this???

obrazek

@gibbed
Copy link

gibbed commented Sep 30, 2023

Mine has started crashing constantly, trying to stop it just restarts it and crashes again... how do I get rid of this???

obrazek

Had this same issue. You've got a pretty outdated version, update "App Installer" via the Microsoft Store or WinGet as described here and the problem should go away.

@chylex
Copy link

chylex commented Sep 30, 2023

I have taken ownership of the file, disabled inheritance of permissions, and left the permissions like this:

obrazek

Permissions on my user are Read and Write, but not Read & Execute:

obrazek

Now the exe is no longer able to launch or restart itself, and I'm not getting spammed with errors anymore.

@MagicAndre1981
Copy link

Install the Windows Performance Toolkit from the Windows ADK (unselect all other entries) and capture a ETW trace by running WPRUI.exe, select First Level Triage and CPU Usage and click on Start. Capture 30-60s of the CPU usage, click Save to save the ETL (make sure you select the checkbox "Compress" to make the ETL file smaller).

Zip the ETL + PDB Folder(s) and share the zip via OneDrive link here.

@nimairsh
Copy link

nimairsh commented Oct 1, 2023

mine is using 10-15%. Here's the link to my etl+pdb if it helps.

@MagicAndre1981
Copy link

mine is using 10-15%.

CPU Usage (Sampled) is somehow missing, precise stack shows usage of WindowsPackageManager.dll, but the PDB is missing, so a team member which has access to the PDB should take a look at it.

@eduarddejong
Copy link

eduarddejong commented Oct 1, 2023

I can report a high CPU issue as well. It was running for a really long time at high 100%, while I am running a pretty modern desktop system.

However after a lot of minutes, it eventually stopped and went quiet, without any intervention by me (I deliberately decided to wait a long time to check if it would eventually stop indeed).

This gives me the impression that the process does not truely hang an infinate loop, but that it's performing some task just really, really inefficiently (is it doing some kind of super heavy search, like through the entire registry or something?).
And if that's something on a scheduled base, it might come back again (no idea yet) on other moments, impacting performance at unexpected moments on the user's system.

@denelon
Copy link
Contributor

denelon commented Oct 1, 2023

We believe we've identified the source of the high CPU events. We're looking to make changes in WinGet to lower the impact as well as working with partner teams to help make their calls and calling pattern more efficient.

@ViPeR5000
Copy link

Same here :(

@ondalin
Copy link

ondalin commented Oct 3, 2023

hello all

i just noticed this program today, also taking much cpu, after i removed some office updates from yesterday, it went away; will keep an eye on task manager, but since removing the updates and a reboot and while creating the account here and typing, the program didnt came back so far.

@ondalin
Copy link

ondalin commented Oct 4, 2023

update, it came back. damn

@mediser
Copy link

mediser commented Oct 4, 2023

hello there, I have the same problem, could anyone solve it??

@mediser
Copy link

mediser commented Oct 4, 2023

hello there, I have the same problem, could anyone solve it??

The last time I had this problem, what I did was restart Windows and after that it worked fine, at least until now...

@ktrance
Copy link

ktrance commented Oct 4, 2023

We believe we've identified the source of the high CPU events. We're looking to make changes in WinGet to lower the impact as well as working with partner teams to help make their calls and calling pattern more efficient.

Ok, you have an ETA on a fix for this? And please, not one that "lowers the impact", but one that fixes the issue.

@chvogel
Copy link

chvogel commented Oct 4, 2023

Same issue here. Started pretty much right after booting up, on a machine that I haven't used since last Friday.

Process has accumulated about 10min of CPU time (100% usage on a single core) and as I am writing this, it has now finally stopped.

I know this is like talking to a wall, but the amount of "random" background processes with zero UI indication of what is happening has really gotten out of hand in recent years. The more stuff constantly running on the background, the bigger the potential of things going wrong or eating up CPU time for no apparent reason.

Yes, user experience (in terms application performance) is not affected too much since most code still runs single-threaded and modern computers are routinely equipped with more than half a dozen cores. But especially on mobile/laptop devices, this causes a large nuisance due to high noise (for no visible reason), causing distraction and reduced battery life.

@ondalin
Copy link

ondalin commented Oct 5, 2023

what else seems to have dialed this down was to right click the sub process in task manager, go to details, right click again, set priority, below normal, reboot and when it came back, it didnt take as much cpu as before.

@drankinatty
Copy link

This problem hit today on a box that was just started and had not been up since last week. Seems like an update is to blame. WindowsPackageManagerServer.exe has been running for hours cycling between 5 and 25 percent CPU relentlessly:

windowspackagemanagerserver-cpu

The problem appears to be its continual connection with akami and an IP at 13.107.213.57, see, e.g.

windowspackagemanagerserver-network

Edge and MS Office 365 were two of the updates I saw come by. This needs to be fixed.

@joedf
Copy link

joedf commented Oct 7, 2023

This is very bad behaviour. Background update processes should never be this disruptive... This is bad oversight...

@Pootleris
Copy link

same porcodio

@rerigam
Copy link

rerigam commented Oct 9, 2023

How to remove the modern App WinGET: open PowerShell as admin and enter the following command:
"get-appxpackage Microsoft.DesktopAppInstaller | remove-appxpackage"

PS: i don't know why but the editor removed 2 asterisks from the command above, one before Microsoft and one after AppInstaller. You must insert them.

@drankinatty
Copy link

drankinatty commented Oct 31, 2023

Agreed. Effectively deleting the file on individual machines for those users just smart enough to know how to take ownership of a system file and renaming it is not a solution. It leaves every other windows user stuck with the problem. Good lord. Think of the 99.9+% of the installed base that has no clue about hacking file permissions. Think of your grandmother or grandfather or child puzzled, concerned their computer is broken because it is busy for hours on end.

@juniortarcisio
Copy link

While there's no patch/solution, another option is simply "suspend" the process using windows resource monitor, instead of killing it.

@denelon
Copy link
Contributor

denelon commented Nov 2, 2023

Hey all. We've identified an area where we can reduce the CPU utilization for this calling pattern. We're working on an update, and we'll publish it as soon as it's ready.

Update: We're performing some testing now to see if we can at least get some anecdotal measurements for the improvement.

@Girofox
Copy link

Girofox commented Nov 3, 2023

Ok, this is weird, I found another possible trigger:

  • Open "Windows Backup" app, then manually start process by clicking "Back Up".

In my case the logfile in AppData\Local\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir topped out at 1.5 Mb when the process eventually terminated.

image

@son0nline
Copy link

son0nline commented Nov 3, 2023

WindowsPackageManagerServer.exe always uses above 10% cpu and can't be killed.
Microsoft.DesktopAppInstaller_1.21.2771.0_x64__8wekyb3d8bbwe

image

@denelon
Copy link
Contributor

denelon commented Nov 3, 2023

Hey all, we've identified a couple of areas where we're able to make substantial improvements in the client. Our initial testing is showing a solid reduction in the time it takes for these processes to complete. It amounts to 5/6 of the time. Something on the order of ~1000 ms per iteration to ~130 ms per iteration on the hardware we're testing on. I'm expecting we will push this update out early next week if the remaining testing all passes.

@sba923
Copy link

sba923 commented Nov 3, 2023

Hey all, we've identified a couple of areas where we're able to make substantial improvements in the client. Our initial testing is showing a solid reduction in the time it takes for these processes to complete. It amounts to 5/6 of the time. Something on the order of ~1000 ms per iteration to ~130 ms per iteration on the hardware we're testing on. I'm expecting we will push this update out early next week if the remaining testing all passes.

Good job! Do you have an explanation for the fact it doesn't affect most machines? What is (are) the trigger(s) for that CPU consumption?

@MagicAndre1981
Copy link

Good job! Do you have an explanation for the fact it doesn't affect most machines? What is (are) the trigger(s) for that CPU consumption?

this is what I also would like to know. I run winget daily and never saw such a high CPU usage. I run it from classic cmd.exe on x64 Windows 10/11 German.

@raphaelcardoso
Copy link

Hey all, we've identified a couple of areas where we're able to make substantial improvements in the client. Our initial testing is showing a solid reduction in the time it takes for these processes to complete. It amounts to 5/6 of the time. Something on the order of ~1000 ms per iteration to ~130 ms per iteration on the hardware we're testing on. I'm expecting we will push this update out early next week if the remaining testing all passes.

I appreciate the optimization effort, and the initial results look promising.
However, based on the observed behavior, even with a 99.9% load reduction, it will still be an unacceptable load.
I suspect there may still be an underlying bug contributing to this.
Is there any connection between the Visual Studio update and the issue we're observing, or is it just a coincidence?

@SanderBouwhuis
Copy link

Like @raphaelcardoso I'm sceptical. For me, the process hung for at least 30 minutes before I managed to finally kill it by uninstalling some things. Endless loop * 1/6 is still an endless loop.

@raphaelcardoso
Copy link

[OFF-TOPIC]

Shifting the focus here, in situations like this where it's causing a single logical processor to run at 100% capacity (25% of my CPU), there should be a fast, automated rollback process.

It might be achievable using components already implemented in advanced technologies like Click-to-Run, containers, MDM, or the Microsoft Store.

In an emergency, authors of Microsoft Store apps should be able to revert to a previous version almost instantaneously, triggering a deployment as lightweight as sending low-priority push notifications and symlink swapping.

@jyoberle
Copy link

jyoberle commented Nov 4, 2023

How to remove the modern App WinGET: open PowerShell as admin and enter the following command: "get-appxpackage Microsoft.DesktopAppInstaller | remove-appxpackage"

PS: i don't know why but the editor removed 2 asterisks from the command above, one before Microsoft and one after AppInstaller. You must insert them.

I tried this command and after a few minutes I had no more Internet! I had to reinstall Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle (downloaded thanks to my phone) to get Internet back

@MagicAndre1981
Copy link

MagicAndre1981 commented Nov 8, 2023

ok, Microsoft made some changes in #3808 to improve it by caching RegEx calls. I also got this cpu usage now but only for a shorter time after running winget upgrade and here I saw icu regex calls. So hopefully we get this fix sone in an update.

@Girofox
Copy link

Girofox commented Nov 8, 2023

For me WindowsPackageManagerServer.exe runs 10 minutes on my laptop with around 25 % cpu usage when i manually trigger backup in Windows Backup app, winget upgrade never was an issue in my case.

image

@SanderBouwhuis
Copy link

SanderBouwhuis commented Nov 8, 2023

@jyoberle

How to remove the modern App WinGET: open PowerShell as admin and enter the following command: "get-appxpackage Microsoft.DesktopAppInstaller | remove-appxpackage"
PS: i don't know why but the editor removed 2 asterisks from the command above, one before Microsoft and one after AppInstaller. You must insert them.

I tried this command and after a few minutes I had no more Internet! I had to reinstall Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle (downloaded thanks to my phone) to get Internet back

I didn't encounter this problem after uninstalling this package.
Did you try to restart?

@jyoberle
Copy link

jyoberle commented Nov 8, 2023

No, I didn't try a restart.

@anonymous-explorer
Copy link

Windows 10 Pro.
Windows Package Manager uses one logical CPU (I have 2 cores, 4 logical CPUs and it uses 25% of the total CPU). It does not use any Disk I/O and almost no network.
I always log in with non-admin user.

Restarting Windows and logging in with Administrator account solves the problem. Some process (Wsappx ...) runs for less than a minute with a high CPU usage and then CPU usage returns to normal.

After restarting with normal user "Please wait" appears (some update are being installed?) and runs fine until the next Windows update.

@Girofox
Copy link

Girofox commented Nov 10, 2023

Ok, that sounds very interesting, seems like something inside Windows "fixes" itself running with admin account.

Do you mean a normal admin account or the "true" admin account of Windows by running Net user Administrator /active: yes in command line? Because my regular user account already has admin rights but user account control gets triggered.

What happens when you open Windows Backup app and manually start backup there? Does that trigger the 25 percent cpu usage process?

By the way i have Windows 11 Edu and it is impossible to remove winget there with powershell. And removing winget breaks some things even in Windows 10. What did work in my case is removing the winget apps folder in %ProgramFiles%\WindowsApps with powerrun and then reinstalling winget over powershell but this did not help either (Warning: deleting any folders or files in WindowsApps can break things so not advised).

@denelon
Copy link
Contributor

denelon commented Nov 10, 2023

Hey all, we've almost completed our internal testing.

We will publish the update here at GitHub (for people actively seeking the update) and to Windows Insider Beta and Windows Insider Release Preview. After a few days of feedback, this will be pushed out to everyone.

@denelon
Copy link
Contributor

denelon commented Nov 11, 2023

@Girofox
Copy link

Girofox commented Nov 11, 2023

This looks promising, downgraded from pre-release version to v1.6.3133 and I can confirm the issue seems gone!

Again as a test I manually started backup in Windows Backup app.

CPU usage of WindowsPackageManagerServer.exe now stays below 10 % and finishes in less than 2 minutes. I used SystemInformer for that graph, Process Explorer should work too.

Downgrading was a bit complicated and commands winget list and winget upgrade only worked after deleting the .db file in %LocalAppData%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\Microsoft.Winget.Source_8wekyb3d8bbwe, this database seems to store which packages you recently installed via winget.

image

@sba923
Copy link

sba923 commented Nov 11, 2023

Hey all, we've almost completed our internal testing.

We will publish the update here at GitHub (for people actively seeking the update) and to Windows Insider Beta and Windows Insider Release Preview. After a few days of feedback, this will be pushed out to everyone.

Terrific.

Would you mind sharing the exact PR / commits for curious minds who want to understand what was going on? 😜

@sba923
Copy link

sba923 commented Nov 11, 2023

Hey all, we've almost completed our internal testing.

We will publish the update here at GitHub (for people actively seeking the update) and to Windows Insider Beta and Windows Insider Release Preview. After a few days of feedback, this will be pushed out to everyone.

@denelon Would you mind sharing the exact PR / commits for curious minds who want to understand what was going on? 😜

@MagicAndre1981
Copy link

Would you mind sharing the exact PR / commits for curious minds who want to understand what was going on? 😜

296a53d

@drankinatty
Copy link

drankinatty commented Nov 11, 2023

Thank you for the link to the note on Performance Improvements related to caching. However, the note does not explain all of the network communications to akami that are associated with the package manager issue. I looked again last night in Resource Explorer while this issue was taking place, and while low in bandwidth, there was continual transmission to/from the microsoft akami servers.

Windows already knows what updates have been downloaded and what updates are needed -- all updates have are completed before this package manager behavior kicks in. So what is this stream of network traffic with akami that your caching is indented to benefit?

Adding that information to the 296a53d link would be very helpful.

@denelon
Copy link
Contributor

denelon commented Nov 13, 2023

We will be working on getting a 1.7-preview version with the improved performance changes out next.

@denelon
Copy link
Contributor

denelon commented Nov 14, 2023

@denelon denelon added this to the 1.8 Client milestone Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Performance Issue related to CPU or memory performance Issue-Bug It either shouldn't be doing this or needs an investigation.
Projects
None yet
Development

No branches or pull requests