Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Defender/Anti-malware Causing Performance Issues after CU update #1932

Open
rdodev opened this Issue Apr 16, 2017 · 80 comments

Comments

Projects
None yet
@rdodev
Copy link

rdodev commented Apr 16, 2017

So, after uninstalling my old Ubuntu Bash lxrun /uninstall /full and after the update lxrun /install I noticed, as promised, that processes running in WSL could be seen in the Windows task manager (yay!); however, there is an issue that consistently causes Windows Defender/Anti-Malware service to consume over 50% of CPU utilization whenever code compilation or builds are happening inside bash. Back last version (before the processes were visible to Windows -- or more sandboxed, not sure) this was not an issue at all. So I'm wondering if this is a worthy trade off? I, for one, as a dev, would prefer to maximize build/compile performance even if that means reverting to a more sandboxed environ (or at least the option to do so).

@aseering

This comment has been minimized.

Copy link
Contributor

aseering commented Apr 16, 2017

For what it's worth, I always add an exclusion for my big source-code build directories to Windows Defender, for exactly this reason. I trust my own code, and the performance hit is too big otherwise.

@kumarharsh

This comment has been minimized.

Copy link

kumarharsh commented Apr 17, 2017

Adding to @aseering's comment, this is a problem even with yarn: yarnpkg/yarn#990

@kayakyakr

This comment has been minimized.

Copy link

kayakyakr commented Apr 30, 2017

Just tried it out @aseering and noticed a pretty big speedup of a rails app load time. Nice!

I added an exclusion for both my networked drive and my lxss folder.

@tanseydavid

This comment has been minimized.

Copy link

tanseydavid commented Aug 24, 2017

Adding an exclusion does NOT fix the performance problem. Adding an exclusion for VS does NOT fix the performance problem. Adding an exclusion for IIS does NOT fix the performance problem.

The only solution that works for us is to disable REALTIME protection in Windows Defender.

I also want to note that the performance impact is DRAMATIC.

For instance, a rebuild of a VS2017 solution that takes ~20seconds to complete with REALTIME protection turned off, takes more than 3 minutes to complete with Windows Defender REALTIME protection enabled.

Another example: a GIT local check-in with one file changed, completes almost instantly with Windows Defender REALTIME protection DISABLED. This same checkin takes more than 2 minutes with Windows Defender REALTIME protextion enabled.

The exclusions DO NOTHING to improve the situation.

@rdodev

This comment has been minimized.

Copy link
Author

rdodev commented Aug 24, 2017

@tanseydavid yes exclusions do not seem to have helped at all for the linux side. :(

@kumarharsh

This comment has been minimized.

Copy link

kumarharsh commented Aug 25, 2017

Yeah, it feels like adding exclusions is just a placebo - the performance is still incredibly slow for a lot of stuff related to development. But how do we even raise an issue with the windows defender team about this?

@sunilmut

This comment has been minimized.

Copy link
Member

sunilmut commented Aug 26, 2017

@tanseydavid - The issue that you have described, is that under WSL? Or, just in Windows general outside WSL?

@rdodev & @kumarharsh - In WSL, are you guys seeing any differences when defender realtime protection is turned off for the build directory? If you have any data to quantify, that will be useful.

I would like to get some clarification here before reaching out to the defender team internally.

@sunilmut

This comment has been minimized.

Copy link
Member

sunilmut commented Aug 28, 2017

Also, can you share your Windows build numbers, using the ver command from CMD?

@kumarharsh

This comment has been minimized.

Copy link

kumarharsh commented Aug 30, 2017

10.0.15063

I stopped using WSL due to performance issues, and don't usually run it now... but even with normal powershell, the performance of, for example yarn install is not anywhere near linux when defender is running. You can see some benchmarks in the yarn issue listed above. I realise I'm not adding much to the WSL part of the discussion, but just wanted to add some details for the defender team's benefit.

@rdodev

This comment has been minimized.

Copy link
Author

rdodev commented Aug 30, 2017

@sunilmut this should be very easy for you folks to replicate and grab as many benchmarks as you want.

  1. Install WSL
  2. Install git/gnumake, etc.
  3. Clone kubernetes/kubernetes (for example, or any large repo)
  4. See your CPUs pegged at 100%
  5. Profit
@fedu

This comment has been minimized.

Copy link

fedu commented Sep 12, 2017

Aww, just ran into this, while I was trying the Bash for Windows 10 for the first time. It actually freezes the whole Webpack process and it will never finnish, no matter how long you wait.

Guess the Bash is still unusable for web developers.

@ow

This comment has been minimized.

Copy link

ow commented Dec 18, 2017

Blaaaah this is killing me as projects get bigger. Has nobody at Microsoft looked at all? I've seen a ton of closed threads but no action. I have to disable Defender all the time to get Git to finish in reasonable amounts of time (2s for a checkout with it disabled vs over a minute). I've tried the workaround - it does nothing.

@justinmchase

This comment has been minimized.

Copy link

justinmchase commented Dec 18, 2017

Same here. I also encounter a lot of disk errors while doing large npm installs. I can see windows defender killing my CPU even though I have a total exclusion on my E drive. It seems like npm may be caching things on an un-excluded directory before unpacking which is triggering windows defender and causing disk issues.

@tara-raj

This comment has been minimized.

Copy link
Member

tara-raj commented Dec 18, 2017

The team is looking into this and will continue investigating. FWIW this rolls up into a broader filesystem perf improvement endeavor.

In the meantime if you can provide us with more specifics so we can generate a repro that would be very helpful. @ow which repo are you using? @justinmchase which errors do you hit?

@Sparkx120

This comment has been minimized.

Copy link

Sparkx120 commented Dec 18, 2017

@tara-raj thanks for the update. Just to toss my own experience out there, I am running 16299 Enterprise and my Windows Defender is locked on due to company policy. As soon as I start WSL I see constant CPU usage from the Defender service hovering around 25%. This goes away the moment I shut down WSL. Inside I am just running tmux and bash, running ssh does not seem to affect it much. Any filesystem actions can take a while especially on /mnt/c. The high CPU usage reduces my computers overall performance and especially the battery life. Adding an exception on the Linux rootfs folder helped, it was 50%+ without it, but it does not solve the issue entirely.

Thanks for looking into this.

@justinmchase

This comment has been minimized.

Copy link

justinmchase commented Dec 20, 2017

Here is a pretty simple repro for the bug I keep seeing:

  1. In WSL install node v8.7.0
  2. Unzip attached file to /mnt/c/bug , cd into dir
  3. npm install

bug.zip

This could be a node/npm bug... I'm not sure yet but I use the same directory linking feature on osx and it doesn't have this problem. The problem appears to be a combination of being in a linked directory and having the same dependency in both modules. Also i am doing everything under /mnt/e, which I have an exclusion for and ends up being reasonably fast but if I were to copy these files to mv bug ~/bug it will trigger windows defender scans which massively slow it down. Also global npm installs or installing a module for the first time which I think is globally caching off of /mnt/e. If I knew where the unmounted files for wsl were located I'd add that to windows defender exclusion. Might be nice if that was done during wsl install by you guys too.

Error code:

22:04:52:justin:/mnt/e/code/bug$ npm install
npm WARN bug@1.0.0 No description
npm WARN bug@1.0.0 No repository field.

npm ERR! path /mnt/e/code/bug/test/node_modules/mongoose/node_modules/async
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/async' -> '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/.async.DELETE'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/justin/.npm/_logs/2017-12-20T04_04_57_053Z-debug.log
@robertquitt

This comment has been minimized.

Copy link

robertquitt commented Dec 21, 2017

I'm having heavy performance issues related to Windows Defender and WSL as well. For me, it's when I'm switching tmux panes and opening files in vim. Disabling realtime protection in the Windows Defender Security Center causes a massive (>100x) speedup in performance for these tasks.

I'd really appreciate it if anyone had a workaround besides just disabling realtime protection.

Microsoft Windows [Version 10.0.16299.125] 
Linux goose 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux
@kumarharsh

This comment has been minimized.

Copy link

kumarharsh commented Dec 21, 2017

FWIW this rolls up into a broader filesystem perf improvement endeavor.

@tara-raj Would this endeavor be limited to WSL, or would also transcend to Windows' NTFS filesystem too?

@ow

This comment has been minimized.

Copy link

ow commented Dec 21, 2017

@tara-raj I can get the poor performance to occur on almost any repo.

Do a git clone on something like this - the performance will be slow in general, but can push it further by committing new changes or just installing with npm - in all seriousness I'm turning Defender off 6-8 times a day to get the performance bottleneck out of the way. I've not timed it, but I'd hazard a guess it's more than 4x as bad with Defender enabled.

There's a ton of documentation about this across repositories — Yarn has a bunch of ongoing threads about this and has been writing workarounds to handle it. Generally speaking, WSL works great, but I will say these performance issues have been crippling and frustrating.

@kayakyakr

This comment has been minimized.

Copy link

kayakyakr commented Dec 21, 2017

I'm working on migrating from the old version of bash to the windows store version of bash and cannot find the new location of the lxss folder. Anyone know where volfs is being stored these days?

@ow

This comment has been minimized.

Copy link

ow commented Dec 28, 2017

For those here having the same issue, I've just completely given up on Defender pinning my CPU all the time. Would prefer to have it on, but currently nigh unusable on my setup.

If you want to force Realtime protection off, you can use this registry key, since Microsoft insists that it will turn itself back on within a few hours on Fall Creator's and up.

Paste below into a file, add .reg extension, double click. Reboot and boom, hey, it's actually usable.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001
@rdodev

This comment has been minimized.

Copy link
Author

rdodev commented Dec 28, 2017

The sad part is that this issue was reported over 8 months ago and we've yet to see any progress or patch to remedy the issue. Instead of having to disable Windows Defender, they should revert to the way WSL worked before and let it be its own isolated subsystem. Integration with Windows is what caused all these perf problems.

@bitcrazed

This comment has been minimized.

Copy link
Collaborator

bitcrazed commented Dec 28, 2017

Please bear with us. We are investigating. If it was a simple issue to fix, it'd have been fixed by now 😉

@marcfor

This comment has been minimized.

Copy link

marcfor commented Dec 29, 2017

@ow Weird, I tried cloning crafty-vagrant, both within the WSL file system and in /mnt/c and neither seemed to take super long, nor did my CPU go above 15%. I only heard about WSL a week ago and installed it very recently. Have you been using it for long? Maybe you've got an older version installed? Or some cruft left over from an old version?

@ow

This comment has been minimized.

Copy link

ow commented Dec 29, 2017

@marcfor I have been using WSL since the day it launched, and have written much about my experience with it so far. Initial repositories appear fine, but once you begin to get complex directories in them -- run npm install, composer install, vagrant up on Crafty-vagrant -- then start trying to commit on a regular basis and it's agony. I can reproduce the behavior across three machines, one is just a few weeks old.

@bohrshaw

This comment has been minimized.

Copy link

bohrshaw commented Dec 30, 2017

I once ran git status in a big repository under /mnt/c from WSL, it's unresponsive for a minute.
While the same operation using the Windows native git is completed within one second.
I suppose this is also related to Windows Defender.

@2ps

This comment has been minimized.

Copy link

2ps commented Jan 2, 2018

As an FYI. We use the WSL Switcher to use centos under WSL. Adding exclusions helped improve git performance by 50%.

$lxss = $env:LOCALAPPDATA
Add-MpPreference -ExclusionPath "${lxss}\lxss"
@2ps

This comment has been minimized.

Copy link

2ps commented Jan 2, 2018

Adding: @bitcrazed , please feel free to reach out to me. Our team uses WSL extensively, and we're happy to get you use cases / situations to reproduce.

@naefl

This comment has been minimized.

Copy link

naefl commented Feb 3, 2018

Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder. Most noticeable is when running yarn start. On MacOs this takes roughly 2-10s, on Ubuntu 16.04 around 2s and on WSL I get between 40-60s which is unbearable. rebuilding takes 200-800ms on Ubuntu, 1-2s on MacOs and 9-20s on WSL. No effect on this by changing Defender/Firewall settings

@galvesribeiro

This comment has been minimized.

Copy link

galvesribeiro commented Jul 11, 2018

@bitcrazed

This comment has been minimized.

Copy link
Collaborator

bitcrazed commented Jul 11, 2018

The status is as it was when I last replied to this thread:

I appreciate this is frustrating, but the fact that it's not yet resolved is just a glimpse into how complex this issue is.

For those who're interested, #873 is the main issue tracking disk perf issues in general, and you'll find more commentary, and detailed feedback there, including from the engineer responsible for most of WSL's disk IO engine.

You can be sure that as soon as we have a decent solution ready for testing that this issue will be updated, and there will be much rejoicing on Twitter and elsewhere.

Until then, please bear with us while @tara-raj works across several teams to figure out and implement a decent solution.

Thanks.

@galvesribeiro

This comment has been minimized.

Copy link

galvesribeiro commented Jul 11, 2018

Thanks for the update @bitcrazed.

Looking forward for the final fix.

@rohityadavrv

This comment has been minimized.

Copy link

rohityadavrv commented Jul 24, 2018

Hi I am facing similar issue when i start any node.exe process even as simple as npm -v on my windows 10 laptop with pretty good configurations as i7 and 8 gb RAM it takes around 40 seconds to come up with version number. My previous laptop with much less configuration took 3 to 4 seconds, is it windows defender causing this issue , i have tried disabling it and adding exceptions in my symantac antivirus but still no progress. Any help would be appreciated

@WSLUser

This comment has been minimized.

Copy link

WSLUser commented Jul 25, 2018

@rohityadavrv If you're running Symantec, then Defender should be disabled by default. If you're running node.exe, then that is just typical Windows behavior and not pertinent to WSL (though Windows itself has suffered as a result of changes to Defender). If you run the Linux node binary however, then I would suggest creating an exception for the entire rootfs which you will find under AppData in your user profile under the distro in use and see if that helps. If not, then you need to file a bug with Symantec to implement WSL support. There's a MS blog post for them to do that and MS is willing to work with A/V vendors to help support WSL. That said, there are other underlying perf issues. Defender is just one of them (if you're only running it and not something else).

@jamesyale

This comment has been minimized.

Copy link

jamesyale commented Jul 25, 2018

In my experience, adding the WSL Linux filesystem root to the directory exceptions for Defender is ineffective, or at least, doesn't make a significant difference to the AV chewing the CPU inspecting running processes. The only method I've found effective is adding each WSL Linux process I use commonly to the process exclusions.

@WSLUser

This comment has been minimized.

Copy link

WSLUser commented Jul 25, 2018

Works better in Avast (which disables Defender), Symantec may or may not get the same results. Both options are valid. This situation is better in Insiders for Defender with Linux processes being treated the same as Windows processes (which makes your method the "right" way when the next build is released)

@ian-p-cooke

This comment has been minimized.

Copy link

ian-p-cooke commented Jul 25, 2018

Here's another workaround which works for me: for each directory in your linux path add a process exclusion rule with the equivalent windows path with a wildcard. So, instead of excluding just bash with a process exclusion rule of C:\Users\ipc\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\bin\bash use C:\Users\ipc\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\bin\*. I did that for /bin, /usr/bin, /sbin, /usr/sbin... basically every linux dir in my $PATH that had an executable in it. I also added a folder exclusion rule for the base dir of the installation because that prevents the executable itself from being scanned by real-time (see this page for details).

With that change I can leave real-time scanning on; my compile times in WSL are just as fast as they are when Defender's real-time scanning is disabled and Defender no longer spikes the CPU as much. Defender still takes up some CPU time but it didn't seem to affect my build times so I'm not too concerned.

Here's a powershell script that I was using to add my rules. Change the username and package name to your own usernames and package.

Does this work for anyone else's test case?

@rohityadavrv

This comment has been minimized.

Copy link

rohityadavrv commented Jul 25, 2018

thanks @DarthSpock and everyone. @DarthSpock , unfortunately i cant use WSL , but these slowness issues goes away in safe mode. Moreover for some of our guys the speed is ok for windows 10 and for some it slow even on Windows 7 and for some not. We need to find out if it is antivirus which is interfering with it or is it Forcepoint DLP.

@WSLUser

This comment has been minimized.

Copy link

WSLUser commented Jul 25, 2018

Forcepoint DLP

Haven't used that particular product but having experience with McAfee DLP, I'd say if it wasn't trusted or exempted, that could cause problems to performance if not outright blockage of some executions. (Thankfully I don't use WSL with DLP installed so don't have this particular factor to consider) Another factor may just be the combo of the Symantec being used with Forcepoint in terms of performance. Try doing a comparison of removing one, executing a large binary, then reinstalling the one you remove and remove the other product and run some benchmarks for both of those scenarios. Also, since this issue is for Defender, you may be better off on the Symantec and/or Forcepoint forum.

@piotrmocko

This comment has been minimized.

Copy link

piotrmocko commented Jul 26, 2018

@ian-p-cooke your solution to exclude processes in bin directories works for me on Ubuntu 16.04 too. Thanks!

@epage

This comment has been minimized.

Copy link

epage commented Jul 30, 2018

Last update (Windows 10, version 1803) was on 6/5/2018

I use WSL nearly daily. On 7/27/2018 I went from it being perfectly usable to 10s of seconds to perform cd ~/git/<TAB>, git status, etc. I found that MsMpEng, via resource Monitor, was at 25% CPU utilization and was performing a steady 1+MB/s read.

This was observed through this morning, 7/30/2018

EDIT: It looks like it might now be resolved (more testing needed). I checked Windows Defender and I did get a definition update at 6:09AM on 7/30/2018 which was after I experienced it this morning. I wonder if I just got a bad definition.

@spences10

This comment has been minimized.

Copy link

spences10 commented Aug 14, 2018

Thanks @ian-p-cooke your solution appears to have helped 👍

Although I have only added the paths explicitly mentioned, if it's still playing up I'll attempt to add everything from my path.

Is there a way to identify where you have executable files located in bash?

@anaxamaxan

This comment has been minimized.

Copy link

anaxamaxan commented Aug 16, 2018

Thanks @ian-p-cooke I think I can finally use git efficiently in WSL. Thanks!

@spences10 The executables can vary a bit, but 99% of the time they're in standard locations. Starting from the WSL root filesystem directory in Windows, which is something like C:\Users\YOURUSERNAME\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs the paths to the Linux executables will be be at:

  • bin\*
  • sbin\*
  • usr\bin\*
  • usr\sbin\*
  • usr\local\bin\*
  • usr\local\sbin\*

This is using Ubuntu - the list might vary in CentOS or other distro.

If you have special cases, you could add those too. For example Ian Cooke's powershell script also excludes home\$linuxuser\.cargo\* - I don't know what .cargo is for, presumably some build tool that he uses. You probably don't have that and can omit it.

@TaraHoleInIt

This comment has been minimized.

Copy link

TaraHoleInIt commented Aug 18, 2018

I've tried the script and changed the .cargo entry to point to a toolchain I installed in my home directory but no such luck, AntiMalware still pins the CPU.

I can see things like cc1 and cc1plus up there in the task manager every now and then, but even at that I manually added the libexec directories containing binaries to the exclusions but with no luck.

@mloskot

This comment has been minimized.

Copy link

mloskot commented Aug 18, 2018

I don't see CPU spinning a lot by the Defender process. I see I/O reads/writes overdose though

@nickcolley

This comment has been minimized.

Copy link

nickcolley commented Sep 5, 2018

@bitcrazed hey Rich, is there a good place to follow along for the status on this, are you having any luck getting this solved? :)

@bmayen

This comment has been minimized.

Copy link

bmayen commented Sep 5, 2018

@Lighfer

This comment has been minimized.

Copy link

Lighfer commented Oct 25, 2018

@ian-p-cooke works for me, thanks!

@mlhpdx

This comment has been minimized.

Copy link

mlhpdx commented Nov 30, 2018

For what it's worth, I have this issue outside WSL with Windows git as well. A simple git branch on the windows command line takes minutes. So I'm not sure this is a WSL problem, but it's a critical issue impacting my entire team right now.

@justinmchase

This comment has been minimized.

Copy link

justinmchase commented Nov 30, 2018

@mlhpdx Did you add an exclusion to Defender for the folder that your source code lives in?

@dss-github

This comment has been minimized.

Copy link

dss-github commented Jan 6, 2019

You guys literally saved my life! My WSL is performing like hell after running the script! It's a world of a difference! Welcome back productivity! :-)

@colinmollenhour

This comment has been minimized.

Copy link

colinmollenhour commented Jan 23, 2019

None of these suggestions worked for me unfortunately. I'm using "git status" to test and the same exact project which takes 0.4 seconds on my Ubuntu running within VirtualBox takes over 4.4 seconds on WSL. I've not disabled real time protection because that seems like a major security threat. So WSL is just about useless for real work, unfortunately.

@ndrewxie

This comment has been minimized.

Copy link

ndrewxie commented Jan 23, 2019

Same. None of the suggestions work for me, too - on my primary PC, which I don't have admin rights to (therefore cannot use these workarounds), these workarounds are hopeless. On my secondary PC, I've tried it, and got similar results to @colinmollenhour . If virtualbox can do it, then it's certainly possible (yes, virtualbox requires a driver - but the performance should still be in the same ballpark. Please don't do anything to WSL that makes it require admin rights on startup/for simple tasks, the guy who as admin rights to my computer is very rarely around).

hoW aBoUt opTimIze WiNdoWs defeNDEr, oR mAkE It noT SCAn EveRy sIngLe file BY DEFAULT (IT hAS tO NOT DO ThiS BY DEFAULT becauSe THeRE ARE sTANdArD uSERS OuT tHEre, AND somE, LiKE mE, WiLl gO fOr 6-7 MoNtHS At a TiMe wIThoUt COntAct WiTH tHe PerSon wHO hAs tHE aDmIn accOuNt)?
Windows defender's probably pretty optimized already tho.

@ayalon

This comment has been minimized.

Copy link

ayalon commented Feb 19, 2019

@colinmollenhour Did you try the ruby script from here:
https://gist.github.com/dayne/313981bc3ee6dbf8ee57eb3d58aa1dc0#file-1-wsl-defender-fix-md

I had to delete all the Windows Path variables from the output. But after running the script in an Adminstrator Powershell, "git status" was about 10x times faster than before.

@colinmollenhour

This comment has been minimized.

Copy link

colinmollenhour commented Feb 19, 2019

I didn't use the Ruby script but rather hand-edited the Powershell script. Afterwards a large number of new entries were seen in the excludes list. It didn't help me.

@lewisdonofrio

This comment has been minimized.

Copy link

lewisdonofrio commented Feb 21, 2019

as @bitcrazed stated
"Dear WSL fans, despite the perf impact of Defender (which we are working to address), THIS is why we DO NOT recommend excluding folders from or disabling Defender"
defender-wsl-good
(https://twitter.com/richturn_ms/status/1095207032292929536)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.