Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Defender/Anti-malware Causing Performance Issues after CU update #1932

Open
rdodev opened this issue Apr 16, 2017 · 120 comments
Open

Windows Defender/Anti-malware Causing Performance Issues after CU update #1932

rdodev opened this issue Apr 16, 2017 · 120 comments

Comments

@rdodev
Copy link

@rdodev rdodev commented Apr 16, 2017

So, after uninstalling my old Ubuntu Bash lxrun /uninstall /full and after the update lxrun /install I noticed, as promised, that processes running in WSL could be seen in the Windows task manager (yay!); however, there is an issue that consistently causes Windows Defender/Anti-Malware service to consume over 50% of CPU utilization whenever code compilation or builds are happening inside bash. Back last version (before the processes were visible to Windows -- or more sandboxed, not sure) this was not an issue at all. So I'm wondering if this is a worthy trade off? I, for one, as a dev, would prefer to maximize build/compile performance even if that means reverting to a more sandboxed environ (or at least the option to do so).

@aseering
Copy link
Contributor

@aseering aseering commented Apr 16, 2017

For what it's worth, I always add an exclusion for my big source-code build directories to Windows Defender, for exactly this reason. I trust my own code, and the performance hit is too big otherwise.

Loading

@kumarharsh
Copy link

@kumarharsh kumarharsh commented Apr 17, 2017

Adding to @aseering's comment, this is a problem even with yarn: yarnpkg/yarn#990

Loading

@kayakyakr
Copy link

@kayakyakr kayakyakr commented Apr 30, 2017

Just tried it out @aseering and noticed a pretty big speedup of a rails app load time. Nice!

I added an exclusion for both my networked drive and my lxss folder.

Loading

@tanseydavid
Copy link

@tanseydavid tanseydavid commented Aug 24, 2017

Adding an exclusion does NOT fix the performance problem. Adding an exclusion for VS does NOT fix the performance problem. Adding an exclusion for IIS does NOT fix the performance problem.

The only solution that works for us is to disable REALTIME protection in Windows Defender.

I also want to note that the performance impact is DRAMATIC.

For instance, a rebuild of a VS2017 solution that takes ~20seconds to complete with REALTIME protection turned off, takes more than 3 minutes to complete with Windows Defender REALTIME protection enabled.

Another example: a GIT local check-in with one file changed, completes almost instantly with Windows Defender REALTIME protection DISABLED. This same checkin takes more than 2 minutes with Windows Defender REALTIME protextion enabled.

The exclusions DO NOTHING to improve the situation.

Loading

@rdodev
Copy link
Author

@rdodev rdodev commented Aug 24, 2017

@tanseydavid yes exclusions do not seem to have helped at all for the linux side. :(

Loading

@kumarharsh
Copy link

@kumarharsh kumarharsh commented Aug 25, 2017

Yeah, it feels like adding exclusions is just a placebo - the performance is still incredibly slow for a lot of stuff related to development. But how do we even raise an issue with the windows defender team about this?

Loading

@sunilmut
Copy link
Member

@sunilmut sunilmut commented Aug 26, 2017

@tanseydavid - The issue that you have described, is that under WSL? Or, just in Windows general outside WSL?

@rdodev & @kumarharsh - In WSL, are you guys seeing any differences when defender realtime protection is turned off for the build directory? If you have any data to quantify, that will be useful.

I would like to get some clarification here before reaching out to the defender team internally.

Loading

@sunilmut
Copy link
Member

@sunilmut sunilmut commented Aug 28, 2017

Also, can you share your Windows build numbers, using the ver command from CMD?

Loading

@kumarharsh
Copy link

@kumarharsh kumarharsh commented Aug 30, 2017

10.0.15063

I stopped using WSL due to performance issues, and don't usually run it now... but even with normal powershell, the performance of, for example yarn install is not anywhere near linux when defender is running. You can see some benchmarks in the yarn issue listed above. I realise I'm not adding much to the WSL part of the discussion, but just wanted to add some details for the defender team's benefit.

Loading

@rdodev
Copy link
Author

@rdodev rdodev commented Aug 30, 2017

@sunilmut this should be very easy for you folks to replicate and grab as many benchmarks as you want.

  1. Install WSL
  2. Install git/gnumake, etc.
  3. Clone kubernetes/kubernetes (for example, or any large repo)
  4. See your CPUs pegged at 100%
  5. Profit

Loading

@fedu
Copy link

@fedu fedu commented Sep 12, 2017

Aww, just ran into this, while I was trying the Bash for Windows 10 for the first time. It actually freezes the whole Webpack process and it will never finnish, no matter how long you wait.

Guess the Bash is still unusable for web developers.

Loading

@ow
Copy link

@ow ow commented Dec 18, 2017

Blaaaah this is killing me as projects get bigger. Has nobody at Microsoft looked at all? I've seen a ton of closed threads but no action. I have to disable Defender all the time to get Git to finish in reasonable amounts of time (2s for a checkout with it disabled vs over a minute). I've tried the workaround - it does nothing.

Loading

@justinmchase
Copy link

@justinmchase justinmchase commented Dec 18, 2017

Same here. I also encounter a lot of disk errors while doing large npm installs. I can see windows defender killing my CPU even though I have a total exclusion on my E drive. It seems like npm may be caching things on an un-excluded directory before unpacking which is triggering windows defender and causing disk issues.

Loading

@tara-raj
Copy link

@tara-raj tara-raj commented Dec 18, 2017

The team is looking into this and will continue investigating. FWIW this rolls up into a broader filesystem perf improvement endeavor.

In the meantime if you can provide us with more specifics so we can generate a repro that would be very helpful. @ow which repo are you using? @justinmchase which errors do you hit?

Loading

@Sparkx120
Copy link

@Sparkx120 Sparkx120 commented Dec 18, 2017

@tara-raj thanks for the update. Just to toss my own experience out there, I am running 16299 Enterprise and my Windows Defender is locked on due to company policy. As soon as I start WSL I see constant CPU usage from the Defender service hovering around 25%. This goes away the moment I shut down WSL. Inside I am just running tmux and bash, running ssh does not seem to affect it much. Any filesystem actions can take a while especially on /mnt/c. The high CPU usage reduces my computers overall performance and especially the battery life. Adding an exception on the Linux rootfs folder helped, it was 50%+ without it, but it does not solve the issue entirely.

Thanks for looking into this.

Loading

@justinmchase
Copy link

@justinmchase justinmchase commented Dec 20, 2017

Here is a pretty simple repro for the bug I keep seeing:

  1. In WSL install node v8.7.0
  2. Unzip attached file to /mnt/c/bug , cd into dir
  3. npm install

bug.zip

This could be a node/npm bug... I'm not sure yet but I use the same directory linking feature on osx and it doesn't have this problem. The problem appears to be a combination of being in a linked directory and having the same dependency in both modules. Also i am doing everything under /mnt/e, which I have an exclusion for and ends up being reasonably fast but if I were to copy these files to mv bug ~/bug it will trigger windows defender scans which massively slow it down. Also global npm installs or installing a module for the first time which I think is globally caching off of /mnt/e. If I knew where the unmounted files for wsl were located I'd add that to windows defender exclusion. Might be nice if that was done during wsl install by you guys too.

Error code:

22:04:52:justin:/mnt/e/code/bug$ npm install
npm WARN bug@1.0.0 No description
npm WARN bug@1.0.0 No repository field.

npm ERR! path /mnt/e/code/bug/test/node_modules/mongoose/node_modules/async
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/async' -> '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/.async.DELETE'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/justin/.npm/_logs/2017-12-20T04_04_57_053Z-debug.log

Loading

@robertquitt
Copy link

@robertquitt robertquitt commented Dec 21, 2017

I'm having heavy performance issues related to Windows Defender and WSL as well. For me, it's when I'm switching tmux panes and opening files in vim. Disabling realtime protection in the Windows Defender Security Center causes a massive (>100x) speedup in performance for these tasks.

I'd really appreciate it if anyone had a workaround besides just disabling realtime protection.

Microsoft Windows [Version 10.0.16299.125] 
Linux goose 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

Loading

@kumarharsh
Copy link

@kumarharsh kumarharsh commented Dec 21, 2017

FWIW this rolls up into a broader filesystem perf improvement endeavor.

@tara-raj Would this endeavor be limited to WSL, or would also transcend to Windows' NTFS filesystem too?

Loading

@ow
Copy link

@ow ow commented Dec 21, 2017

@tara-raj I can get the poor performance to occur on almost any repo.

Do a git clone on something like this - the performance will be slow in general, but can push it further by committing new changes or just installing with npm - in all seriousness I'm turning Defender off 6-8 times a day to get the performance bottleneck out of the way. I've not timed it, but I'd hazard a guess it's more than 4x as bad with Defender enabled.

There's a ton of documentation about this across repositories — Yarn has a bunch of ongoing threads about this and has been writing workarounds to handle it. Generally speaking, WSL works great, but I will say these performance issues have been crippling and frustrating.

Loading

@kayakyakr
Copy link

@kayakyakr kayakyakr commented Dec 21, 2017

I'm working on migrating from the old version of bash to the windows store version of bash and cannot find the new location of the lxss folder. Anyone know where volfs is being stored these days?

Loading

@ow
Copy link

@ow ow commented Dec 28, 2017

For those here having the same issue, I've just completely given up on Defender pinning my CPU all the time. Would prefer to have it on, but currently nigh unusable on my setup.

If you want to force Realtime protection off, you can use this registry key, since Microsoft insists that it will turn itself back on within a few hours on Fall Creator's and up.

Paste below into a file, add .reg extension, double click. Reboot and boom, hey, it's actually usable.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001

Loading

@rdodev
Copy link
Author

@rdodev rdodev commented Dec 28, 2017

The sad part is that this issue was reported over 8 months ago and we've yet to see any progress or patch to remedy the issue. Instead of having to disable Windows Defender, they should revert to the way WSL worked before and let it be its own isolated subsystem. Integration with Windows is what caused all these perf problems.

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Dec 28, 2017

Please bear with us. We are investigating. If it was a simple issue to fix, it'd have been fixed by now 😉

Loading

@marcfor
Copy link

@marcfor marcfor commented Dec 29, 2017

@ow Weird, I tried cloning crafty-vagrant, both within the WSL file system and in /mnt/c and neither seemed to take super long, nor did my CPU go above 15%. I only heard about WSL a week ago and installed it very recently. Have you been using it for long? Maybe you've got an older version installed? Or some cruft left over from an old version?

Loading

@ow
Copy link

@ow ow commented Dec 29, 2017

@marcfor I have been using WSL since the day it launched, and have written much about my experience with it so far. Initial repositories appear fine, but once you begin to get complex directories in them -- run npm install, composer install, vagrant up on Crafty-vagrant -- then start trying to commit on a regular basis and it's agony. I can reproduce the behavior across three machines, one is just a few weeks old.

Loading

@bohrshaw
Copy link

@bohrshaw bohrshaw commented Dec 30, 2017

I once ran git status in a big repository under /mnt/c from WSL, it's unresponsive for a minute.
While the same operation using the Windows native git is completed within one second.
I suppose this is also related to Windows Defender.

Loading

@2ps
Copy link

@2ps 2ps commented Jan 2, 2018

As an FYI. We use the WSL Switcher to use centos under WSL. Adding exclusions helped improve git performance by 50%.

$lxss = $env:LOCALAPPDATA
Add-MpPreference -ExclusionPath "${lxss}\lxss"

Loading

@2ps
Copy link

@2ps 2ps commented Jan 2, 2018

Adding: @bitcrazed , please feel free to reach out to me. Our team uses WSL extensively, and we're happy to get you use cases / situations to reproduce.

Loading

@naefl
Copy link

@naefl naefl commented Feb 3, 2018

Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder. Most noticeable is when running yarn start. On MacOs this takes roughly 2-10s, on Ubuntu 16.04 around 2s and on WSL I get between 40-60s which is unbearable. rebuilding takes 200-800ms on Ubuntu, 1-2s on MacOs and 9-20s on WSL. No effect on this by changing Defender/Firewall settings

Loading

@ithiru
Copy link

@ithiru ithiru commented Jul 24, 2019

I know this is not perfect, but SSD helps a lot if you can't turn off third party scanners.

Loading

@tara-raj tara-raj removed their assignment Aug 22, 2019
@drichardson
Copy link

@drichardson drichardson commented Feb 18, 2020

@ithiru I'm using SSDs and the performance is still horrible. I see this regularly with Debian 10 using apt, like:

sudo apt-get update
sudo apt-get dist-upgrade

Always very slow, Task Manager shows Antimalware Service Executable busy. The same thing under Hyper-V is faster (and that one is backed by an old spinning disk).

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Feb 19, 2020

I do not know what you all are talking about with performance issues, I live daily in xfce4 on my w10 workstations - tinyurl.com/donofrioworkremmina2020 and I daily update and it is fine, I mean sure firefox is not working but arora does everything I need within wsl1 and ubuntu 18.04 granted I do mostly ssh/rdp (yes I rdp from ubuntu because I use remmina - it is cross platform) I but I have used many application/projects from aweather to ansible. my update as documented in my raw setup text file in that link:
"sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get dist-upgrade -y && sudo apt-get autoremove -y && sudo apt-get autoclean -y && sudo sync && sudo sync"

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Feb 19, 2020

@drichardson I am guessing your Debian distro is still running on WSL1.

When you run wsl.exe -l -v at your Windows command-line, what do you see?

C:\> wsl -l -v
  NAME                  STATE           VERSION
* Ubuntu                Running         2
  openSUSE-Leap-15-1    Stopped         1

As you can see from above, I am currently running Ubuntu on WSL2 and openSUSE on WSL1.

If your Debian is running as Version 1, then you're running Debian atop WSL1 whereupon you'll likely bump into the well known WSL1 file IO perf challenges that come from running Linux binaries atop Windows IO stack. If, on the other hand, you run a distro atop WSL2, your distro/binaries will be running in a container directly on a Linux kernel instance in lightweight VM, which doesn't use Windows' IO stack and isn't affected by anti-malware tools like Defender etc.

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Feb 20, 2020

I've always looked at a vm as "a way around the rules" in that I'm guessing like normal vm's that scom cannot scan the wsl2 vm? Unlike wsl1 that lives within ntfs and is scanned by local antivirus but also scan have it's file scanned from scom I'm guessing as well? let me know if I've got this wrong and I'll remove this reply....

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Feb 20, 2020

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Feb 20, 2020

The Defender team are currently assessing how/what/when/if Defender on the host interacts with workloads running in VMs and/or Defender running IN VMs. I do not yet have anything to share on what the final outcomes will be, but know that we're working closely with them on this matter.

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Feb 20, 2020

so I was correct that wsl2 vm's are loopback filesystems and not normalized (within the same partition/ntfs?) like in wsl1?

Loading

@therealkenc
Copy link
Collaborator

@therealkenc therealkenc commented Feb 20, 2020

so I was correct that wsl2 vm's are loopback filesystem

Close. WSL2 is Hyper-V VM, and on WSL2 Linux /dev/sda (which does not exist in WSL1) is backed by a Windows VHDX. Presumably (?) Defender leaves activity on the backing file alone; but similar to Rich's answer, I've never really investigated the extent to which Defender peeks reads and writes on that file. But whatever Defender does (if anything) as of this writing, it does it more quietly than the WSL1 hidden filesystem, and the performance is better.

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Feb 20, 2020

ok I guess I am just trying to see without just grabbing an insider box and trying it, is wsl2 a loopback filesystem sitting on a ntfs partition (akin to vmware workstation does?) if so this is a cheat unlike wsl1 you could scan with scom and whatever tools Microsoft provided now with wsl2 I'm guessing this is why we are now seeing ATP applications all of a sudden?

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Feb 20, 2020

In WSL2, each distro's filesystem is stored in individual EXT formatted VHDX files which are mounted within the distro's container. As such, the distros' filesystems aren't stored in NTFS like in WSL1.

It's nothing to do with cheating. It's all about compatibility and performance.

As I said, we're working with Defender and several other teams on a number of initiatives to further improve WSL2's integration, performance, interop, and manageability. Stay tuned to our blog for updates as we make progress.

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Feb 21, 2020

What I mean by 'cheating' is if your corp says you use windows and you setup a vm just to run your linux the way you want, unimpeded by domain policy's on the network like airwatch and other MDM clients. wsl1 allowed - perhaps mistakenly in my eye the ability to absorb linux into windows tools and domain policy's like scom and defender just worked (at least for me in last three years) but for some reason vm's seem to "get a pass" at least as far as I view it, I'm flexable if you can tell me I'm mistaken or incorrect.

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Feb 21, 2020

I totally hear you re. management of WSL instances. The fact that WSL1 ran atop NT and thus could be managed and monitored by existing Windows tooling and infrastructure is something many enterprises in particular have expressed considerable appreciation for.

But the reality is that for those using WSL, WSL1's compatibility and file IO performance just weren't where they needed to be, so we had to change tack and utilize our new lightweight container infrastructure to deliver high performance and 100% compat.

This does mean, however, that many of the management tools & tech that you'd use to manage, secure, configure, and operate physical Linux boxes and/or VM's are also likely to work just the same in WSL2!

And as I said above, there are teams actively looking into providing enhanced management, monitoring, and protection of Linux VMs (inc. WSL) and physical boxes, details of which will emerge over time.

If you'd like to enquire/discuss further, do reach out via email to crloewen and richturn at you know where dot com.

Loading

@zallarak
Copy link

@zallarak zallarak commented Mar 20, 2020

Really looking forward to the perf improvements. After running Linux native for years, I switched to windows due to WSL, and love it; I get the Windows OS experience with a Debian development environment -- not to mention the VSCode WSL plugin ❤️ . The only problem is performance, and I look forward to improvements. Congratulations on shipping such a wonderful piece of software 🎈

Loading

@fpopineau
Copy link

@fpopineau fpopineau commented Mar 20, 2020

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Mar 20, 2020

Why use MinGW64? is that for compile or for GUI like xfce4 accelerated? I only ask because I use "daily" full gui using VcXsrv (for last three years) my build guide and photo's at tinyurl.com/donofrioworkremmina2020 (yes I rdp from ubuntu w/xfce4 within remmina client)

Loading

@fpopineau
Copy link

@fpopineau fpopineau commented Mar 20, 2020

Loading

@lewisdonofrio
Copy link

@lewisdonofrio lewisdonofrio commented Mar 20, 2020

I compile using native gcc, and I use bash, all native, have done this for past three years and all is working great, so long as it doesn't want to touch non-existing hardware (aka usermode rocks for my needs)

Loading

@breadnone
Copy link

@breadnone breadnone commented Nov 5, 2020

Any update on this yet?

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Nov 5, 2020

@breadnone Are you still seeing Defender consume considerable amounts of CPU while running workloads in WSL?

If you are, I am guessing that you're running WSL1 instances?

FWIW, if you're running Windows 10 1903 or later, I encourage you to upgrade your WSL1 instances to WSL2 (instructions here). WSL2 runs your distros and binaries in containers atop a real Linux kernel in a lightweight VM, and runs at near native performance.

Loading

@breadnone
Copy link

@breadnone breadnone commented Nov 5, 2020

The performance kinda terrible.. I believe I'm still on WSL1, on my way upgrading. Thanks for the reply!

Loading

@bitcrazed
Copy link
Contributor

@bitcrazed bitcrazed commented Nov 5, 2020

@breadnone Once you've converted your distros to WSL2, be sure to move any substantial collections of files (e.g. dev projects, repos, etc.) into your distro(s') filesystem if that's where you'll access them most intensively. While the 9P fileserver that ferries files back and forth between Windows and Linux will work, it'll be slower than accessing files locally.

HTH.

Loading

@breadnone
Copy link

@breadnone breadnone commented Nov 5, 2020

Thanks for the heads up @bitcrazed

Loading

@Dragonphy
Copy link

@Dragonphy Dragonphy commented Apr 26, 2021

Updating to WSL2 doesn't help. It's a good solution to close REALTIME protection. For those who use IDEA or WebStorm's WSL, please use WSL1. After comparison, WSL2 takes up a lot of resources.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet