Windows Defender/Anti-malware Causing Performance Issues after CU update

[rdodev]

So, after uninstalling my old Ubuntu Bash lxrun /uninstall /full and after the update lxrun /install I noticed, as promised, that processes running in WSL could be seen in the Windows task manager (yay!); however, there is an issue that consistently causes Windows Defender/Anti-Malware service to consume over 50% of CPU utilization whenever code compilation or builds are happening inside bash. Back last version (before the processes were visible to Windows -- or more sandboxed, not sure) this was not an issue at all. So I'm wondering if this is a worthy trade off? I, for one, as a dev, would prefer to maximize build/compile performance even if that means reverting to a more sandboxed environ (or at least the option to do so).

[aseering]

For what it's worth, I always add an exclusion for my big source-code build directories to Windows Defender, for exactly this reason. I trust my own code, and the performance hit is too big otherwise.

[kumarharsh]

Adding to @aseering's comment, this is a problem even with yarn: yarnpkg/yarn#990

[kayakyakr]

Just tried it out @aseering and noticed a pretty big speedup of a rails app load time. Nice!

I added an exclusion for both my networked drive and my lxss folder.

[tanseydavid]

Adding an exclusion does NOT fix the performance problem. Adding an exclusion for VS does NOT fix the performance problem. Adding an exclusion for IIS does NOT fix the performance problem.

The only solution that works for us is to disable REALTIME protection in Windows Defender.

I also want to note that the performance impact is DRAMATIC.

For instance, a rebuild of a VS2017 solution that takes ~20seconds to complete with REALTIME protection turned off, takes more than 3 minutes to complete with Windows Defender REALTIME protection enabled.

Another example: a GIT local check-in with one file changed, completes almost instantly with Windows Defender REALTIME protection DISABLED. This same checkin takes more than 2 minutes with Windows Defender REALTIME protextion enabled.

The exclusions DO NOTHING to improve the situation.

[rdodev]

@tanseydavid yes exclusions do not seem to have helped at all for the linux side. :(

[kumarharsh]

Yeah, it feels like adding exclusions is just a placebo - the performance is still incredibly slow for a lot of stuff related to development. But how do we even raise an issue with the windows defender team about this?

[sunilmut]

@tanseydavid - The issue that you have described, is that under WSL? Or, just in Windows general outside WSL?

@rdodev & @kumarharsh - In WSL, are you guys seeing any differences when defender realtime protection is turned off for the build directory? If you have any data to quantify, that will be useful.

I would like to get some clarification here before reaching out to the defender team internally.

[sunilmut]

Also, can you share your Windows build numbers, using the ver command from CMD?

[kumarharsh]

10.0.15063

I stopped using WSL due to performance issues, and don't usually run it now... but even with normal powershell, the performance of, for example yarn install is not anywhere near linux when defender is running. You can see some benchmarks in the yarn issue listed above. I realise I'm not adding much to the WSL part of the discussion, but just wanted to add some details for the defender team's benefit.

[rdodev]

@sunilmut this should be very easy for you folks to replicate and grab as many benchmarks as you want.

1. Install WSL
2. Install git/gnumake, etc.
3. Clone kubernetes/kubernetes (for example, or any large repo)
4. See your CPUs pegged at 100%
5. Profit

[fedu]

Aww, just ran into this, while I was trying the Bash for Windows 10 for the first time. It actually freezes the whole Webpack process and it will never finnish, no matter how long you wait.

Guess the Bash is still unusable for web developers.

[ow]

Blaaaah this is killing me as projects get bigger. Has nobody at Microsoft looked at all? I've seen a ton of closed threads but no action. I have to disable Defender all the time to get Git to finish in reasonable amounts of time (2s for a checkout with it disabled vs over a minute). I've tried the workaround - it does nothing.

[justinmchase]

Same here. I also encounter a lot of disk errors while doing large npm installs. I can see windows defender killing my CPU even though I have a total exclusion on my E drive. It seems like npm may be caching things on an un-excluded directory before unpacking which is triggering windows defender and causing disk issues.

[tara-raj]

The team is looking into this and will continue investigating. FWIW this rolls up into a broader filesystem perf improvement endeavor.

In the meantime if you can provide us with more specifics so we can generate a repro that would be very helpful. @ow which repo are you using? @justinmchase which errors do you hit?

[Sparkx120]

@tara-raj thanks for the update. Just to toss my own experience out there, I am running 16299 Enterprise and my Windows Defender is locked on due to company policy. As soon as I start WSL I see constant CPU usage from the Defender service hovering around 25%. This goes away the moment I shut down WSL. Inside I am just running tmux and bash, running ssh does not seem to affect it much. Any filesystem actions can take a while especially on /mnt/c. The high CPU usage reduces my computers overall performance and especially the battery life. Adding an exception on the Linux rootfs folder helped, it was 50%+ without it, but it does not solve the issue entirely.

Thanks for looking into this.

[justinmchase]

Here is a pretty simple repro for the bug I keep seeing:

1. In WSL install node v8.7.0
2. Unzip attached file to /mnt/c/bug , cd into dir
3. npm install

bug.zip

This could be a node/npm bug... I'm not sure yet but I use the same directory linking feature on osx and it doesn't have this problem. The problem appears to be a combination of being in a linked directory and having the same dependency in both modules. Also i am doing everything under /mnt/e, which I have an exclusion for and ends up being reasonably fast but if I were to copy these files to mv bug ~/bug it will trigger windows defender scans which massively slow it down. Also global npm installs or installing a module for the first time which I think is globally caching off of /mnt/e. If I knew where the unmounted files for wsl were located I'd add that to windows defender exclusion. Might be nice if that was done during wsl install by you guys too.

Error code:

22:04:52:justin:/mnt/e/code/bug$ npm install
npm WARN bug@1.0.0 No description
npm WARN bug@1.0.0 No repository field.

npm ERR! path /mnt/e/code/bug/test/node_modules/mongoose/node_modules/async
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/async' -> '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/.async.DELETE'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/justin/.npm/_logs/2017-12-20T04_04_57_053Z-debug.log

[robertquitt]

I'm having heavy performance issues related to Windows Defender and WSL as well. For me, it's when I'm switching tmux panes and opening files in vim. Disabling realtime protection in the Windows Defender Security Center causes a massive (>100x) speedup in performance for these tasks.

I'd really appreciate it if anyone had a workaround besides just disabling realtime protection.

Microsoft Windows [Version 10.0.16299.125] 
Linux goose 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

[kumarharsh]

FWIW this rolls up into a broader filesystem perf improvement endeavor.

@tara-raj Would this endeavor be limited to WSL, or would also transcend to Windows' NTFS filesystem too?

[ow]

@tara-raj I can get the poor performance to occur on almost any repo.

Do a git clone on something like this - the performance will be slow in general, but can push it further by committing new changes or just installing with npm - in all seriousness I'm turning Defender off 6-8 times a day to get the performance bottleneck out of the way. I've not timed it, but I'd hazard a guess it's more than 4x as bad with Defender enabled.

There's a ton of documentation about this across repositories — Yarn has a bunch of ongoing threads about this and has been writing workarounds to handle it. Generally speaking, WSL works great, but I will say these performance issues have been crippling and frustrating.

[kayakyakr]

I'm working on migrating from the old version of bash to the windows store version of bash and cannot find the new location of the lxss folder. Anyone know where volfs is being stored these days?

[ow]

For those here having the same issue, I've just completely given up on Defender pinning my CPU all the time. Would prefer to have it on, but currently nigh unusable on my setup.

If you want to force Realtime protection off, you can use this registry key, since Microsoft insists that it will turn itself back on within a few hours on Fall Creator's and up.

Paste below into a file, add .reg extension, double click. Reboot and boom, hey, it's actually usable.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001

[rdodev]

The sad part is that this issue was reported over 8 months ago and we've yet to see any progress or patch to remedy the issue. Instead of having to disable Windows Defender, they should revert to the way WSL worked before and let it be its own isolated subsystem. Integration with Windows is what caused all these perf problems.

[bitcrazed]

Please bear with us. We are investigating. If it was a simple issue to fix, it'd have been fixed by now 😉

[marcfor]

@ow Weird, I tried cloning crafty-vagrant, both within the WSL file system and in /mnt/c and neither seemed to take super long, nor did my CPU go above 15%. I only heard about WSL a week ago and installed it very recently. Have you been using it for long? Maybe you've got an older version installed? Or some cruft left over from an old version?

[ow]

@marcfor I have been using WSL since the day it launched, and have written much about my experience with it so far. Initial repositories appear fine, but once you begin to get complex directories in them -- run npm install, composer install, vagrant up on Crafty-vagrant -- then start trying to commit on a regular basis and it's agony. I can reproduce the behavior across three machines, one is just a few weeks old.

[bohrshaw]

I once ran git status in a big repository under /mnt/c from WSL, it's unresponsive for a minute.
While the same operation using the Windows native git is completed within one second.
I suppose this is also related to Windows Defender.

[2ps]

As an FYI. We use the WSL Switcher to use centos under WSL. Adding exclusions helped improve git performance by 50%.

$lxss = $env:LOCALAPPDATA
Add-MpPreference -ExclusionPath "${lxss}\lxss"

[2ps]

Adding: @bitcrazed , please feel free to reach out to me. Our team uses WSL extensively, and we're happy to get you use cases / situations to reproduce.

[naefl]

Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder. Most noticeable is when running yarn start. On MacOs this takes roughly 2-10s, on Ubuntu 16.04 around 2s and on WSL I get between 40-60s which is unbearable. rebuilding takes 200-800ms on Ubuntu, 1-2s on MacOs and 9-20s on WSL. No effect on this by changing Defender/Firewall settings

[lewisdonofrio]

so I was correct that wsl2 vm's are loopback filesystems and not normalized (within the same partition/ntfs?) like in wsl1?

[therealkenc]

so I was correct that wsl2 vm's are loopback filesystem

Close. WSL2 is Hyper-V VM, and on WSL2 Linux /dev/sda (which does not exist in WSL1) is backed by a Windows VHDX. Presumably (?) Defender leaves activity on the backing file alone; but similar to Rich's answer, I've never really investigated the extent to which Defender peeks reads and writes on that file. But whatever Defender does (if anything) as of this writing, it does it more quietly than the WSL1 hidden filesystem, and the performance is better.

[lewisdonofrio]

ok I guess I am just trying to see without just grabbing an insider box and trying it, is wsl2 a loopback filesystem sitting on a ntfs partition (akin to vmware workstation does?) if so this is a cheat unlike wsl1 you could scan with scom and whatever tools Microsoft provided now with wsl2 I'm guessing this is why we are now seeing ATP applications all of a sudden?

[bitcrazed]

In WSL2, each distro's filesystem is stored in individual EXT formatted VHDX files which are mounted within the distro's container. As such, the distros' filesystems aren't stored in NTFS like in WSL1.

It's nothing to do with cheating. It's all about compatibility and performance.

As I said, we're working with Defender and several other teams on a number of initiatives to further improve WSL2's integration, performance, interop, and manageability. Stay tuned to our blog for updates as we make progress.

[lewisdonofrio]

What I mean by 'cheating' is if your corp says you use windows and you setup a vm just to run your linux the way you want, unimpeded by domain policy's on the network like airwatch and other MDM clients. wsl1 allowed - perhaps mistakenly in my eye the ability to absorb linux into windows tools and domain policy's like scom and defender just worked (at least for me in last three years) but for some reason vm's seem to "get a pass" at least as far as I view it, I'm flexable if you can tell me I'm mistaken or incorrect.

[bitcrazed]

I totally hear you re. management of WSL instances. The fact that WSL1 ran atop NT and thus could be managed and monitored by existing Windows tooling and infrastructure is something many enterprises in particular have expressed considerable appreciation for.

But the reality is that for those using WSL, WSL1's compatibility and file IO performance just weren't where they needed to be, so we had to change tack and utilize our new lightweight container infrastructure to deliver high performance and 100% compat.

This does mean, however, that many of the management tools & tech that you'd use to manage, secure, configure, and operate physical Linux boxes and/or VM's are also likely to work just the same in WSL2!

And as I said above, there are teams actively looking into providing enhanced management, monitoring, and protection of Linux VMs (inc. WSL) and physical boxes, details of which will emerge over time.

If you'd like to enquire/discuss further, do reach out via email to crloewen and richturn at you know where dot com.

[zallarak]

Really looking forward to the perf improvements. After running Linux native for years, I switched to windows due to WSL, and love it; I get the Windows OS experience with a Debian development environment -- not to mention the VSCode WSL plugin ❤️ . The only problem is performance, and I look forward to improvements. Congratulations on shipping such a wonderful piece of software 🎈

[fpopineau]

Given the move from WSL1 to WSL2 and the fact that it is not possible to share a $HOME directory between Windows and Linux subsystem with decent performance, I switched back to *not* using WSL. I rely on Anaconda/MinGW64 and I have GPU access.

[lewisdonofrio]

Why use MinGW64? is that for compile or for GUI like xfce4 accelerated? I only ask because I use "daily" full gui using VcXsrv (for last three years) my build guide and photo's at tinyurl.com/donofrioworkremmina2020 (yes I rdp from ubuntu w/xfce4 within remmina client)

[fpopineau]

Le ven. 20 mars 2020 à 17:34, Lewis L. Donofrio <notifications@github.com> a écrit :

Why use MinGW64?

MinGW64 provides bash, gcc etc. to build native Win64 app from Gnu/Linux sources.

[lewisdonofrio]

I compile using native gcc, and I use bash, all native, have done this for past three years and all is working great, so long as it doesn't want to touch non-existing hardware (aka usermode rocks for my needs)

[breadnone]

Any update on this yet?

[bitcrazed]

@breadnone Are you still seeing Defender consume considerable amounts of CPU while running workloads in WSL?

If you are, I am guessing that you're running WSL1 instances?

FWIW, if you're running Windows 10 1903 or later, I encourage you to upgrade your WSL1 instances to WSL2 (instructions here). WSL2 runs your distros and binaries in containers atop a real Linux kernel in a lightweight VM, and runs at near native performance.

[breadnone]

The performance kinda terrible.. I believe I'm still on WSL1, on my way upgrading. Thanks for the reply!

[bitcrazed]

@breadnone Once you've converted your distros to WSL2, be sure to move any substantial collections of files (e.g. dev projects, repos, etc.) into your distro(s') filesystem if that's where you'll access them most intensively. While the 9P fileserver that ferries files back and forth between Windows and Linux will work, it'll be slower than accessing files locally.

HTH.

[breadnone]

Thanks for the heads up @bitcrazed

[Loongphy]

Updating to WSL2 doesn't help. It's a good solution to close REALTIME protection. For those who use IDEA or WebStorm's WSL, please use WSL1. After comparison, WSL2 takes up a lot of resources.

[waterloomatt]

❤️ Been using WSL2 for 2 years now and loving it! A big Thank You to the whole team for such a dramatic improvement in speed and stability. Makes developing on Windows and Ubuntu a joy.

[lewisdonofrio]

How do you deal with vmmem issues and what do you do to bring up the gui on one monitor not all four of them?

[bgolinvaux]

I would like to add a comment. Contrary to what Loongphy has seen above, my experience with WSL2 is very positive : I launched a test script that accesses ~3000 files (and performs a lot of computation too) and the impact of turning Real Time protection off was less than 0.5%

The very important thing here is that you should never use the Windows filesystem to work under Linux (that is, do not share workspaces through /mnt/c/... but work on the Linux native filesystem). Only use it for transient file exchange (or use cases where speed does not matter)

A direct comparison between WSL1 and WSL2 when using a workflow that involves /mnt/c/... will be terrible for WSL2, because of the completely different architecture.

[muhammadmoazzam]

This is still an issue, using git on project of size 100MB costs a lot of time than to just run git in Windows.

[remy90]

Does anyone have an intuitive defender exclusion link (similar to @aseering's comment)? The link no longer exists

[mamama1]

for the love of god, switch to a different AV scanner. defender is a steaming pile of dog turd in terms of performance and efficiency.

i consider this as constructive feedback and a positive contribution to this thread as my proposed solution will solve the mentioned issues.

[lgallindo]

for the love of god, switch to a different AV scanner. defender is a steaming pile of dog turd in terms of performance and efficiency.

i consider this as constructive feedback and a positive contribution to this thread as my proposed solution will solve the mentioned issues.

Which one do you suggest?

[mamama1]

try the major brands like trend micro, sophos, mcaffee, crowdstrike. i won't recommend one as there would be the danger of MS buying and destroying it, as they do with everything they buy.

[hooluupog]

Any update? Windows defender still slows apps first time ran on win10 and win11.