Nonpaged pool increases during compilation

[firewave]

During the compilation of a bigger project on WSL using CMake my system started to run out of memory. The Task Manager was showing a memory consumption in high 90% range but none of the processes are using any of the memory (which also sometimes happens with CPU usage and is very confusing - but that's a different issue).

I stopped the compilation in WSL and nothing changed. So I opened up RAMMap to take a closer look and it was showing me this.

As you can see the "Nonpaged Pool" value is at about 7GB (it was at 800MB before I started the compilation).

Running "top" shows me this

I closed all WSL instances afterwards but it didn't change the memory consumption of the system at all. When I open WSL again the values in top were also unchanged.

Based on https://blogs.msdn.microsoft.com/ntdebugging/2012/08/30/troubleshooting-pool-leaks-part-2-poolmon/ I got the WDK and ran poolmon with

poolmon -b -n poolmon.log

poolmon.log

The biggest entry was

 MFE0 Nonp          1294242223         1291739578           2502645     5351579744             2138  

Currently I need to restart my machine to fix this problem.

I am using Windows 10 Version 1709 (OS Build 16299.214).

[firewave]

I did some more research on this and it turns out, that McAfee (which is running on my machine) had some issues with leaking Nonpaged pool memory in the past - so maybe this is a side effect of #1932 on my system.

[firewave]

It some brief (unintentional) tests and it seems, that McAfee is not leaking pooled memory in general (it scanned a lot of stuff for 10+ minutes and the Nonpaged pool memory didn't change at all) - so they might actually be caused by WSL.

[therealkenc]

The Task Manager was showing a memory consumption in high 90% range but none of the processes are using any of the memory

Sounds a bit like #2887. Maybe.

[benhillis]

@firewave - Thanks for reporting. I took a look through your poolmon log and don't see any outstanding allocations for any of our lxcore pool tags. I suspect McAfee might be leaking some memory when dealing with pico (WSL) processes.

[firewave]

"some memory" :P

It mainly seems to be the "MFE0" pool - and I didn't really pay attention to my findings earlier so I kinda missed, that that name is a tag and can be chosen at will.

So based on these two search results

https://community.mcafee.com/t5/VirusScan-Enterprise-VSE/VirusScan-Enterprise-leaks-nonpaged-pool/td-p/346717?tstart=0

https://www.experts-exchange.com/questions/22926786/Windows-Server-2003-nonpaged-memory-leaks.html

the pool with that tag is indeed related to McAfee. I posted a new support question

https://community.mcafee.com/t5/Endpoint-Security-ENS/Nonpaged-pool-memory-leak-with-WSL-on-Windows-10/m-p/582374

[benhillis]

Pool tags can be chosen at will but components will often choose something to make it easier to identify what component allocated the memory. For example, our driver's pool tags begin in "Lx"

[firewave]

I wrote a small script

#!/bin/bash
for (( ; ; ))
do
        gcc -c test.c &
done

which compiles this simple source

int main() {
        return 0;
}

When monitoring the system with poolmon you can see the size of the MFE0 pool increasing. It's not as massive as a leak I am having with the actual scenario I ran into and it's not clear if it is caused by the processes, compilation or even I/O - but at least it's a simple test case to reproduce it with.

Edit: Silly me - there already was a small test case in the other ticket - well, at least another one this could be verified with.

[BobBall]

Just FYI I have disabled McAfee (can't uninstall it because it's in managed mode) and to confirm I do not see a reproduction with McAfee disabled.

[therealkenc]

Calling this external since there is pretty clear confirmation on McAfee and no WSL actionable, for now at least.

[firewave]

Fair enough.
But we have seen there was no reaction at all from McAfee. I wonder if this maybe could be escalated from the Microsoft side to McAfee.
Since WSL is now coming to Windows Server I am pretty sure we will have many more people running into this issue very soon on more critical machines which definitely can't just be restarted because they ran out of memory.

[firewave]

McAfee finally replied and acknowledged the problem. The initial response stated, that "It is slated to be fixed in ENS 10.5.4" but that sentence has been removed now. I guess since that version was already released almost two weeks ago and the release notes do not mention this being fixed - see https://kc.mcafee.com/resources/sites/MCAFEE/content/live/PRODUCT_DOCUMENTATION/27000/PD27598/en_US/ens_1054_rn_ePOCloud_0-00_en-us.pdf

Let's hope it will make a Hotfix.

[firewave]

ENS 10.5.4 was rolled out for me yesterday (and I also updated to 1803) and currently it seems like the memory consumption is stable.