Inconsistency in memory reporting on Windows 8/8.1/10 only

[khartling]

Since the folks at Microsoft were not of much help for this issue, I'm hoping someone here may have seen this issue and found a solution either with psutil or a windows setting even though everything so far is pointing this to being a windows issue.
Using Windows 8/8.1/10 only (Does not show this on Windows 7) I have python scripts that run the same exact call to one of our tools over and over again to make sure after the tool runs it is not holding onto any memory. First it runs the tool to initialize a few things, then runs it a second time and uses its stats as a base with which to compare each iteration. I see some pretty strange results when looking at the amount of memory the python process is using after each call to the tool (see attached file)
repro_IDENTITYTEST502_resLog_2016_02_01_11_30_57.xlsx
iteration Mem (mb) VMem (mb) diff Mem (mb) diff VMem (mb)
2 179.6 138.4 2.3 7.4
3 179.8 137.3 2.4 6.3
4 178.4 133.3 1.0 2.3
5 181.1 139.6 3.7 8.6
6 180.4 139.0 3.0 8.0
7 182.5 139.7 5.1 8.7
8 181.6 138.1 4.2 7.2
9 182.3 135.3 5.0 4.4
10 283.8 241.3 106.5 110.3
11 183.1 139.3 5.8 8.3
12 290.0 248.4 112.6 117.4
13 300.7 258.7 123.3 127.7
14 303.5 261.4 126.2 130.4
15 306.3 264.3 129.0 133.3
16 307.8 265.2 130.4 134.2
17 311.0 268.3 133.6 137.3
18 185.7 141.5 8.3 10.5
19 319.5 280.2 142.1 149.2
... etc

Note how on occasion it seems that the process has held onto most of the memory the tool had used (peak python.exe process size in this case was about 435MB) , but eventually lets go of this memory on another iteration... only to hold onto it again later.

If I run exactly the same script on a Windows 7 machine, this does not happen. The memory from iteration to iteration stays flat (basically stays about the same as the base).

On W8/8.1/10 looking at the Resource Monitor it appears that psutil is reporting what is in reality happening. W8/8.1/10 is holding onto memory as noted above. Running the script and manually noting the memory usage of the python.exe process appears to also confirm the W8/8.1/10 behaviour. If I remove the psutil import from the script, the issue still seems to occur.

Here's how I'm getting the size of the python.exe process:

def getMemory(self):
        if os.getenv('WINEPREFIX'):
            # elinux.org/Runtime_Memory_Measurement
            import linecache
            linecache.clearcache()
            vmem = linecache.getline(
                        '/proc/{0}/status'.format(self.pID), 16).strip().split(' ')
            assert(vmem[0] == 'VmRSS:\t') # insure we got the right line
            try:
                ivmem = int(vmem[2])
            except ValueError:
                ivmem = int(vmem[1])
            return 0.0, ivmem/1024.0 # kb to mb

        else:
            code  = 'import psutil\n'
            code += 'p = psutil.Process({0})\n'.format(self.pID)
            code += 'pMem = p.memory_info()\n'
            code += 'print(pMem.rss/1048576.0, pMem.vms/1048576.0)'
            proc = subprocess.Popen([sys.executable],
                                    stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
                                    stderr=subprocess.STDOUT,
                                    universal_newlines=True,
                                    )
            r = proc.communicate(code)[0].strip()
            if sys.version_info[0] == 2:
                r = r.lstrip('(').rstrip(')').replace(',','')
            return map(float,r.split(' '))

[giampaolo]

I'm personally not sure what to recommend as this seems a question related to how Windows 8+ handles (process) memory internally, and I'm ignorant about that. I think you might have better luck asking on some specific windows forum ("sysinternals" or something).

[khartling]

Yeah... I agree... and I did (thus the "Since the folks at Microsoft were not of much help for this issue"). They wanted me to do a RAM check...