Memory leak when pickling ndarrays with NumPy 1.16.0

[itayperl]

Reproducing code example:

import numpy as np
import pickle
import os
import itertools
import psutil
import gc
import sys

print ('NumPy version:', np.__version__, 'Python version:', sys.version)
for i in itertools.count():
    pickle.dumps(np.zeros(1024*1024))
    gc.collect()
    if i % 100 == 0:
        info = psutil.Process(os.getpid()).memory_info()
        print(f'vms={info.vms/1024/1024}MB rss={info.rss/1024/1024}MB')

Output on NumPy 1.15.4

NumPy version: 1.15.4 Python version: 3.6.3 (default, Nov 23 2017, 18:55:59) 
[GCC 5.4.0 20160609]
vms=434.71875MB rss=27.421875MB
vms=434.72265625MB rss=27.52734375MB
vms=434.72265625MB rss=27.52734375MB
vms=434.72265625MB rss=27.52734375MB
vms=434.72265625MB rss=27.52734375MB
vms=434.72265625MB rss=27.52734375MB
vms=434.72265625MB rss=27.52734375MB
vms=434.72265625MB rss=27.52734375MB

Output on NumPy 1.16.0

$ python test.py
NumPy version: 1.16.0 Python version: 3.6.3 (default, Nov 23 2017, 18:55:59) 
[GCC 5.4.0 20160609]
vms=431.58984375MB rss=27.34765625MB
vms=1239.5859375MB rss=827.47265625MB
vms=2039.58984375MB rss=1627.4765625MB
vms=2839.58984375MB rss=2427.4765625MB
vms=3639.59375MB rss=3227.4921875MB
vms=4439.59375MB rss=4027.5078125MB
vms=5239.59375MB rss=4827.51953125MB
vms=6039.59765625MB rss=5627.5390625MB

[mjpieters]

Simpler reproducing code example:

import numpy as np
import pickle
import sys

print ('NumPy version:', np.__version__, 'Python version:', sys.version)
f = np.zeros(0)
refcount = sys.getrefcount(f)
pickle.dumps(f)
print("Refcount delta:", sys.getrefcount(f) - refcount)

This prints Refcount delta: 1 on Numpy 1.16.0, while on 1.15.4 it reports a delta of 0.

[charris]

A number of memory leak fixes have gone in recently but this is still failing. @seberg Thoughts?

[seberg]

I do not remember touching this area, my guess it has to do with the Pickle protocol 5 support changes, so pinging @pitrou. But I can probably track it down later in any case.

[mjpieters]

It does look like 64a855f is the culprit here, because I can simplify the test to calling f.__reduce_ex__(4) (any int between 0 and 4 will do, or 5 when using Python 3.8 or pickle5).

[mjpieters]

Correction: when using protocol 5 the reference count does not go up, only for 0 - 4.

[pitrou]

Yes, it seems the original code has reference leaks unfortunately. Those leaks should be fixed in PR #12748 (at least this present one is).

[seberg]

@pitrou makes this more important, can you add that refcount example as a test there? I guess we need the whole thing anyway for the next 1.16 release.

[pitrou]

@seberg Done.