ndarray.ctypes creates a reference cycle

[pijyoi]

Reading attribute "ctypes.data" from an ndarray causes a leak.
version <= 1.14.5 doesn't leak
version >= 1.16.2 leaks
latest 1.16.4 leaks

Reproducing code example:

import numpy as np
import sys

print(np.version.full_version)

data = np.empty(8192)

before = sys.getrefcount(data)
data.ctypes.data
after = sys.getrefcount(data)
print(before, after)

assert before==after

Error message:

File "/mnt/d/code/python/test_numpy_leak.py", line 17, in
assert before==after
AssertionError

Numpy/Python version information:

1.16.4 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]

---

Summary (@seberg) 2019-12-03

This has been partially fixed by gh-14469; There are still issues here, although they are mainly issues with upstream ctypes creating deep reference cycles.

[eric-wieser]

What happens if you call gc.collect() before getting the refcount for a second time?

[seberg]

Unfortunately, that does not help. I tracked down a little and found

numpy/numpy/core/_internal.py

Line 272 in d94acc7

def _get_void_ptr(arr):

to be central to it. Specifically the last call to cast may be important.

I poked at circular references on the arr.base but reviving an older branch implementing tp_traverse did not seem to help (yet). So, not sure yet where it comes from.

[eric-wieser]

.data is a red herring.

There is a circular reference here:

arr.ctypes._data._some_internal_ref = arr

I think the problem is that arr.ctypes.data needs to construct _data every time, rather than holding onto the reference

[seberg]

Yeah, that fixes it. I am just a bit surprised right now that the reference cycle cannot be broken...

[eric-wieser]

The cycle isn't leaked at this point, it's still referenced by globals()['data']. The test here is not a good one

[seberg]

Ack, maybe I was just being stupid anyway, adding the collects at the wrong spot, oh man... Anyway, I think we can close this. The cycle can be broken and it is definitely cleaned up on master.

[seberg]

Ah well, at least that circular reference stuff is rebased on master again. Eventually that might end up being useful ;).

[eric-wieser]

Reopening since actually there shouldn't be a cycle here, array.ctypes returns a temporary.

At worst this is an unnecessary cycle though, not a leak - so not worth milestoning.

[pijyoi]

Although technically there isn't a leak, however the following code will balloon in memory usage with numpy 1.16, unless gc.collect() is called explicitly. With numpy 1.15, memory usage stays constant.

import os
import time
import gc
import numpy as np
import psutil

def stuff():
    data = np.ones(100*1048576, dtype=np.uint8)
    addr = data.ctypes.data

process = psutil.Process(os.getpid())

cnt = 0
while 1:
    stuff()
    # gc.collect()
    print(cnt, process.memory_info().rss//1048576)

    cnt += 1
    time.sleep(0.1)