Memory leak when called function raises error

[Tobi995]

Reporting a bug

• I have tried using the latest released version of Numba (most recent is
  visible in the change log (https://github.com/numba/numba/blob/main/CHANGE_LOG).
• I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

import gc

import numpy as np

from numba import njit
from numba.core.extending import register_jitable
from numba.core.runtime import rtsys


@register_jitable
def raise_error():
    raise ValueError("test")


@njit(parallel=False)
def leak():
    data = np.zeros((100, 2))
    raise_error()
    return data


print(rtsys.get_allocation_stats())
try:
    leak()
except Exception as e:
    ...
gc.collect()
print(rtsys.get_allocation_stats())

Prints:

nrt_mstats(alloc=0, free=0, mi_alloc=0, mi_free=0)
nrt_mstats(alloc=1, free=0, mi_alloc=1, mi_free=0)

I'm not 100% sure, but I think it's also causing leaks when calling np.argmin or np.argmax with the axis argument and a 0-length sequence:

@njit()
def jitargmin(arr, axis):
    return np.argmin(arr, axis)

print(rtsys.get_allocation_stats())
try:
    result = jitargmin(np.zeros((3, 0)), axis=1)
except Exception as e:
    ...
gc.collect()
print(rtsys.get_allocation_stats())

printing:

nrt_mstats(alloc=0, free=0, mi_alloc=0, mi_free=0)
nrt_mstats(alloc=6, free=4, mi_alloc=5, mi_free=3)

[stuartarchibald]

Thanks for the report. Unfortunately this is known behaviour, Numba doesn't implement unwinding etc to permit clean up. Thankfully, if performance is the goal, this doesn't often cause a great deal of impact as exceptions should occur only in "exceptional" circumstances and are not typically used in standard control flow decisions i.e. exception raised causes leak, but program also halts.

RE: the first example containing the leak function. Yes, this would trigger the problem.
RE: the second example using np.argmin. Yes, this would also trigger the problem.

I think "solving" this problem is likely to be quite involved.

[Tobi995]

Is this documented somewhere? I searched before creating the issue. I do agree that it's probably not too important, but it might confuse others, too

[stuartarchibald]

Is this documented somewhere? I searched before creating the issue. I do agree that it's probably not too important, but it might confuse others, too

I think this statement in the docs intends to cover it:

numba/docs/source/reference/pysemantics.rst

Lines 21 to 28 in 61e4b01

	Exceptions and Memory Allocation
	--------------------------------
	Due to limitations in the current compiler when handling exceptions, memory
	allocated (almost always NumPy arrays) within a function that raises an
	exception will **leak**. This is a known issue that will be fixed, but in the
	meantime, it is best to do memory allocation outside of functions that can
	also raise exceptions.

[stuartarchibald]

xref: #1549 (duplicate).

[Tobi995]

Ok, I missed that ones. Sorry for the effort. I guess we can close the issue then?

[stuartarchibald]

Ok, I missed that ones. Sorry for the effort. I guess we can close the issue then?

No problem. I'll close this as a duplicate. Thanks!