BUG: reference cycle in np.vectorize

[mattip]

Issue #11867 demonstrated a leak when using np.vectorize on a bound class method in CPython 3. The test demonstrates the issue.

Still WIP since I have not found the root cause of the leak.

Edit: adding GC_Track to ufunc allows the garbage collector to break the cycle

[eric-wieser]

My guess would be that the problem is np.ufunc not implementing a traverse method to tell the GC that it holds a cycle

[mattip]

ping

[eric-wieser]

Have we worked out why this is the case? Am I right in thinking that this patch changes it from being unbreakable to just requiring a gc to break?

[mattip]

Correct. I do not really understand the root cause and why only in this specific case

[mattip]

Squashed to a single commit.

[eric-wieser]

I think the commit message is wrong - you didn't break the reference cycle, you simply allowed the GC to find it (and break it on the python side)

[eric-wieser]

I wonder if you can use assert_no_gc_cycles here, which prints out the cycle upon failure, if possible

[mattip]

fixed first commit message, reworked test to focus on class.atrribute = frompyfunc(class.method) where the reference cycle is actually created

[eric-wieser]

Would be good to finish this up

[eric-wieser]

tp_traverse will need minor changes to accommodate the extra reference added in #8955 - I suppose the PR of the two that doesn't go in first will need updating.

Fix will be as simple as:

if (ufunc->identity == PyUFunc_IdentityValue) {
    Py_VISIT(ufunc->identity_value);
}

[mattip]

merging with master to resolve conflicts

[mattip]

i will wait for #8955 and then rework this PR

[mattip]

rebased at git merge-base master maintenance/1.16.x. Would it be possible to get this into 1.16?

[eric-wieser]

I'm very confused by what exactly we're trying to test here.

[mattip]

self.bound is part of a reference cycle that requires a gc collection cycle to break. I added more clarification in the test docstring. Improvements welcome

[seberg]

The PR looks good to me, if nobody complains will merge in a bit probably. The test couldbe a bit more obvious to read, or maybe add a trivial one:

def func(a):
    return 0
func.vectorized = np.frompyfunc(func)

giving a minimal cycle. The other part would probably be simpler by making it a realistic use case:

class A(object):
    def __init__(self):
        # Wrap bound_method into a ufunc with `np.frompyfunc`. This creates
        # a reference cycle:
        #     self.method -> self   # since it is a bound method (knows "self")
        #     self -> self.wrapped_method  # Set here
        #     self.wrapped_method -> self.method  # Stores self.method to call it
        self.wrapped_method = np.frompyfunc(self.method)

    def method(self):
        return 0

[seberg]

Ah, I guess that breaks the parameterization. Although, I do not think you need the types.MethodType if the just do the self.bound_vectorized = np.frompyfunc(self.bound, 1, 1) before returning it?

[mattip]

I'm not following. Where would I add that line?

[seberg]

Sorry, I think what confuses me is that I believe you do not need this at all. You can simply remove the npy_bound and npy_unbound logic and just make it a.f = np.frompyfunc(func, 1, 1).

[mattip]

Adopted

[mattip]

should be squash-merged or I can squash and force-push

[charris]

If you want it in 1.16, use the 1.16.1 milestone.

[mattip]

@eric-wieser ping

[seberg]

Ah, I guess that breaks the parameterization. Although, I do not think you need the types.MethodType if the just do the self.bound_vectorized = np.frompyfunc(self.bound, 1, 1) before returning it?

[seberg]

Thanks @mattip lets put it in before we find something else to nitpick about ;).

[eric-wieser]

Test now makes sense to me too - thanks for persisting

[seberg]

Cool, mac OS job is expected to fail. Next thing will be the object arrays ;).