another memory leak in indexing ? arrayfire+python

[mmllee]

It seems there's a leak while using arrayfire with python bindings
way to reproduce a bug:

import arrayfire as af
import numpy as np
af.set_backend('cpu') # or 'cuda', problem occurs for both
af_arr = af.np_to_af_array(np.random.uniform(size=(1000,1000,3)))
for i in xrange(10000):
for b in xrange(32):
af_arr[:,:,1]
print 'iter %s' % i
raw_input('check the memory now')

on my machine initially takes 738 MB but after 10000 iters it takes 944 MB

It's pretty clear the leak must be somewhere in the indexing

i compiled arrayfire (62dc920) myself on ubuntu 16.04 vanilla, python bindings (cafc208)

initially i reported it here: arrayfire/arrayfire#1649, but I was instructed to report it in this repo instead

here's the interesting part from valgrind output:

==10902== 61,423,904 (51,187,360 direct, 10,236,544 indirect) bytes in 319,921 blocks are definitely lost in loss record 1,669 of 1,669
==10902== at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10902== by 0x13D0D550: cpu::Array* cpu::initArray() (in /usr/lib/libafcpu.so.3.5.0)
==10902== by 0x1483D8B1: af_create_handle (in /usr/lib/libafcpu.so.3.5.0)
==10902== by 0x55A2D387: af_err unified::AFSymbolManager::call<void**, unsigned int, long long const*, af_dtype>(char const*, void**, unsigned int, long long const*, af_dtype) (in /usr/lib/libaf.so.3.5.0)
==10902== by 0xEEE1E3F: ffi_call_unix64 (in /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4)
==10902== by 0xEEE18AA: ffi_call (in /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4)
==10902== by 0xECC73DE: _ctypes_callproc (in /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so)
==10902== by 0xECCBD81: ??? (in /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so)
==10902== by 0x4B0CB2: PyObject_Call (in /usr/bin/python2.7)
==10902== by 0x4C9FAE: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==10902== by 0x4C9D8E: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==10902== by 0x4C2764: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==10902==
==10902== LEAK SUMMARY:
==10902== definitely lost: 51,187,544 bytes in 319,923 blocks
==10902== indirectly lost: 10,236,544 bytes in 319,892 blocks
==10902== possibly lost: 184,638 bytes in 276 blocks
==10902== still reachable: 3,985,207 bytes in 5,652 blocks
==10902== suppressed: 0 bytes in 0 blocks
==10902== Reachable blocks (those to which a pointer was found) are not shown.
==10902== To see them, rerun with: --leak-check=full --show-leak-kinds=all

[pavanky]

@mmllee Just an update.

I have been able to reproduce this bug in the python wrapper but not in the C++ version. I am digging further into the issue.

[pavanky]

Hmm on the other hand, there is no memory leak if I comment out af_index_gen from array.py. It is weird that the following code doesn't produce a memory leak in simple C code

        af_array in, out;
        dim_t dims[] = {1000, 1000, 3};
        af_randu(&in, 3, dims, f32);
        af_index_t indices[3];

        indices[0].isSeq = true;
        indices[0].isBatch = true;   
        indices[0].idx.seq = af_span;

        indices[1] = indices[0];
        indices[2] = indices[0];
        indices[2].idx.seq.begin = 1;
        indices[2].idx.seq.end = 1;
        indices[2].idx.seq.step = 1;

        printf("Starting loop. Hit any key to begin\n");
        getchar();
        for (int i = 0; i < 32000; i++) {
            af_index_gen(&out, in, 3, indices);
            af_release_array(out);
            printf("%d\n", i);
        }
        printf("End of loop loop. Hit any key to end\n");
        getchar();

[pavanky]

@mmllee after looking more closely into your valgrind output, I think I found the problem :) Thanks for posting that.

This is a bug that should affect almost all functions and I am not sure when it was introduced. But I think this is an easy fix that should be done fairly soon.

[pavanky]

@mmllee can you try the fix from #105 ? I'll be testing a bit and merge another fix from devel and releasing a bugfix release tomorrow if everything is OK.

[mmllee]

@pavanky thanks a lot for looking into it so promptly. #105 fixes the leak.
After isolating the leak even to a single index access I was quite puzzled.
As you found out it was not only the index but actually almost every single function in python was affected by it. Strange that noone noticed before.

[pavanky]

@mmllee The leak may have always existed, but the size of the leak may have increased recently with a combination of things changing upstream and the python wrapper.

Once again, thanks for taking the time to point this out.

While debugging this, I also noticed that indexing is extremely slow (compared to pure C) as discussed here: #84.

I am trying to see if I can change that as well :)

[pavanky]

Fixed in master.