Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Free vector memory in cython #3037

Open
herleeyandi opened this issue Jul 9, 2019 · 4 comments

Comments

Projects
None yet
4 participants
@herleeyandi
Copy link

commented Jul 9, 2019

Hi I have a problem while free the memory spaces after the vector usage.

cdef vector[Mystruct] cb
.... #do something
cb.clear()

After running the program, the memory usage is not reduced which indicate the usage memory is not deallocated. in vector .clear() function it is not free the memory usage. In C++ 11 we can use shrink_to_fit() function as state in here, however I can't use shrink_to_fit function inside the cython, and throws this error while compiling it.

has no member named ‘shrink_to_fit’
   __pyx_v_cb.shrink_to_fit();

I also consider swap() function where we can just swap the vector by 0 length vector but its still not working.

cdef vector[Mystruct] cb
cdef vector[Mystruct] dummy
.... #do something
cb.swap(dummy)

I also use garbage collector in python but also not working.

import gc
cdef vector[Mystruct] cb
.... #do something
cb.clear()
gc.collect()

So is there any solution for this problem?, I am using Cython 0.29.11 and gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10).

@scoder

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2019

the memory usage is not reduced

Do you mean the memory used by the process? That's not necessarily a good indicator as it tends to be influenced by many factors, not just single chunks of memory.

can't use shrink_to_fit

You have to enable C++11 support in your C compiler for that, e.g. CXXFLAGS=-std=c++11.

Can you provide a complete example that shows your problem? It's quite likely that the #do something has more impact on what you are witnessing than the cb.clear().

In any case, this is most likely not related to a bug in Cython, and thus shouldn't be discussed on the bug tracker. The cython-users mailing list seems a more appropriate place.

@herleeyandi

This comment has been minimized.

Copy link
Author

commented Jul 9, 2019

Hello @scoder, thanks for the reply.
Yes, it is the memory used by the process because I have assigned a huge vector to record some data, then after I used it I want to free the memory. However, I can't free it. So do something is only cython vector push_back() function which spends about 2GB of memory. I think cb.clear() is the problem because before and after call cb.clear() it not reduce some memory, maybe at least 1 GB or even 2GB. Second is how to enable C++11 for cython?, I have a problem while using extra_compile_args=["-std=c++11"] in linux. It gives me this error ValueError: got unknown compilation option, please remove: extra_compile_args . The last I am sorry since I am newbie in here I don't know that the mailing list is exist. Is there any cython forum discussion which more convenient rather than a mailing list?
-Thank you-

@david-cortes

This comment has been minimized.

Copy link

commented Jul 15, 2019

Hello @scoder, thanks for the reply.
Yes, it is the memory used by the process because I have assigned a huge vector to record some data, then after I used it I want to free the memory. However, I can't free it. So do something is only cython vector push_back() function which spends about 2GB of memory. I think cb.clear() is the problem because before and after call cb.clear() it not reduce some memory, maybe at least 1 GB or even 2GB. Second is how to enable C++11 for cython?, I have a problem while using extra_compile_args=["-std=c++11"] in linux. It gives me this error ValueError: got unknown compilation option, please remove: extra_compile_args . The last I am sorry since I am newbie in here I don't know that the mailing list is exist. Is there any cython forum discussion which more convenient rather than a mailing list?
-Thank you-

You can try here: https://groups.google.com/forum/#!forum/cython-users
But answering the question, it looks like you're putting the extra_compile_args in the wrong place, they should be inside the arguments for Extension, not for setup (if you're using a setup file). Otherwise, you can pass arguments to cythonize if that's hou you're building it, or you can launch the compilation command pre-defining the flags beforehand, e.g. CXXFLAGS=-std=c++11 python setup.py build_ext --inplace --force.

BTW, if you're returning this object outside of cython and then passing it again, it won't be a C++ vector anymore (so no methods like those), in case that's what you're doing.

@sehoffmann

This comment has been minimized.

Copy link

commented Jul 16, 2019

clear() must not reduce allocated memory:

from https://en.cppreference.com/w/cpp/container/vector/clear :

Leaves the capacity() of the vector unchanged (note: the standard's restriction on the changes to capacity is in the specification of vector::reserve, see [1])

As your vector is (assumily) stack allocated, simply leaving the function should suffice to not only destruct every object but also to free any space held by the vector. You can introduce an extra scope in Cython by creating a new sub-function. Not nice but also not too bad.

Only if you want to reuse the same vector, or if you hold the vector in a Extension Type and need to control its lifetime, a call to clear() and shrink_to_fit() should be neccessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.