-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Windows bug #33
Comments
It sounds like this due to reusing the print("Test armadillo matrix to numpy array functions.")
import numpy as np
import test_carma as carma
print("Before")
print("Test mat_to_arr with c++ addition.")
sample = np.asarray(
np.random.normal(size=(10, 2)),
dtype=np.float,
order='F'
)
mat = carma.mat_to_arr_plus_one(sample, False)
assert np.allclose(mat, sample + 1)
# This should trigger a destructor call
sample = None
# This just removes the sample from the namespace
del sample
print("After") This would also mean this is a fairly significant memory bug. |
But the above wouldn't explain why the first print statement isn't reached or why |
If this corruption occurs in the Python interpreter, it is possible that it crashes before to able to print previously computer results or outputs (buffered outputs are not yet flushed). Since the problem occurs, only this two successive tests (in any order; and ok with only one), it could help to understand where the corruption is. |
Next thing to do could be to try with I plan also to add a mode with memory sanitizer options. |
Your example has also failed (cf https://travis-ci.com/github/hpwxf/carma/jobs/358021677). I'm going to try to check what happens on Linux with valgrind (using https://pypi.org/project/pytest-valgrind/) |
An error seems to occur here:
|
If I remove the |
Adds an additional argument to define how to manage associated memory for py::capsule. Visible at runtime in Windows or using pytest-valgring in Linux. (closes RUrlus#33)
Adds an additional argument to define how to manage associated memory for py::capsule. Visible at runtime in Windows or using pytest-valgring in Linux. (closes RUrlus#33)
I think the bug might be in get_data. template <typename armaT, typename = std::enable_if_t<is_convertible<armaT>::value>>
inline typename armaT::elem_type * get_data(armaT * src, bool copy) {
using T = typename armaT::elem_type;
if (copy) {
size_t N = src->n_elem;
T * data = new T[N];
std::memcpy(data, src->memptr(), sizeof(T) * N);
return data;
} else {
T * data = src->memptr();
arma::access::rw(src->mem) = 0;
return data;
}
} /* get_data */ If you don't copy in but do copy out we have:
However, under conditions the input array will get copied, e.g. wrong memory order, owndata is false, which prevents us from always taking ownership away from armadillo. Perhaps we can set a global flag that indicates when a copy occurred on the way in and use that to either take away ownership from armadillo when a copy is made out. |
What do you think about the fix I proposed in #35 ? |
Wouldn't an intermediate solution be to use a macro that sets delete when OS is not Windows? That way we can support windows albeit with a memory leak but don't interfere with performance on *NIX systems |
Since I've planned to use carma in another project (portable on Linux/Mac/Windows), I prefer to solve this problem without any leak in any OS. |
Using clang sanitizer, the analysis seems to be easier
I got:
That means the related bunch of memory has been allocated using |
Understood ! To deallocate a bunch of memory allocated by armadillo, you cannot assume what allocator it is. It depends on the implementation. |
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Of course, I was thinking as a stop-gap solution. |
Good point, nooby mistake to assume delete was the appropriate way too free. What do you think about letting armadillo handle the copy out as well. In that case we can always keep the armadillo matrix/row/... in the capsule and release the memory from armadillo? In the case of a steal we can move the mat/row/col etc. This might also solve another issue where we can't reliable steal the memory from a Col or Row. Something like: template <typename armaT, typename = std::enable_if_t<is_convertible<armaT>::value>>
inline py::capsule create_capsule(const armaT & src) {
using T = typename armaT::elem_type;
armaT data = armaT(src.memptr(), src.n_rows, src.n_cols, true);
return py::capsule(data.memptr(), [](void *f){
T *data = reinterpret_cast<T *>(f);
#ifndef NDEBUG
// if in debug mode let us know what pointer is being freed
std::cerr << "freeing memory @ " << f << std::endl;
#endif
arma::memory::release(data);
});
} /* create_capsule */
template <typename armaT, typename = std::enable_if_t<is_convertible<armaT>::value>>
inline py::capsule create_capsule(armaT && src) {
using T = typename armaT::elem_type;
armaT data = std::move(src);
return py::capsule(data.memptr(), [](void *f){
T *data = reinterpret_cast<T *>(f);
#ifndef NDEBUG
// if in debug mode let us know what pointer is being freed
std::cerr << "freeing memory @ " << f << std::endl;
#endif
arma::memory::release(data);
});
} /* create_capsule */ |
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes RUrlus#33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes #33)
Embed deallocator type with data in get_data return value. Visible at runtime in Windows or using pytest-valgring or clang address sanitizer in Linux (closes #33)
To reproduce it:
.travis-ci.yml
and run CI testWhen
test_mat_to_arr.py
is reduced toIt is still failing. But if we remove first sub-test or second sub-test, it is OK.
NB: when it fails, there is no output even for the first
print
s.or by hand:
PS:
test_arr_to_mat.py
is OK.test_arraystore.py
,test_nparray.py
,test_type_caster.py
are KO liketest_mat_to_arr.py
The text was updated successfully, but these errors were encountered: