Possible Memory Corruption for xtensor with mismatching type (using xsmid)

[Mightrider]

I recently run into a memory corruption when I returned an xtensor of a different type. Here is a short example on to get what I did:

template<typename OutType, typename CreateType>
xtensor<OutType, 2> get(const size_t vShape) {
    xtensor<CreateType, 2> vTensor = ones<CreateType>({ vShape, vShape });
    return vTensor;
}

xtensor<OutType, 2> vResult = get<uint8_t, float>(vExpected.shape(0));

In my example I return a xtensor<float,2> but the function returns xtensor<uint8_t,2>. In some cases my program crashed badly and in some it seemed to work fine. Unfortunately I did not manage to write a small test which actually crashes but if I check it with valgrind I get an error which starts with Invalid write of size 8. I attached a log below. Interestingly the error only exists if I implicitly cast from:

• float to uint8/int8
• double to uint16/int16
• double to uint16/int16

valgrind error

==19475== Invalid write of size 8
==19475==    at 0x1251E4: _mm_storel_epi64 (emmintrin.h:727)
==19475==    by 0x1251E4: store_aligned (xsimd_sse_float.hpp:400)
==19475==    by 0x1251E4: store_aligned (xsimd_load_store.hpp:392)
==19475==    by 0x1251E4: store_aligned<unsigned char, float> (xsimd_load_store.hpp:540)
==19475==    by 0x1251E4: store_simd<unsigned char, float> (xsimd_load_store.hpp:628)
==19475==    by 0x1251E4: store_simd<xsimd::aligned_mode, xsimd::batch<float, 4> > (xcontainer.hpp:710)
==19475==    by 0x1251E4: run<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xassign.hpp:670)
==19475==    by 0x1251E4: assign_data<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xassign.hpp:389)
==19475==    by 0x1251E4: assign_xexpression<xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> >, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > > (xassign.hpp:412)
==19475==    by 0x1251E4: operator()<xtl::identity> (xassign.hpp:204)
==19475==    by 0x1251E4: decltype(auto) xtl::mpl::static_if<void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#1}, void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#2}>(std::integral_constant<bool, false>, void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#1} const&, void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#2} const&) (xmeta_utils.hpp:539)
==19475==    by 0x1255E7: static_if<false, xt::assign_xexpression(xt::xexpression<D>&, const xt::xexpression<E2>&) [with E1 = xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>; E2 = xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>]::<lambda(auto:69)>, xt::assign_xexpression(xt::xexpression<D>&, const xt::xexpression<E2>&) [with E1 = xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>; E2 = xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>]::<lambda(auto:70)> > (xmeta_utils.hpp:545)
==19475==    by 0x1255E7: assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xassign.hpp:198)
==19475==    by 0x1255E7: assign_xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xsemantic.hpp:623)
==19475==    by 0x1255E7: assign<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xsemantic.hpp:489)
==19475==    by 0x1255E7: xtensor_container<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xtensor.hpp:527)
==19475==    by 0x1255E7: xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> xt::get<unsigned char, float>(unsigned long) (test_Invalid_write_size.cpp:12)
==19475==    by 0x120EBF: xt::InvalidWriteOfSizeTest_FloatInt8ValgrindFail_Test::TestBody() (test_Invalid_write_size.cpp:20)
==19475==    by 0x1574A8: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2433)
==19475==    by 0x1574A8: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2469)
==19475==    by 0x14C820: testing::Test::Run() (gtest.cc:2508)
==19475==    by 0x14C936: testing::TestInfo::Run() (gtest.cc:2684)
==19475==    by 0x14C9EC: testing::TestSuite::Run() (gtest.cc:2816)
==19475==    by 0x14CE26: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:5338)
==19475==    by 0x1579A3: HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (gtest.cc:2433)
==19475==    by 0x1579A3: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2469)
==19475==    by 0x14D08A: testing::UnitTest::Run() (gtest.cc:4925)
==19475==    by 0x157E71: RUN_ALL_TESTS (gtest.h:2473)
==19475==    by 0x157E71: main (gtest_main.cc:45)
==19475==  Address 0x5fab23c is 1,048,588 bytes inside a block of size 1,048,592 alloc'd
==19475==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19475==    by 0x125174: xaligned_malloc (xsimd_aligned_allocator.hpp:285)
==19475==    by 0x125174: aligned_malloc (xsimd_aligned_allocator.hpp:305)
==19475==    by 0x125174: allocate (xsimd_aligned_allocator.hpp:176)
==19475==    by 0x125174: safe_init_allocate<xsimd::aligned_allocator<unsigned char, 16> > (xstorage.hpp:172)
==19475==    by 0x125174: resize_impl (xstorage.hpp:224)
==19475==    by 0x125174: resize (xstorage.hpp:370)
==19475==    by 0x125174: resize_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> > > (xutils.hpp:336)
==19475==    by 0x125174: resize_data_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, long unsigned int> (xcontainer.hpp:888)
==19475==    by 0x125174: resize<const std::array<long unsigned int, 2>&> (xcontainer.hpp:922)
==19475==    by 0x125174: resize<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xassign.hpp:492)
==19475==    by 0x125174: assign_xexpression<xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> >, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > > (xassign.hpp:411)
==19475==    by 0x125174: operator()<xtl::identity> (xassign.hpp:204)
==19475==    by 0x125174: decltype(auto) xtl::mpl::static_if<void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#1}, void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#2}>(std::integral_constant<bool, false>, void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#1} const&, void xt::assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >(xt::xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> >&, xt::xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> > const&)::{lambda(auto:1)#2} const&) (xmeta_utils.hpp:539)
==19475==    by 0x1255E7: static_if<false, xt::assign_xexpression(xt::xexpression<D>&, const xt::xexpression<E2>&) [with E1 = xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>; E2 = xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>]::<lambda(auto:69)>, xt::assign_xexpression(xt::xexpression<D>&, const xt::xexpression<E2>&) [with E1 = xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>; E2 = xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>]::<lambda(auto:70)> > (xmeta_utils.hpp:545)
==19475==    by 0x1255E7: assign_xexpression<xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag>, xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xassign.hpp:198)
==19475==    by 0x1255E7: assign_xexpression<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xsemantic.hpp:623)
==19475==    by 0x1255E7: assign<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xsemantic.hpp:489)
==19475==    by 0x1255E7: xtensor_container<xt::xtensor_container<xt::uvector<float, xsimd::aligned_allocator<float, 16> >, 2, (xt::layout_type)1, xt::xtensor_expression_tag> > (xtensor.hpp:527)
==19475==    by 0x1255E7: xt::xtensor_container<xt::uvector<unsigned char, xsimd::aligned_allocator<unsigned char, 16ul> >, 2ul, (xt::layout_type)1, xt::xtensor_expression_tag> xt::get<unsigned char, float>(unsigned long) (test_Invalid_write_size.cpp:12)
==19475==    by 0x120EBF: xt::InvalidWriteOfSizeTest_FloatInt8ValgrindFail_Test::TestBody() (test_Invalid_write_size.cpp:20)
==19475==    by 0x1574A8: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2433)
==19475==    by 0x1574A8: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2469)
==19475==    by 0x14C820: testing::Test::Run() (gtest.cc:2508)
==19475==    by 0x14C936: testing::TestInfo::Run() (gtest.cc:2684)
==19475==    by 0x14C9EC: testing::TestSuite::Run() (gtest.cc:2816)
==19475==    by 0x14CE26: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:5338)
==19475==    by 0x1579A3: HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (gtest.cc:2433)
==19475==    by 0x1579A3: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2469)
==19475==    by 0x14D08A: testing::UnitTest::Run() (gtest.cc:4925)
==19475==    by 0x157E71: RUN_ALL_TESTS (gtest.h:2473)
==19475==    by 0x157E71: main (gtest_main.cc:45)

See the pull request for additional the full tests.

I compiled it with xsimd instructions, gcc 7.4.0 on Ubuntu 18.04.

[emmenlau]

We experienced this with the pre-latest xsimd and pre-latest xtl, so please take it with a grain of salt. Today I'll update to latest xsimd and latest xtl, and then we can see if its reproducible.

[Mightrider]

Unfortunately valgrind still shows the same results.

[yjqww6]

I came across the same problem.
I think it is because _mm_loadl_epi64/_mm_storel_epi64 is used in many places where 8x4/16x2=32 bits need to be read/write, and it simply goes past the end.