Sanitizers do not work with JNI #34

alexbaden · 2022-06-07T19:10:11Z

Needs investigation.

leshikus · 2022-06-24T20:01:19Z

I started with investigation how ASAN worked for omniscidb

Garra1980 · 2022-06-24T20:20:00Z

@alexbaden do you have smth in mind already?

leshikus · 2022-06-27T09:38:55Z

There are link errors when linking ArrowBasedExecuteTest with ASAN, namely _start absence in Scrt.o; comparing the linking command line to the one which works

alexbaden · 2022-06-27T20:26:02Z

@ienkovich and I suspect JNI will be the biggest problem getting ASAN to work with HDK. So, our idea was to drop Calcite out of the unit tests when compiling with ASAN or TSAN on. We will intercept the payload from Calcite (optimized RA tree), write it to disk, then run the ASAN binary using that payload (bypassing SQL parse and Calcite optimization). A first PR to support this was already merged: https://github.com/intel-ai/omniscidb/pull/430

ienkovich · 2022-06-27T21:30:24Z

Right. So far it is supported only in ArrowBasedExecuteTest. You can run it with --build-rel-alg-cache=<file> to collect Calcite responses and then run again with --use-rel-alg-cache=<file> to avoid Calcite construction and any JNI calls. I didn't try to use it with sanitizers yet.

leshikus · 2022-06-28T16:32:38Z

There are some old reports that ASAN works with JNI with instrumentation, I wonder if we can get something out of it. google/sanitizers#271

leshikus · 2022-07-01T13:02:56Z

With ArrowBasedExecuteTest related tests excluded and ASAN_OPTIONS="detect_leaks=0" most other tests pass. I can commit this to CI if no one objects to the exclusion.

leshikus · 2022-07-01T13:03:49Z

asan.log

alexbaden · 2022-07-01T17:16:21Z

ResultSetTest large buffers should be skipped under TSAN and ASAN because the memory usage is too large; the increase from TSAN/ASAN tagging makes the buffers too big.

What errors are you getting with detect leaks on? Only limited to JNI? We might try a suppression in that case.

leshikus · 2022-07-01T17:36:49Z

Most of leaks are reported in libjvm.so and originate from standard jni memory allocation routines; there is a report that jvm can be rebuilt in a way that the leaks are shown correctly.

alexbaden · 2022-07-01T17:38:15Z

Did you try suppressing all of libjvm? https://clang.llvm.org/docs/AddressSanitizer.html#suppressing-reports-in-external-libraries

leshikus · 2022-07-01T18:32:00Z

thanks for the link, will try

leshikus · 2022-07-05T08:11:17Z

Tried suppressions feature, it did not work for me yet. Also tried LSAN_OPTIONS

Looking further

ienkovich · 2022-09-13T15:33:38Z

Most of leaks are reported in libjvm.so and originate from standard jni memory allocation routines; there is a report that jvm can be rebuilt in a way that the leaks are shown correctly.

We can avoid JNI by using cached Calcite responses. So far it is supported only in ArrowBasedExecuteTest. You can run it with --build-rel-alg-cache=<file> to collect Calcite responses and then run it again with --use-rel-alg-cache=<file> to avoid Calcite construction and any JNI calls. I tried it with ASAN some time ago and it worked fine. If it works for you, I can support this feature for all other test suites.

leshikus · 2022-09-26T19:35:08Z

I have tried both options, total Calcite parsing time decreased from 28263 ms to 3753 ms; seems working

leshikus · 2022-09-26T21:11:09Z

ASAN aborted after the first testcase of ArrowBasedExecuteTest, the following errors are not related to Calcite as expected. I believe one may continue ASAN enabling after this is fixed.

=================================================================
==377441==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x614000428440 in thread T0:
  object passed to delete has wrong type:
  size of the allocated type:   408 bytes;
  size of the deallocated type: 304 bytes.
    #0 0x7ff4c4b19c65 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:177
    #1 0x562c06d4e764 in query_template(llvm::Module*, unsigned long, bool, bool, QueryMemoryDescriptor const&, ExecutorDeviceType, bool, GpuSharedMemoryContext const&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x2034764)
    #2 0x562c06cef531 in Executor::compileWorkUnit(std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, RelAlgExecutionUnit const&, CompilationOptions const&, ExecutionOptions const&, GpuMgr const*, bool, std::shared_ptr<RowSetMemoryOwner>, unsigned long, signed char, bool, DataProvider*, std::unordered_map<int, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > > > > >&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1fd5531)
    #3 0x562c075d105d in QueryCompilationDescriptor::compile(unsigned long, signed char, bool, RelAlgExecutionUnit const&, std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, ColumnFetcher const&, CompilationOptions const&, ExecutionOptions const&, Executor*) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x28b705d)
    #4 0x562c069aa4e3 in Executor::executeWorkUnitImpl(unsigned long&, bool, bool, std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, RelAlgExecutionUnit const&, CompilationOptions const&, ExecutionOptions const&, std::shared_ptr<RowSetMemoryOwner>, bool, DataProvider*, std::unordered_map<int, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > > > > >&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1c904e3)
    #5 0x562c069afca2 in Executor::executeWorkUnit(unsigned long&, bool, std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, RelAlgExecutionUnit const&, CompilationOptions const&, ExecutionOptions const&, bool, DataProvider*, std::unordered_map<int, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > > > > >&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1c95ca2)
    #6 0x562c06f75ce8 in ExecutionResult RelAlgExecutor::executeWorkUnit(RelAlgExecutor::WorkUnit const&, std::vector<TargetMetaInfo, std::allocator<TargetMetaInfo> > const&, bool, CompilationOptions const&, ExecutionOptions const&, long, std::optional<unsigned long>)::{lambda(auto:1, bool, bool)#1}::operator()<unsigned long>(unsigned long, bool, bool) const (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x225bce8)
    #7 0x562c06fdb9dd in RelAlgExecutor::executeWorkUnit(RelAlgExecutor::WorkUnit const&, std::vector<TargetMetaInfo, std::allocator<TargetMetaInfo> > const&, bool, CompilationOptions const&, ExecutionOptions const&, long, std::optional<unsigned long>) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22c19dd)
    #8 0x562c06fddf5f in RelAlgExecutor::executeCompound(RelCompound const*, CompilationOptions const&, ExecutionOptions const&, long) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22c3f5f)
    #9 0x562c06fe2f5d in RelAlgExecutor::executeRelAlgStep(RaExecutionSequence const&, unsigned long, CompilationOptions const&, ExecutionOptions const&, long) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22c8f5d)
    #10 0x562c06fe5998 in RelAlgExecutor::executeRelAlgSeq(RaExecutionSequence const&, CompilationOptions const&, ExecutionOptions const&, long, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22cb998)
    #11 0x562c06fe875e in RelAlgExecutor::executeRelAlgQueryNoRetry(CompilationOptions const&, ExecutionOptions const&, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22ce75e)
    #12 0x562c06feb15d in RelAlgExecutor::executeRelAlgQuery(CompilationOptions const&, ExecutionOptions const&, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22d115d)
    #13 0x562c0660f6b0 in TestHelpers::ArrowSQLRunner::(anonymous namespace)::ArrowSQLRunnerImpl::runSqlQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CompilationOptions const&, ExecutionOptions const&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18f56b0)
    #14 0x562c06610a69 in TestHelpers::ArrowSQLRunner::(anonymous namespace)::ArrowSQLRunnerImpl::run_multiple_agg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ExecutorDeviceType, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18f6a69)
    #15 0x562c0662154e in TestHelpers::ArrowSQLRunner::run_simple_agg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ExecutorDeviceType, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x190754e)
    #16 0x562c05e7db6a in Distributed50_FailOver_Test::TestBody() (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1163b6a)
    #17 0x562c06608217 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18ee217)
    #18 0x562c065cfab4 in testing::Test::Run() [clone .part.0] (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18b5ab4)
    #19 0x562c065d071c in testing::TestInfo::Run() [clone .part.0] (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18b671c)
    #20 0x562c065d0e99 in testing::TestSuite::Run() [clone .part.0] (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18b6e99)
    #21 0x562c065d46d1 in testing::internal::UnitTestImpl::RunAllTests() (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18ba6d1)
    #22 0x562c065d5342 in testing::UnitTest::Run() (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18bb342)
    #23 0x562c05c9aa87 in main (/omniscidb/build/Tests/ArrowBasedExecuteTest+0xf80a87)
    #24 0x7ff4c07df082 in __libc_start_main ../csu/libc-start.c:308
    #25 0x562c05db7b2d in _start (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x109db2d)

0x614000428440 is located 0 bytes inside of 408-byte region [0x614000428440,0x6140004285d8)
allocated by thread T0 here:
    #0 0x7ff4c4b18587 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:104
    #1 0x562c06d4e993 in query_template(llvm::Module*, unsigned long, bool, bool, QueryMemoryDescriptor const&, ExecutorDeviceType, bool, GpuSharedMemoryContext const&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x2034993)
    #2 0x562c06cef531 in Executor::compileWorkUnit(std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, RelAlgExecutionUnit const&, CompilationOptions const&, ExecutionOptions const&, GpuMgr const*, bool, std::shared_ptr<RowSetMemoryOwner>, unsigned long, signed char, bool, DataProvider*, std::unordered_map<int, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > > > > >&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1fd5531)
    #3 0x562c075d105d in QueryCompilationDescriptor::compile(unsigned long, signed char, bool, RelAlgExecutionUnit const&, std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, ColumnFetcher const&, CompilationOptions const&, ExecutionOptions const&, Executor*) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x28b705d)
    #4 0x562c069aa4e3 in Executor::executeWorkUnitImpl(unsigned long&, bool, bool, std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, RelAlgExecutionUnit const&, CompilationOptions const&, ExecutionOptions const&, std::shared_ptr<RowSetMemoryOwner>, bool, DataProvider*, std::unordered_map<int, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > > > > >&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1c904e3)
    #5 0x562c069afca2 in Executor::executeWorkUnit(unsigned long&, bool, std::vector<InputTableInfo, std::allocator<InputTableInfo> > const&, RelAlgExecutionUnit const&, CompilationOptions const&, ExecutionOptions const&, bool, DataProvider*, std::unordered_map<int, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unordered_map<int, std::shared_ptr<ColumnarResults const>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::shared_ptr<ColumnarResults const> > > > > > >&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1c95ca2)
    #6 0x562c06f75ce8 in ExecutionResult RelAlgExecutor::executeWorkUnit(RelAlgExecutor::WorkUnit const&, std::vector<TargetMetaInfo, std::allocator<TargetMetaInfo> > const&, bool, CompilationOptions const&, ExecutionOptions const&, long, std::optional<unsigned long>)::{lambda(auto:1, bool, bool)#1}::operator()<unsigned long>(unsigned long, bool, bool) const (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x225bce8)
    #7 0x562c06fdb9dd in RelAlgExecutor::executeWorkUnit(RelAlgExecutor::WorkUnit const&, std::vector<TargetMetaInfo, std::allocator<TargetMetaInfo> > const&, bool, CompilationOptions const&, ExecutionOptions const&, long, std::optional<unsigned long>) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22c19dd)
    #8 0x562c06fddf5f in RelAlgExecutor::executeCompound(RelCompound const*, CompilationOptions const&, ExecutionOptions const&, long) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22c3f5f)
    #9 0x562c06fe2f5d in RelAlgExecutor::executeRelAlgStep(RaExecutionSequence const&, unsigned long, CompilationOptions const&, ExecutionOptions const&, long) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22c8f5d)
    #10 0x562c06fe5998 in RelAlgExecutor::executeRelAlgSeq(RaExecutionSequence const&, CompilationOptions const&, ExecutionOptions const&, long, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22cb998)
    #11 0x562c06fe875e in RelAlgExecutor::executeRelAlgQueryNoRetry(CompilationOptions const&, ExecutionOptions const&, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22ce75e)
    #12 0x562c06feb15d in RelAlgExecutor::executeRelAlgQuery(CompilationOptions const&, ExecutionOptions const&, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x22d115d)
    #13 0x562c0660f6b0 in TestHelpers::ArrowSQLRunner::(anonymous namespace)::ArrowSQLRunnerImpl::runSqlQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CompilationOptions const&, ExecutionOptions const&) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18f56b0)
    #14 0x562c06610a69 in TestHelpers::ArrowSQLRunner::(anonymous namespace)::ArrowSQLRunnerImpl::run_multiple_agg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ExecutorDeviceType, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18f6a69)
    #15 0x562c0662154e in TestHelpers::ArrowSQLRunner::run_simple_agg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ExecutorDeviceType, bool) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x190754e)
    #16 0x562c05e7db6a in Distributed50_FailOver_Test::TestBody() (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x1163b6a)
    #17 0x562c06608217 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18ee217)
    #18 0x562c065cfab4 in testing::Test::Run() [clone .part.0] (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18b5ab4)
    #19 0x562c065d071c in testing::TestInfo::Run() [clone .part.0] (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18b671c)
    #20 0x562c065d0e99 in testing::TestSuite::Run() [clone .part.0] (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18b6e99)
    #21 0x562c065d46d1 in testing::internal::UnitTestImpl::RunAllTests() (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18ba6d1)
    #22 0x562c065d5342 in testing::UnitTest::Run() (/omniscidb/build/Tests/ArrowBasedExecuteTest+0x18bb342)
    #23 0x562c05c9aa87 in main (/omniscidb/build/Tests/ArrowBasedExecuteTest+0xf80a87)
    #24 0x7ff4c07df082 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: new-delete-type-mismatch ../../../../src/libsanitizer/asan/asan_new_delete.cc:177 in operator delete(void*, unsigned long)
==377441==HINT: if you don't care about these errors you may set ASAN_OPTIONS=new_delete_type_mismatch=0
==377441==ABORTING

leshikus · 2022-10-04T07:54:05Z

How to reproduce:

git clone https://github.com/intel-ai/omniscidb.git
python3 scripts/conda/make-m2-proxy.py # create proxy if needed

cmake -B ./build -S . -DENABLE_TESTS=on -DENABLE_CUDA=off -DENABLE_ASAN=off # build
cmake --build /omniscidb/build/ --parallel 4

cd build/Tests # build cache
./ArrowBasedExecuteTest --build-rel-alg-cache=cache.txt

cd ../.. # rebuild with ASAN
cmake -B ./build -S . -DENABLE_TESTS=on -DENABLE_CUDA=off -DENABLE_ASAN=on
cmake --build ./build/ --parallel 16

cd build/Tests # run the test with cache and ASAN
./ArrowBasedExecuteTest --use-rel-alg-cache=cache.txt >asan.log

leshikus · 2022-10-05T11:58:59Z

I've rerun the tests with ASAN_OPTIONS="detect_leaks=0" and ArrowBasedExecuteTest excluded. A new memory problem was found in UdfTest.

$ ./UdfTest
[==========] Running 7 tests from 1 test suite.
[----------] Global test environment set-up.
=================================================================
==2209597==ERROR: AddressSanitizer: requested allocation size 0x7fe144de3169 (0x7fe144de4170 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
    #0 0x7fe14e3bc587 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:104
    #1 0x55c9ac652694 in (anonymous namespace)::get_clang_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/omniscidb/build/Tests/UdfTest+0xce1694)
    #2 0x55c9ac6533e6 in UdfCompiler::UdfCompiler(CudaMgr_Namespace::NvidiaDeviceArch, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) (/omniscidb/build/Tests/UdfTest+0xce23e6)
    #3 0x55c9ac543792 in (anonymous namespace)::SQLTestEnv::SetUp() (/omniscidb/build/Tests/UdfTest+0xbd2792)
    #4 0x55c9ac6117ea in testing::internal::UnitTestImpl::RunAllTests() (/omniscidb/build/Tests/UdfTest+0xca07ea)
    #5 0x55c9ac614132 in testing::UnitTest::Run() (/omniscidb/build/Tests/UdfTest+0xca3132)
    #6 0x55c9ac48ae7c in main (/omniscidb/build/Tests/UdfTest+0xb19e7c)
    #7 0x7fe1448cd082 in __libc_start_main ../csu/libc-start.c:308

==2209597==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: allocation-size-too-big ../../../../src/libsanitizer/asan/asan_new_delete.cc:104 in operator new(unsigned long)
==2209597==ABORTING```

The leak detector does not work for most tests which invoke Calcite via JNI and create unreadable log.

leshikus · 2022-10-05T12:16:19Z

The error in UdfTest can be mitigated via adding /usr/lib/llvm-12/bin to PATH; the problem does not happen on the test main execution branch. The change happens due to transition to LLVM 12.

leshikus · 2022-10-05T12:24:42Z

Here is a generic test run log
asan.log

alexbaden · 2022-10-05T12:54:27Z

Ok, thanks! I have an idea of where the first issue is coming from, but fixing it opened up some additional issues. Will look at the UDF test issue afterwards.

leshikus · 2022-10-05T13:04:25Z

TSAN run results
when a test fails due to race condition, tsan hangs, see StringDictionaryTest, ParallelSortTest
Here is the log
tsan.log

alexbaden · 2022-10-08T02:22:13Z

ASAN should be fixed w/ https://github.com/intel-ai/omniscidb/commit/676b290418968a1d4a45b5d54cd5666718ff849e

leshikus · 2022-10-08T10:59:17Z

@alexbaden thanks, now all test cases of ArrowBasedExecuteTest are able to run; there is still a problem with TBB:

=================================================================
==485705==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 24600 byte(s) in 3 object(s) allocated from:
    #0 0x7f50ff502787 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:107
    #1 0x7f50fc9c966f  (/lib/x86_64-linux-gnu/libtbb.so.2+0x2766f)

SUMMARY: AddressSanitizer: 24600 byte(s) leaked in 3 allocation(s).

leshikus · 2022-10-08T11:02:53Z

The guys from TBB blame pthread_key_create() https://community.intel.com/t5/Intel-oneAPI-Threading-Building/TBB-memory-leaks-even-when-entire-program-is-commented-out/m-p/1009653

alexbaden · 2022-10-08T16:20:43Z

@aregm said tbb had improved, so maybe we need to update. We could also add a suppression: https://github.com/intel-ai/omniscidb/blob/jit-engine/config/asan.suppressions for TSAN I would prefer not to do that, but for ASAN it's probably ok.

leshikus · 2022-10-11T16:57:26Z

With exception to TBB and JNI it seems there are no ASAN issues in most tests. The following tests have JNI issues, @ienkovich maybe Ilya's option is needed here:

CorrelatedSubqueryTest
DataRecyclerTest
ArrayTest
TableFunctionsTest
NoCatalogSqlTest
SQLHintTest
TopKTest
ResultSetArrowConversion
ArrowStorageSqlTest
StringFunctionsTest
UdfTest
ResultSetTest
GroupByTest
JoinHashTableTest
BumpAllocatorTest

alexbaden · 2022-10-18T20:37:12Z

Let's split ASAN into two remaining tasks:

TBB -- can the issues be rectified by either building TBB from source with ASAN enabled, or by adding a suppression?
Let's ignore the remaining tests in favor of getting ArrowBasedExecuteTest working, and go from there. Likely the remaining tests have JNI issues.

leshikus · 2022-10-19T20:41:43Z

I'm ok with both solutions;
I will try to setup CI for ArrowBasedExecuteTest separately

leshikus · 2022-10-19T21:26:34Z

I've heard there are plans to merge hdk and omniscidb workspaces; better to do 2 after workspaces are merged

alexbaden · 2022-10-19T21:42:10Z

There are plans to do that, but I don't know that it will happen in the next few weeks.

leshikus · 2022-11-11T03:57:11Z

Here is a PR which is required to add tests to HDK build, https://github.com/intel-ai/omniscidb/pull/581

leshikus · 2022-11-18T22:25:08Z

When running tests in HDK, I get multiple problems, see the log below. Some of them are related to function get_root_abs_path

LastTest.log

leshikus · 2022-11-22T14:21:11Z

The implementation of get_root_abs_path differs because BUILD_SHARED_LIBS and ENABLE_SHARED_LIBS have different default values for omniscidb and hdk. I wonder if this is intentional. Should get_root_abs_path point to omniscidb dir or hdk dir?

leshikus · 2022-11-22T14:31:06Z

The following patch seems to fix the issue

diff --git a/OSDependent/Unix/omnisci_path.cpp b/OSDependent/Unix/omnisci_path.cpp
index 0a113fe53..40f98729e 100644
--- a/OSDependent/Unix/omnisci_path.cpp
+++ b/OSDependent/Unix/omnisci_path.cpp
@@ -52,9 +52,10 @@ std::string get_root_abs_path() {
     /* Despite the dlinfo man page claim that l_name is absolute path,
        it is so only when the location path to the library is absolute,
        say, as specified in LD_LIBRARY_PATH. */
+
     std::filesystem::path abs_exe_dir(std::filesystem::absolute(
         std::filesystem::canonical(std::string(link_map->l_name))));
-    const auto mapd_root = abs_exe_dir.parent_path().parent_path();
+    const auto mapd_root = abs_exe_dir.parent_path().parent_path().parent_path();
     return mapd_root.string();
   }
 #endif

leshikus · 2022-11-22T14:38:34Z

The next problem is at

  void createCalciteServerHandler(JNIEnv* env, const std::string& udf_filename) {
    jclass handler_cls = findClass(env, "com/mapd/parser/server/CalciteServerHandler");

I need to correct the class path somehow

Upd. This patch fixes the class path

diff --git a/Calcite/CalciteJNI.cpp b/Calcite/CalciteJNI.cpp
index a98c3787c..c99f5c704 100644
--- a/Calcite/CalciteJNI.cpp
+++ b/Calcite/CalciteJNI.cpp
@@ -119,7 +119,7 @@ class JVM {
   static std::shared_ptr<JVM> createJVM(size_t max_mem_mb) {
     auto root_abs_path = omnisci::get_root_abs_path();
     std::string class_path_arg = "-Djava.class.path=" + root_abs_path +
-                                 "/bin/calcite-1.0-SNAPSHOT-jar-with-dependencies.jar";
+                                 "/../bin/calcite-1.0-SNAPSHOT-jar-with-dependencies.jar";
     std::string max_mem_arg = "-Xmx" + std::to_string(max_mem_mb) + "m";
     JavaVMInitArgs vm_args;
     auto options = std::make_unique<JavaVMOption[]>(2);

leshikus · 2022-11-23T18:55:40Z

The problem with paths is moved to #120

leshikus · 2022-11-28T18:19:12Z

It was good @Garra1980 suggested to try launching ASAN with omniscidb. Another problem manifested itself - Github virtual machines do not have enough virtual memory for ASAN to start due to their virtualization settings.

==12699==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v

leshikus · 2022-11-30T17:29:10Z

I have tried a small test and it worked; is it possible to scale a data size via option for ArrowBasedExecuteTest?

https://github.com/leshikus/ghtest/actions/runs/3585577524/jobs/6033734506

leshikus · 2023-11-27T15:06:39Z

ASAN works now

alexbaden added the tests label Jun 7, 2022

alexbaden mentioned this issue Jun 7, 2022

Add TSAN, ASAN, UBSAN to CI #35

Closed

leshikus self-assigned this Jun 23, 2022

leshikus closed this as completed Nov 27, 2023

Sanitizers do not work with JNI #34

Sanitizers do not work with JNI #34

Comments

alexbaden commented Jun 7, 2022

leshikus commented Jun 24, 2022 • edited

Garra1980 commented Jun 24, 2022

leshikus commented Jun 27, 2022

alexbaden commented Jun 27, 2022

ienkovich commented Jun 27, 2022

leshikus commented Jun 28, 2022

leshikus commented Jul 1, 2022

leshikus commented Jul 1, 2022

alexbaden commented Jul 1, 2022

leshikus commented Jul 1, 2022

alexbaden commented Jul 1, 2022

leshikus commented Jul 1, 2022

leshikus commented Jul 5, 2022

ienkovich commented Sep 13, 2022

leshikus commented Sep 26, 2022

leshikus commented Sep 26, 2022

leshikus commented Oct 4, 2022 • edited

leshikus commented Oct 5, 2022 • edited

leshikus commented Oct 5, 2022

leshikus commented Oct 5, 2022

alexbaden commented Oct 5, 2022

leshikus commented Oct 5, 2022

alexbaden commented Oct 8, 2022

leshikus commented Oct 8, 2022

leshikus commented Oct 8, 2022

alexbaden commented Oct 8, 2022

leshikus commented Oct 11, 2022

alexbaden commented Oct 18, 2022

leshikus commented Oct 19, 2022

leshikus commented Oct 19, 2022

alexbaden commented Oct 19, 2022

leshikus commented Nov 11, 2022

leshikus commented Nov 18, 2022

leshikus commented Nov 22, 2022 • edited

leshikus commented Nov 22, 2022

leshikus commented Nov 22, 2022 • edited

leshikus commented Nov 23, 2022

leshikus commented Nov 28, 2022

leshikus commented Nov 30, 2022

leshikus commented Nov 27, 2023

leshikus commented Jun 24, 2022 •

edited

leshikus commented Oct 4, 2022 •

edited

leshikus commented Oct 5, 2022 •

edited

leshikus commented Nov 22, 2022 •

edited

leshikus commented Nov 22, 2022 •

edited