[CH] Crashes caused by abnormal task cancellation.

[lgbo-ustc]

Backend

CH (ClickHouse)

Bug description

The abnormal termination of a task often results in a crash.

Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

[lgbo-ustc]

After the native destructor (Java_org_apache_gluten_vectorized_ExpressionEvaluatorJniWrapper_nativeDestroyNative) was called, ClickHouse code continued to be invoked. This access to freed resources resulted in an invalid memory access and a crash.

It has been observed that after the execution of Java_org_apache_gluten_vectorized_ExpressionEvaluatorJniWrapper_nativeDestroyNative, the following issues persist:

1. There are still active LocalExecutor instances. Forcibly shutting down these LocalExecutor instances at this point would be an appropriate action.
2. Other JNI interfaces (e.g., Java_org_apache_gluten_vectorized_BatchIterator_nativeFetchMetrics) may still be invoked by the Java side. A mechanism should be implemented to ​​ensure​​ that Java_org_apache_gluten_vectorized_ExpressionEvaluatorJniWrapper_nativeDestroyNative is only executed when no other JNI interfaces are active. Concurrently, any subsequent calls to other JNI interfaces must be directly rejected.

[lgbo-ustc]

A crash stack

[Current thread is 1 (Thread 0x7fd23199e700 (LWP 242449))]
(gdb) bt
#0  0x00007fd25ccbe428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd25ccc002a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fd2598406b5 in os::abort(bool) () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#3  0x00007fd2599debf3 in VMError::report_and_die() () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#4  0x00007fd259845edf in JVM_handle_linux_signal () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#5  0x00007fd25983c673 in signalHandler(int, siginfo*, void*) ()
   from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x00007fd26c5f9a79 in DB::ThreadStatus::flushUntrackedMemory (this=0x7fd205ef9940)
    at ./build/./src/Common/ThreadStatus.cpp:244
#8  0x00007fd270e7f441 in DB::ThreadStatus::detachFromGroup (this=0x7fd205ef9940)
    at ./build/./src/Interpreters/ThreadStatusExt.cpp:372
#9  0x00007fd26c5f9be6 in std::__1::__function::__policy_func<void ()>::operator()[abi:ne190107]() const (
    this=0x7fd205ef9d48) at ./contrib/llvm-project/libcxx/include/__functional/function.h:716
#10 std::__1::function<void ()>::operator()() const (this=0x7fd205ef9d48)
    at ./contrib/llvm-project/libcxx/include/__functional/function.h:989
#11 DB::ThreadStatus::~ThreadStatus (this=0x7fd205ef9940) at ./build/./src/Common/ThreadStatus.cpp:277
#12 0x00007fd26cb284b6 in std::__1::__shared_count::__release_shared[abi:ne190107]() (this=<optimized out>)
    at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:156
#13 std::__1::__shared_weak_count::__release_shared[abi:ne190107]() (this=<optimized out>)
    at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:185
#14 std::__1::shared_ptr<DB::ThreadStatus>::~shared_ptr[abi:ne190107]() (this=<optimized out>)
    at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:668
#15 local_engine::QueryContext::Data::~Data (this=<optimized out>)
    at ./build/./utils/extern-local-engine/Common/QueryContext.cpp:49
#16 _ZNSt3__112__destroy_atB8ne190107IN12local_engine12QueryContext4DataETnNS_9enable_ifIXntsr8is_arrayIT_EE5valueEiE4typeELi0EEEvPS5_ (__loc=<optimized out>) at ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:67
#17 0x00007fd26cb25971 in std::__1::__shared_count::__release_shared[abi:ne190107]() (this=0x7fd24252c9e0)
    at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:156
#18 std::__1::__shared_weak_count::__release_shared[abi:ne190107]() (this=0x7fd24252c9e0)
    at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:185
#19 std::__1::shared_ptr<local_engine::QueryContext::Data>::~shared_ptr[abi:ne190107]() (this=0x7fd2425453f8)
    at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:668
#20 std::__1::pair<long const, std::__1::shared_ptr<local_engine::QueryContext::Data> >::~pair (this=0x7fd2425453f0)
    at ./contrib/llvm-project/libcxx/include/__utility/pair.h:65
#21 _ZNSt3__112__destroy_atB8ne190107INS_4pairIKlNS_10shared_ptrIN12local_engine12QueryContext4DataEEEEETnNS_9enable_ifIXntsr8is_arrayIT_EE5valueEiE4typeELi0EEEvPSA_ (__loc=0x7fd2425453f0)
--Type <RET> for more, q to quit, c to continue without paging--
    at ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:67
#22 _ZNSt3__116allocator_traitsINS_9allocatorINS_11__hash_nodeINS_17__hash_value_typeIlNS_10shared_ptrIN12local_engine12QueryContext4DataEEEEEPvEEEEE7destroyB8ne190107INS_4pairIKlS8_EEvTnNS_9enable_ifIXntsr13__has_destroyISC_PT_EE5valueEiE4typeELi0EEEvRSC_SK_ (__p=0x7fd2425453f0) at ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:339
#23 std::__1::__hash_table<std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >, std::__1::__unordered_map_hasher<long, std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >, std::__1::hash<long>, std::__1::equal_to<long>, true>, std::__1::__unordered_map_equal<long, std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >, std::__1::equal_to<long>, std::__1::hash<long>, true>, std::__1::allocator<std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> > > >::__deallocate_node (this=0x7fd279848d28 <local_engine::QueryContext::instance()::instance+16>,
    __np=0x7fd2425453e0) at ./contrib/llvm-project/libcxx/include/__hash_table:1139
#24 std::__1::__hash_table<std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >, std::__1::__unordered_map_hasher<long, std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >, std::__1::hash<long>, std::__1::equal_to<long>, true>, std::__1::__unordered_map_equal<long, std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >, std::__1::equal_to<long>, std::__1::hash<long>, true>, std::__1::allocator<std::__1::__hash_value_type<long, std::__1::shared_ptr<local_engine::QueryContext::Data> > > >::~__hash_table (this=0x7fd279848d28 <local_engine::QueryContext::instance()::instance+16>)
    at ./contrib/llvm-project/libcxx/include/__hash_table:1107
#25 std::__1::unordered_map<long, std::__1::shared_ptr<local_engine::QueryContext::Data>, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, std::__1::shared_ptr<local_engine::QueryContext::Data> > > >::~unordered_map[abi:ne190107]() (this=0x7fd279848d28 <local_engine::QueryContext::instance()::instance+16>)
    at ./contrib/llvm-project/libcxx/include/unordered_map:1178
#26 local_engine::ConcurrentMap<long, std::__1::shared_ptr<local_engine::QueryContext::Data> >::~ConcurrentMap (
    this=0x7fd279848d28 <local_engine::QueryContext::instance()::instance+16>)
    at ./utils/extern-local-engine/Common/ConcurrentMap.h:25
#27 local_engine::QueryContext::~QueryContext (this=0x7fd279848d18 <local_engine::QueryContext::instance()::instance>)
    at ./build/./utils/extern-local-engine/Common/QueryContext.cpp:136
#28 0x00007fd25ccc2ff8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#29 0x00007fd25ccc3045 in exit () from /lib/x86_64-linux-gnu/libc.so.6
#30 0x00007fd2595ad6d7 in vm_direct_exit(int) () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#31 0x00007fd2599e7c35 in VM_Operation::evaluate() () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#32 0x00007fd2599e600a in VMThread::evaluate_operation(VM_Operation*) ()
   from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#33 0x00007fd2599e638e in VMThread::loop() () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#34 0x00007fd2599e6800 in VMThread::run() () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#35 0x00007fd259841b88 in java_start(Thread*) () from /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
#36 0x00007fd25d4746ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#37 0x00007fd25cd9041d in clone () from /lib/x86_64-linux-gnu/libc.so.6

[lgbo-ustc]

Another crash stack

MemoryTracker::alloc called after ProfileEvents finalized.
 0# 0x00007F7CEA4EDF57 in ./libch.so
 1# 0x00007F7CEA527D5A in ./libch.so
 2# 0x00007F7CEA4417B1 in ./libch.so
 3# 0x00007F7CEA43E2F2 in ./libch.so
 4# 0x00007F7CF6EB283E in ./libch.so
 5# 0x00007F7CDB3A1439 in /lib/x86_64-linux-gnu/libpthread.so.0
 6# 0x00007F7CDB3A2870 in /lib/x86_64-linux-gnu/libpthread.so.0
 7# clone in /lib/x86_64-linux-gnu/libc.so.6

MemoryTracker::allocImpl(long, bool, MemoryTracker*, double)
DB::ThreadStatus::flushUntrackedMemory()
CurrentMemoryTracker::free(long)
operator delete(void*, unsigned long)
__cxxabiv1::(anonymous namespace)::run_dtors(void*)

This is a thread status created inside Java_org_apache_gluten_vectorized_ExpressionEvaluatorJniWrapper_nativeDestroyNative

ThreadStatus 0x7f4bbaf44340 for thread 41038
 0# 0x00007F4C258BB8CB in ./libch.so
 1# Java_org_apache_gluten_vectorized_ExpressionEvaluatorJniWrapper_nativeDestroyNative in ./libch.so


~ThreadStatus 0x7f4bbaf44340 thread id 41038
MemoryTracker::alloc called after ProfileEvents finalized.
 0# 0x00007F4C25883217 in ./libch.so
 1# 0x00007F4C258BD19A in ./libch.so
 2# 0x00007F4C257D6A71 in ./libch.so
 3# 0x00007F4C257D35B2 in ./libch.so
 4# 0x00007F4C3224783E in ./libch.so
 5# 0x00007F4C16736439 in /lib/x86_64-linux-gnu/libpthread.so.0
 6# 0x00007F4C16737870 in /lib/x86_64-linux-gnu/libpthread.so.0
 7# clone in /lib/x86_64-linux-gnu/libc.so.6