-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
修改为maven就好了 #5
Comments
为什么选用 ant,而不是 maven 和 graddle?考虑如下:
我们在选用一些技术时,做了自己的一些考虑,不是带来实际收益,我们一般不会考虑引入。这里还是追求简洁有效。 |
morningman
referenced
this issue
in morningman/doris
Aug 5, 2020
Check all rest apis and implement docs
Closed
3 tasks
3 tasks
morrySnow
pushed a commit
to morrySnow/incubator-doris
that referenced
this issue
Mar 16, 2022
merge apache master
This was referenced Mar 30, 2022
3 tasks
3 tasks
morningman
referenced
this issue
in morningman/doris
May 4, 2022
``` ==12681==ERROR: LeakSanitizer: detected memory leaks Direct leak of 128 byte(s) in 1 object(s) allocated from: #0 0x562b93d0fc57 in operator new(unsigned long) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x16e02c57) #1 0x562b955fd176 in doris::ArrowWorkFlowTest::init_runtime_state() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:92 #2 0x562b955fcf58 in doris::ArrowWorkFlowTest::init() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:87 #3 0x562b9560ba7d in doris::ArrowWorkFlowTest::SetUp() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:61 #4 0x562b9f38263c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #5 0x562b9f37ca78 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #6 0x562b9f3540e1 in testing::Test::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224470e1) apache#7 0x562b9f354b7f in testing::TestInfo::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22447b7f) apache#8 0x562b9f35543f in testing::TestSuite::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x2244843f) apache#9 0x562b9f364aff in testing::internal::UnitTestImpl::RunAllTests() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22457aff) apache#10 0x562b9f3833ed in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (Impl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224763ed) apache#11 0x562b9f37d932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), e/ut_build_ASAN/test/doris_be_test+0x22470932) apache#12 0x562b9f3632f4 in testing::UnitTest::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224562f4) apache#13 0x562b95369a7c in RUN_ALL_TESTS() /var/local/thirdparty/installed/include/gtest/gtest.h:2490 apache#14 0x562b95366048 in main /root/doris/be/test/testutil/run_all_tests.cpp:43 apache#15 0x7f40dcbb6554 in __libc_start_main (/lib64/libc.so.6+0x22554) Direct leak of 104 byte(s) in 1 object(s) allocated from: #0 0x562b93d0fc57 in operator new(unsigned long) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x16e02c57) #1 0x562b955fd213 in doris::ArrowWorkFlowTest::init_runtime_state() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:93 #2 0x562b955fcf58 in doris::ArrowWorkFlowTest::init() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:87 #3 0x562b9560ba7d in doris::ArrowWorkFlowTest::SetUp() /root/doris/be/test/util/arrow/arrow_work_flow_test.cpp:61 #4 0x562b9f38263c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #5 0x562b9f37ca78 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/ #6 0x562b9f3540e1 in testing::Test::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224470e1) apache#7 0x562b9f354b7f in testing::TestInfo::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22447b7f) apache#8 0x562b9f35543f in testing::TestSuite::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x2244843f) apache#9 0x562b9f364aff in testing::internal::UnitTestImpl::RunAllTests() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x22457aff) apache#10 0x562b9f3833ed in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (Impl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224763ed) apache#11 0x562b9f37d932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), e/ut_build_ASAN/test/doris_be_test+0x22470932) apache#12 0x562b9f3632f4 in testing::UnitTest::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x224562f4) apache#13 0x562b95369a7c in RUN_ALL_TESTS() /var/local/thirdparty/installed/include/gtest/gtest.h:2490 apache#14 0x562b95366048 in main /root/doris/be/test/testutil/run_all_tests.cpp:43 apache#15 0x7f40dcbb6554 in __libc_start_main (/lib64/libc.so.6+0x22554) SUMMARY: AddressSanitizer: 232 byte(s) leaked in 2 allocation(s). ```
3 tasks
3 tasks
dataroaring
pushed a commit
that referenced
this issue
Jun 7, 2024
…#35458) ## Proposed changes Move memtable core when use multi table load: ``` 0x51f000c73860 is located 3040 bytes inside of 3456-byte region [0x51f000c72c80,0x51f000c73a00) freed by thread T4867 (FragmentMgrThre) here: #0 0x558f6ad7f43d in operator delete(void*) (/mnt/hdd01/STRESS_ENV/be/lib/doris_be+0x22eec43d) (BuildId: b46f73d1f76dfcd6) #1 0x558f6e6cea2c in std::__new_allocator<doris::PTabletID>::deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:168:2 #2 0x558f6e6ce9e7 in std::allocator<doris::PTabletID>::deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:210:25 #3 0x558f6e6ce9e7 in std::allocator_traits<std::allocator<doris::PTabletID>>::deallocate(std::allocator<doris::PTabletID>&, doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:516:13 #4 0x558f6e6ce9e7 in std::_Vector_base<doris::PTabletID, std::allocator<doris::PTabletID>>::_M_deallocate(doris::PTabletID*, unsigned long) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_vector.h:387:4 #5 0x558f6e6d0780 in void std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>::_M_range_insert<__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>>(__gnu_cxx::__normal_iterator<doris::PTabletID*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, std::forward_iterator_tag) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/vector.tcc:832:3 #6 0x558f6e6c54c5 in __gnu_cxx::__normal_iterator<doris::PTabletID*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>> std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>::insert<__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, void>(__gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>, __gnu_cxx::__normal_iterator<doris::PTabletID const*, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>>>) /mnt/disk2/xujianxu/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_vector.h:1483:4 #7 0x558f9b4b214f in doris::LoadStreamMap::save_tablets_to_commit(long, std::vector<doris::PTabletID, std::allocator<doris::PTabletID>> const&) /mnt/disk2/xujianxu/doris/be/src/vec/sink/load_stream_map_pool.cpp:90:13 #8 0x558f9b7258dd in doris::vectorized::VTabletWriterV2::_calc_tablets_to_commit() /mnt/disk2/xujianxu/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:650:27 #9 0x558f9b7229f1 in doris::vectorized::VTabletWriterV2::close(doris::Status) /mnt/disk2/xujianxu/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:547:9 ``` Multiple sinks with different table loads use the load id, causing confusion in the use of shared data structures between sinks.
dataroaring
pushed a commit
that referenced
this issue
Jun 13, 2024
## Proposed changes If group commit time interval is larger than the load timeout, and there is no new client load to reuse the internal group commit load, the group commit can not cancel in time because it stuck in wait: ``` #0 0x00007f33937a47aa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005651105dbd05 in __gthread_cond_timedwait(pthread_cond_t*, pthread_mutex_t*, timespec const*) () #2 0x000056511063f385 in std::__condvar::wait_until(std::mutex&, timespec&) () #3 0x000056511063dc2e in std::cv_status std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #4 0x000056511063cedf in std::cv_status std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #5 0x0000565110824f48 in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l> >(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&) () #6 0x0000565113b5612a in doris::LoadBlockQueue::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*, bool*) () #7 0x000056513f900941 in doris::pipeline::GroupCommitOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) () #8 0x000056513c69c0b6 in doris::pipeline::ScanOperatorX<doris::pipeline::GroupCommitLocalState>::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) () #9 0x000056514009d5f1 in doris::pipeline::PipelineTask::execute(bool*) () #10 0x00005651400fb24a in doris::pipeline::TaskScheduler::_do_work(unsigned long) () ```
jacktengg
added a commit
that referenced
this issue
Jun 14, 2024
…t is not nullable (#36263) ## Proposed changes For a null safe equal join, FE may generate a plan that both sides of the conjuct are not nullable, which is a valid plan, BE will coredump in this situation. Issue Number: close #xxx Coredump stacktrace: ``` #0 0x00007fff042caacf in raise () from /lib64/libc.so.6 #1 0x00007fff0429dea5 in abort () from /lib64/libc.so.6 #2 0x00005555a7ec7cad in ?? () #3 0x00005555a7eba34a in google::LogMessage::Fail() () #4 0x00005555a7ebd436 in google::LogMessage::SendToLog() () #5 0x00005555a7eb9e80 in google::LogMessage::Flush() () #6 0x00005555a7ebdc79 in google::LogMessageFatal::~LogMessageFatal() () #7 0x000055557824c9ac in doris::vectorized::IColumn::get_raw_data (this=0x60400387df90) at /mnt/disk2/tengjianping/doris-39/be/src/vec/columns/column.h:590 #8 0x00005555a4a85f32 in doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>::pack_fixeds<unsigned long> (this=0x6110086dcb50, row_numbers=21, key_columns=std::vector of length 1, capacity 1 = {...}, nullmap_columns=std::vector of length 0, capacity 0, result=std::vector of length 21, capacity 21 = {...}) at /mnt/disk2/tengjianping/doris-39/be/src/vec/common/hash_table/hash_map_context.h:405 #9 0x00005555a4a83a77 in doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>::init_serialized_keys (this=0x6110086dcb50, key_columns=std::vector of length 1, capacity 1 = {...}, num_rows=21, null_map=0x0, is_join=true, is_build=true, bucket_size=32) at /mnt/disk2/tengjianping/doris-39/be/src/vec/common/hash_table/hash_map_context.h:470 #10 0x00005555a48d9228 in doris::pipeline::ProcessHashTableBuild<doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false> >::run<1, false, false, false> (this=0x75d6195d92a0, hash_table_ctx=..., null_map=0x0, has_null_key=0x6120057db3c0) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.h:208 #11 0x00005555a456dd23 in doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeS tate*, doris::vectorized::Block&)::$_1::operator()<doris::vectorized::MethodKeysFixed<doris::JoinHashTable< unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<do--Type <RET> for more, q to qu --Type <RET> for more, q to quit, c to continue without paging--c ris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>) const (this=0x75d61989aa10, arg=..., join_op=..., has_null_value=..., short_circuit_for_null_in_build_side=..., with_other_conjuncts=...) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:296 #12 0x00005555a456d93c in std::__invoke_impl<doris::Status, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(std::__invoke_other, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) (__f=..., __args=..., __args=..., __args=..., __args=..., __args=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 #13 0x00005555a456d88c in std::__invoke<doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) (__fn=..., __args=..., __args=..., __args=..., __args=..., __args=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96 #14 0x00005555a43eef75 in _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIN5doris6StatusEEEONS4_10vectorized8OverloadIJZNS4_8pipeline27HashJoinBuildSinkLocalState19process_build_blockEPNS4_12RuntimeStateERNS7_5BlockEE3$_0ZNSA_19process_build_blockESC_SE_E3$_1EEERSt7variantIJSt9monostateNS7_16MethodSerializedINS4_13JoinHashTableINS4_9StringRefE11DefaultHashISN_vEEEEENS7_15MethodOneNumberIhNSM_Ih9HashCRC32IhEEEEENSS_ItNSM_ItST_ItEEEEENSS_IjNSM_IjST_IjEEEEENSS_ImNSM_ImST_ImEEEEENSS_IN4wide7integerILm128EjEENSM_IS18_ST_IS18_EEEEENSS_INS17_ILm256EjEENSM_IS1C_ST_IS1C_EEEEENS7_15MethodKeysFixedIS14_Lb1EEENS1G_IS14_Lb0EEENS1G_IS1A_Lb1EEENS1G_IS1A_Lb0EEENS1G_IS1E_Lb1EEENS1G_IS1E_Lb0EEENS1G_INSM_INS7_7UInt136EST_IS1N_EEELb1EEENS1G_IS1P_Lb0EEEEERSJ_IJSt17integral_constantINS4_7TJoinOp4typeELS1W_0EES1U_IS1W_LS1W_2EES1U_IS1W_LS1W_8EES1U_IS1W_LS1W_1EES1U_IS1W_LS1W_4EES1U_IS1W_LS1W_3EES1U_IS1W_LS1W_5EES1U_IS1W_LS1W_7EES1U_IS1W_LS1W_9EES1U_IS1W_LS1W_10EES1U_IS1W_LS1W_11EEEEOSJ_IJS1U_IbLb0EES1U_IbLb1EEEES2D_S2D_EJEEESt16integer_sequenceImJLm9ELm3ELm0ELm0ELm0EEEE14__visit_invokeESI_S1T_S29_S2D_S2D_S2D_ (__visitor=..., __vars=..., __vars=..., __vars=..., __vars=..., __vars=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013 #15 0x00005555a43cf068 in std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> > >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&) (__visitor=..., __variants=..., __variants=..., __variants=..., __variants=..., __variants=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714 #16 0x00005555a43c8dea in std::visit<doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> > >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&) (__visitor=..., __variants=..., __variants=..., __variants=..., __variants=..., __variants=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769 #17 0x00005555a43c848d in doris::pipeline::HashJoinBuildSinkLocalState::process_build_block (this=0x616000669080, state=0x61e003eca080, block=...) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:278 #18 0x00005555a43cd0a2 in doris::pipeline::HashJoinBuildSinkOperatorX::sink (this=0x613001702bc0, state=0x61e003eca080, in_block=0x60d0000f7d40, eos=true) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:557 #19 0x00005555a7bdd4e1 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const (this=0x75d619925bd0) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/pipeline_task.cpp:356 #20 0x00005555a7bda84c in doris::pipeline::PipelineTask::execute (this=0x61700073d800, eos=0x75d6198f4230) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/pipeline_task.cpp:359 #21 0x00005555a7c34c6a in doris::pipeline::TaskScheduler::_do_work (this=0x60b00226b9e0, index=10) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/task_scheduler.cpp:135 #22 0x00005555a7c374bb in doris::pipeline::TaskScheduler::start()::$_0::operator()() const (this=0x6060055527f8) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/task_scheduler.cpp:64 ```
dataroaring
pushed a commit
that referenced
this issue
Jun 17, 2024
## Proposed changes If group commit time interval is larger than the load timeout, and there is no new client load to reuse the internal group commit load, the group commit can not cancel in time because it stuck in wait: ``` #0 0x00007f33937a47aa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005651105dbd05 in __gthread_cond_timedwait(pthread_cond_t*, pthread_mutex_t*, timespec const*) () #2 0x000056511063f385 in std::__condvar::wait_until(std::mutex&, timespec&) () #3 0x000056511063dc2e in std::cv_status std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #4 0x000056511063cedf in std::cv_status std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) () #5 0x0000565110824f48 in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l> >(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&) () #6 0x0000565113b5612a in doris::LoadBlockQueue::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*, bool*) () #7 0x000056513f900941 in doris::pipeline::GroupCommitOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) () #8 0x000056513c69c0b6 in doris::pipeline::ScanOperatorX<doris::pipeline::GroupCommitLocalState>::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) () #9 0x000056514009d5f1 in doris::pipeline::PipelineTask::execute(bool*) () #10 0x00005651400fb24a in doris::pipeline::TaskScheduler::_do_work(unsigned long) () ```
eldenmoon
pushed a commit
that referenced
this issue
Jun 18, 2024
If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) #0 0x00007fa0bf784387 in ?? () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libc.so.6 #1 0x0000557dc8e617a0 in google::LogMessage::Flush() () #2 0x0000557dc8e65599 in google::LogMessageFatal::~LogMessageFatal() () #3 0x0000557dbe4ec939 in doris::vectorized::IColumn::get_shrinked_column (this=0x7fa094ac16a0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column.h:134 #4 0x0000557dc236d8a2 in doris::vectorized::Block::shrink_char_type_column_suffix_zero (this=0x7f9a372a4b70, char_type_idx=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 #5 0x0000557dbee9ea95 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=<optimized out>, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 #6 0x0000557dbee9cc5a in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 #7 doris::segment_v2::SegmentIterator::next_batch (this=0x1c5d, block=0x6) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2089 #8 0x0000557dbeebbb44 in doris::BetaRowsetReader::next_block (this=0x7f9a3a041600, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 #9 0x0000557dc8590ddd in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 #10 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 #11 0x0000557dc8591055 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref (this=0x1c5d) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 #12 0x0000557dc85934f2 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref (this=0x7fa090e36c00) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 #13 0x0000557dc858ddc9 in doris::vectorized::VCollectIterator::build_heap (this=0x7f9d80976dc0, rs_readers=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 #14 0x0000557dc857e808 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f9d80976800, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 #15 0x0000557dc857f64d in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 #16 0x0000557dc370dc2c in doris::vectorized::NewOlapScanner::open (this=0x7f9a2aa68210, state=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 #17 0x0000557dc3718d89 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=std::shared_ptr<doris::vectorized::ScannerContext> (use count 4, weak count 1) = {...}, scan_task=std::shared_ptr<doris::vectorized::ScanTask> (use count 2, weak count 0) = {...}) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:242 #18 0x0000557dc3719b77 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const (this=0x0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:158 #19 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 #20 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 #21 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 #22 0x0000557dbf3c7788 in doris::ThreadPool::dispatch_thread (this=0x7f9d0bbdc400) at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:543 #23 0x0000557dbf3bcb41 in std::function<void ()>::operator()() const (this=0x1471) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 #24 doris::Thread::supervise_thread (arg=0x7f9d0bbdf4c0) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:498 #25 0x00007fa0bee1dea5 in sem_open () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libpthread.so.0 ```
airborne12
added a commit
to airborne12/apache-doris
that referenced
this issue
Jun 18, 2024
…6275) If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) #0 0x00007fa0bf784387 in ?? () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libc.so.6 #1 0x0000557dc8e617a0 in google::LogMessage::Flush() () #2 0x0000557dc8e65599 in google::LogMessageFatal::~LogMessageFatal() () apache#3 0x0000557dbe4ec939 in doris::vectorized::IColumn::get_shrinked_column (this=0x7fa094ac16a0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column.h:134 apache#4 0x0000557dc236d8a2 in doris::vectorized::Block::shrink_char_type_column_suffix_zero (this=0x7f9a372a4b70, char_type_idx=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 apache#5 0x0000557dbee9ea95 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=<optimized out>, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 apache#6 0x0000557dbee9cc5a in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 apache#7 doris::segment_v2::SegmentIterator::next_batch (this=0x1c5d, block=0x6) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2089 apache#8 0x0000557dbeebbb44 in doris::BetaRowsetReader::next_block (this=0x7f9a3a041600, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 apache#9 0x0000557dc8590ddd in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 apache#10 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 apache#11 0x0000557dc8591055 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref (this=0x1c5d) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 apache#12 0x0000557dc85934f2 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref (this=0x7fa090e36c00) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 apache#13 0x0000557dc858ddc9 in doris::vectorized::VCollectIterator::build_heap (this=0x7f9d80976dc0, rs_readers=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 apache#14 0x0000557dc857e808 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f9d80976800, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 apache#15 0x0000557dc857f64d in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 apache#16 0x0000557dc370dc2c in doris::vectorized::NewOlapScanner::open (this=0x7f9a2aa68210, state=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 apache#17 0x0000557dc3718d89 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=std::shared_ptr<doris::vectorized::ScannerContext> (use count 4, weak count 1) = {...}, scan_task=std::shared_ptr<doris::vectorized::ScanTask> (use count 2, weak count 0) = {...}) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:242 apache#18 0x0000557dc3719b77 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const (this=0x0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:158 apache#19 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#20 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 apache#21 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 apache#22 0x0000557dbf3c7788 in doris::ThreadPool::dispatch_thread (this=0x7f9d0bbdc400) at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:543 apache#23 0x0000557dbf3bcb41 in std::function<void ()>::operator()() const (this=0x1471) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 apache#24 doris::Thread::supervise_thread (arg=0x7f9d0bbdf4c0) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:498 apache#25 0x00007fa0bee1dea5 in sem_open () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libpthread.so.0 ```
dataroaring
pushed a commit
that referenced
this issue
Jun 21, 2024
…t is not nullable (#36263) ## Proposed changes For a null safe equal join, FE may generate a plan that both sides of the conjuct are not nullable, which is a valid plan, BE will coredump in this situation. Issue Number: close #xxx Coredump stacktrace: ``` #0 0x00007fff042caacf in raise () from /lib64/libc.so.6 #1 0x00007fff0429dea5 in abort () from /lib64/libc.so.6 #2 0x00005555a7ec7cad in ?? () #3 0x00005555a7eba34a in google::LogMessage::Fail() () #4 0x00005555a7ebd436 in google::LogMessage::SendToLog() () #5 0x00005555a7eb9e80 in google::LogMessage::Flush() () #6 0x00005555a7ebdc79 in google::LogMessageFatal::~LogMessageFatal() () #7 0x000055557824c9ac in doris::vectorized::IColumn::get_raw_data (this=0x60400387df90) at /mnt/disk2/tengjianping/doris-39/be/src/vec/columns/column.h:590 #8 0x00005555a4a85f32 in doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>::pack_fixeds<unsigned long> (this=0x6110086dcb50, row_numbers=21, key_columns=std::vector of length 1, capacity 1 = {...}, nullmap_columns=std::vector of length 0, capacity 0, result=std::vector of length 21, capacity 21 = {...}) at /mnt/disk2/tengjianping/doris-39/be/src/vec/common/hash_table/hash_map_context.h:405 #9 0x00005555a4a83a77 in doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>::init_serialized_keys (this=0x6110086dcb50, key_columns=std::vector of length 1, capacity 1 = {...}, num_rows=21, null_map=0x0, is_join=true, is_build=true, bucket_size=32) at /mnt/disk2/tengjianping/doris-39/be/src/vec/common/hash_table/hash_map_context.h:470 #10 0x00005555a48d9228 in doris::pipeline::ProcessHashTableBuild<doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false> >::run<1, false, false, false> (this=0x75d6195d92a0, hash_table_ctx=..., null_map=0x0, has_null_key=0x6120057db3c0) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.h:208 #11 0x00005555a456dd23 in doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeS tate*, doris::vectorized::Block&)::$_1::operator()<doris::vectorized::MethodKeysFixed<doris::JoinHashTable< unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<do--Type <RET> for more, q to qu --Type <RET> for more, q to quit, c to continue without paging--c ris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>) const (this=0x75d61989aa10, arg=..., join_op=..., has_null_value=..., short_circuit_for_null_in_build_side=..., with_other_conjuncts=...) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:296 #12 0x00005555a456d93c in std::__invoke_impl<doris::Status, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(std::__invoke_other, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) (__f=..., __args=..., __args=..., __args=..., __args=..., __args=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 #13 0x00005555a456d88c in std::__invoke<doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false> >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>&, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) (__fn=..., __args=..., __args=..., __args=..., __args=..., __args=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96 #14 0x00005555a43eef75 in _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIN5doris6StatusEEEONS4_10vectorized8OverloadIJZNS4_8pipeline27HashJoinBuildSinkLocalState19process_build_blockEPNS4_12RuntimeStateERNS7_5BlockEE3$_0ZNSA_19process_build_blockESC_SE_E3$_1EEERSt7variantIJSt9monostateNS7_16MethodSerializedINS4_13JoinHashTableINS4_9StringRefE11DefaultHashISN_vEEEEENS7_15MethodOneNumberIhNSM_Ih9HashCRC32IhEEEEENSS_ItNSM_ItST_ItEEEEENSS_IjNSM_IjST_IjEEEEENSS_ImNSM_ImST_ImEEEEENSS_IN4wide7integerILm128EjEENSM_IS18_ST_IS18_EEEEENSS_INS17_ILm256EjEENSM_IS1C_ST_IS1C_EEEEENS7_15MethodKeysFixedIS14_Lb1EEENS1G_IS14_Lb0EEENS1G_IS1A_Lb1EEENS1G_IS1A_Lb0EEENS1G_IS1E_Lb1EEENS1G_IS1E_Lb0EEENS1G_INSM_INS7_7UInt136EST_IS1N_EEELb1EEENS1G_IS1P_Lb0EEEEERSJ_IJSt17integral_constantINS4_7TJoinOp4typeELS1W_0EES1U_IS1W_LS1W_2EES1U_IS1W_LS1W_8EES1U_IS1W_LS1W_1EES1U_IS1W_LS1W_4EES1U_IS1W_LS1W_3EES1U_IS1W_LS1W_5EES1U_IS1W_LS1W_7EES1U_IS1W_LS1W_9EES1U_IS1W_LS1W_10EES1U_IS1W_LS1W_11EEEEOSJ_IJS1U_IbLb0EES1U_IbLb1EEEES2D_S2D_EJEEESt16integer_sequenceImJLm9ELm3ELm0ELm0ELm0EEEE14__visit_invokeESI_S1T_S29_S2D_S2D_S2D_ (__visitor=..., __vars=..., __vars=..., __vars=..., __vars=..., __vars=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013 #15 0x00005555a43cf068 in std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> > >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&) (__visitor=..., __variants=..., __variants=..., __variants=..., __variants=..., __variants=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714 #16 0x00005555a43c8dea in std::visit<doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> > >(doris::vectorized::Overload<doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_0, doris::pipeline::HashJoinBuildSinkLocalState::process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1>&&, std::variant<std::monostate, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, doris::vectorized::MethodOneNumber<unsigned char, doris::JoinHashTable<unsigned char, HashCRC32<unsigned char> > >, doris::vectorized::MethodOneNumber<unsigned short, doris::JoinHashTable<unsigned short, HashCRC32<unsigned short> > >, doris::vectorized::MethodOneNumber<unsigned int, doris::JoinHashTable<unsigned int, HashCRC32<unsigned int> > >, doris::vectorized::MethodOneNumber<unsigned long, doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> > >, doris::vectorized::MethodOneNumber<wide::integer<128ul, unsigned int>, doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > > >, doris::vectorized::MethodOneNumber<wide::integer<256ul, unsigned int>, doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > > >, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<unsigned long, HashCRC32<unsigned long> >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<128ul, unsigned int>, HashCRC32<wide::integer<128ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<wide::integer<256ul, unsigned int>, HashCRC32<wide::integer<256ul, unsigned int> > >, false>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, true>, doris::vectorized::MethodKeysFixed<doris::JoinHashTable<doris::vectorized::UInt136, HashCRC32<doris::vectorized::UInt136> >, false> >&, std::variant<std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)0>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)2>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)8>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)1>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)4>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)3>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)5>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)7>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)9>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)10>, std::integral_constant<doris::TJoinOp::type, (doris::TJoinOp::type)11> >&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true> >&&) (__visitor=..., __variants=..., __variants=..., __variants=..., __variants=..., __variants=...) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib64/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769 #17 0x00005555a43c848d in doris::pipeline::HashJoinBuildSinkLocalState::process_build_block (this=0x616000669080, state=0x61e003eca080, block=...) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:278 #18 0x00005555a43cd0a2 in doris::pipeline::HashJoinBuildSinkOperatorX::sink (this=0x613001702bc0, state=0x61e003eca080, in_block=0x60d0000f7d40, eos=true) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/exec/hashjoin_build_sink.cpp:557 #19 0x00005555a7bdd4e1 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const (this=0x75d619925bd0) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/pipeline_task.cpp:356 #20 0x00005555a7bda84c in doris::pipeline::PipelineTask::execute (this=0x61700073d800, eos=0x75d6198f4230) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/pipeline_task.cpp:359 #21 0x00005555a7c34c6a in doris::pipeline::TaskScheduler::_do_work (this=0x60b00226b9e0, index=10) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/task_scheduler.cpp:135 #22 0x00005555a7c374bb in doris::pipeline::TaskScheduler::start()::$_0::operator()() const (this=0x6060055527f8) at /mnt/disk2/tengjianping/doris-39/be/src/pipeline/task_scheduler.cpp:64 ```
dataroaring
pushed a commit
that referenced
this issue
Jun 21, 2024
If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) #0 0x00007fa0bf784387 in ?? () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libc.so.6 #1 0x0000557dc8e617a0 in google::LogMessage::Flush() () #2 0x0000557dc8e65599 in google::LogMessageFatal::~LogMessageFatal() () #3 0x0000557dbe4ec939 in doris::vectorized::IColumn::get_shrinked_column (this=0x7fa094ac16a0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column.h:134 #4 0x0000557dc236d8a2 in doris::vectorized::Block::shrink_char_type_column_suffix_zero (this=0x7f9a372a4b70, char_type_idx=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 #5 0x0000557dbee9ea95 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=<optimized out>, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 #6 0x0000557dbee9cc5a in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 #7 doris::segment_v2::SegmentIterator::next_batch (this=0x1c5d, block=0x6) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2089 #8 0x0000557dbeebbb44 in doris::BetaRowsetReader::next_block (this=0x7f9a3a041600, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 #9 0x0000557dc8590ddd in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 #10 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 #11 0x0000557dc8591055 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref (this=0x1c5d) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 #12 0x0000557dc85934f2 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref (this=0x7fa090e36c00) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 #13 0x0000557dc858ddc9 in doris::vectorized::VCollectIterator::build_heap (this=0x7f9d80976dc0, rs_readers=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 #14 0x0000557dc857e808 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f9d80976800, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 #15 0x0000557dc857f64d in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 #16 0x0000557dc370dc2c in doris::vectorized::NewOlapScanner::open (this=0x7f9a2aa68210, state=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 #17 0x0000557dc3718d89 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=std::shared_ptr<doris::vectorized::ScannerContext> (use count 4, weak count 1) = {...}, scan_task=std::shared_ptr<doris::vectorized::ScanTask> (use count 2, weak count 0) = {...}) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:242 #18 0x0000557dc3719b77 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const (this=0x0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:158 #19 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 #20 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 #21 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 #22 0x0000557dbf3c7788 in doris::ThreadPool::dispatch_thread (this=0x7f9d0bbdc400) at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:543 #23 0x0000557dbf3bcb41 in std::function<void ()>::operator()() const (this=0x1471) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 #24 doris::Thread::supervise_thread (arg=0x7f9d0bbdf4c0) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:498 #25 0x00007fa0bee1dea5 in sem_open () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libpthread.so.0 ```
924060929
added a commit
that referenced
this issue
Jun 28, 2024
## Proposed changes The legacy coordinator act not only scheduler but also distribute planner. The code is so complex to understand, and hard to extend, and exist many limitations. This pr extract and refine the computation of degree of parallel(dop) to a new DistributePlanner and resolve the limitations. ## How to use this function This function only use for nereids + pipelinex, and current only support query statement, and non cloud mode. Open this session variables to use this function: ```sql set enable_nereids_distribute_planner=true; -- default is false set enable_nereids_planner=true; -- default is true ``` ## Core process and concepts ``` ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌───────────────┐ ┌───────────────────┐ ┌─────────────────────────┐ │ │ Translate │ │ Typed │ │ Assign │ │ Wrap │ │ │ │ ──────────► │ PlanFragment │ ──────► │ UnassignedJob │ ───────► │ StaticAssignedJob │ ─────► │ PipelineDistributedPlan │ │ │ │ │ │ │ │ │ │ │ │ │ └──────────────┘ └───────────────┘ └───────────────────┘ └─────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ └──────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────┐ ┌─────────────────┐ ┌───────────────────┐ │ │ Distribute │ │ AdHoc │ │ │ PhysicalPlan │ ───────────► │ DistributedPlan │ ──────► │ PipelineScheduler │ │ │ │ │ │ │ └──────────────┘ └─────────────────┘ └───────────────────┘ ``` DistributePlanner is a new planner to compute dop and generate instances, it consume PlanFragment and do this tasks 1. Use PlanFragment to generate `UnassignedJob`, it's a **Typed Fragment**, decided how to calculate dop and how to select the datasource, but this fragment not yet assigned some backends and datasources. These are some unassignedJobs: UnassignedScanSingleOlapTableJob, UnassignedScanBucketOlapTableJob, UnassignedShuffleJob, UnassignedQueryConstantJob. Keep UnassignedJob different can decoupling unrelated logic, and easy to extend: just and a new type of UnassignedJob. 2. Use UnassignedJob to select datasource, compute dop, and generate `AssignedJob`, means a instance, which already assigned datasource and backend. There are StaticAssignedJob and LocalShuffleAssignedJob, we will add DynamicAssignedJob when support StageScheduler and adaptive query execution 3. Wrap PlanFragment, UnassignedJob and AssignedJob to `PipelineDistributedPlan`, the coordinator will consume the DistributedPlan and translate to TPlan and schedule instances ## Resolve limitations **1. left table shuffle to right table** if right table has distribution which distribute by `storage hash`, and left table has distribution which distribute by `compute hash`, we can shuffle left to right by `storage hash` to do shuffle bucket join, and keep right side not move. ```sql select * from ( select id2 from test_shuffle_left group by id2 ) a inner join [shuffle] test_shuffle_left b on a.id2=b.id; | PhysicalResultSink[288] ( outputExprs=[id2#1, id#2, id2#3] ) ... | +--PhysicalHashJoin[285]@4 ( type=INNER_JOIN, stats=3, hashCondition=[(id2#1 = id#2)], otherCondition=[], markCondition=[], hint=[shuffle] ) ... | |--PhysicalDistribute[281]@2 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=STORAGE_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=... | | +--PhysicalHashAggregate[278]@2 ( aggPhase=GLOBAL, aggMode=BUFFER_TO_RESULT, maybeUseStreaming=false, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], requir... | | +--PhysicalDistribute[275]@7 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, parti... | | +--PhysicalHashAggregate[272]@7 ( aggPhase=LOCAL, aggMode=INPUT_TO_BUFFER, maybeUseStreaming=true, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], req... | | +--PhysicalProject[269]@1 ( stats=3, projects=[id2#1] ) ... | | +--PhysicalOlapScan[test_shuffle_left]@0 ( stats=3 ) ... | +--PhysicalOlapScan[test_shuffle_left]@3 ( stats=3 ) ``` **2. support colocate union numbers function** support use one instance to union/join numbers, note this plan no any PhysicalDistribute plan: ```sql explain physical plan select * from numbers('number'='3')a union all select * from numbers('number'='4')b PhysicalResultSink[98] ( outputExprs=[number#2] ) +--PhysicalUnion@ ( qualifier=ALL, outputs=[number#2], regularChildrenOutputs=[[number#0], [number#1]], constantExprsList=[], stats=7 ) |--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#0], function=numbers('number' = '3') ) +--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#1], function=numbers('number' = '4') ) ``` **3. support bucket prune with right outer bucket shuffle join** left table prune some buckets, say [bucket 1, bucket 3] we should process the right bucket shuffle join like this ``` [ (left bucket 1) right outer join (exchange right table which should process by bucket 1), (empty bucket) right outer join (exchange right table which should process by bucket 2), (left bucket 3) right outer join (exchange right table which should process by bucket 3) ] ``` the left bucket 2 is pruned, so right table can not shuffle to left, because the left instance not exists, so bucket 2 will return empty rows and wrong. new DistributePlanner can fill up this instance. the case: ```sql explain physical plan SELECT * FROM (select * from test_outer_join1 where c0 =1)a RIGHT OUTER JOIN (select * from test_outer_join2)b ON a.c0 = b.c0 ``` ### New feature add an explain statement to show distributed plans ```sql explain distributed plan select ... ``` for example, you can use this function to check how many instances generated, how many bytes the instance will scan, which backend will process the instance: ```sql MySQL root@127.0.0.1:test> explain distributed plan select * from test_shuffle_left2 a join [shuffle] test_shuffle_left2 b on a.id2=b.id; Explain String(Nereids Planner) ------------------------------------------------------------------------------------------------------- PipelineDistributedPlan( id: 0, parallel: 2, fragmentJob: UnassignedScanSingleOlapTableJob, fragment: { OUTPUT EXPRS: id[#8] id2[#9] id[#10] id2[#11] PARTITION: HASH_PARTITIONED: id2[#3] HAS_COLO_PLAN_NODE: false VRESULT SINK MYSQL_PROTOCAL 3:VHASH JOIN(152) | join op: INNER JOIN(PARTITIONED)[] | equal join conjunct: (id2[#3] = id[#0]) | cardinality=3 | vec output tuple id: 3 | output tuple id: 3 | vIntermediate tuple ids: 2 | hash output slot ids: 0 1 2 3 | isMarkJoin: false | final projections: id[#4], id2[#5], id[#6], id2[#7] | final project output tuple id: 3 | distribute expr lists: id2[#3] | distribute expr lists: id[#0] | tuple ids: 1 0 | |----0:VOlapScanNode(149) | TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON | partitions=1/1 (test_shuffle_left2) | tablets=10/10, tabletList=22038,22040,22042 ... | cardinality=3, avgRowSize=0.0, numNodes=1 | pushAggOp=NONE | tuple ids: 0 | 2:VEXCHANGE offset: 0 distribute expr lists: id[#2] tuple ids: 1 }, instanceJobs: [ LocalShuffleAssignedJob( index: 0, worker: BackendWorker(id: 10095, address: 192.168.126.1:9050), shareScanIndex: 0, scanSource: [ { scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 400, ranges: [ tablet 22038, bytes: 0, tablet 22042, bytes: 0, tablet 22046, bytes: 0, tablet 22050, bytes: 400, tablet 22054, bytes: 0 ]) } ] ), LocalShuffleAssignedJob( index: 1, worker: BackendWorker(id: 10096, address: 192.168.126.2:9051), shareScanIndex: 1, scanSource: [ { scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 796, ranges: [ tablet 22040, bytes: 397, tablet 22044, bytes: 0, tablet 22048, bytes: 399, tablet 22052, bytes: 0, tablet 22056, bytes: 0 ]) } ] ) ] ) PipelineDistributedPlan( id: 1, parallel: 2, fragmentJob: UnassignedScanSingleOlapTableJob, fragment: { PARTITION: HASH_PARTITIONED: id[#2] HAS_COLO_PLAN_NODE: false STREAM DATA SINK EXCHANGE ID: 02 HASH_PARTITIONED: id2[#3] 1:VOlapScanNode(145) TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON partitions=1/1 (test_shuffle_left2) tablets=10/10, tabletList=22038,22040,22042 ... cardinality=3, avgRowSize=0.0, numNodes=1 pushAggOp=NONE tuple ids: 1 }, instanceJobs: [ LocalShuffleAssignedJob( index: 0, worker: BackendWorker(id: 10095, address: 192.168.126.1:9050), shareScanIndex: 0, scanSource: [ { scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 400, ranges: [ tablet 22038, bytes: 0, tablet 22042, bytes: 0, tablet 22046, bytes: 0, tablet 22050, bytes: 400, tablet 22054, bytes: 0 ]) } ] ), LocalShuffleAssignedJob( index: 1, worker: BackendWorker(id: 10096, address: 192.168.126.2:9051), shareScanIndex: 1, scanSource: [ { scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 796, ranges: [ tablet 22040, bytes: 397, tablet 22044, bytes: 0, tablet 22048, bytes: 399, tablet 22052, bytes: 0, tablet 22056, bytes: 0 ]) } ] ) ] ) Hint log: Used: [shuffle]_2 UnUsed: SyntaxError: ``` ## TODO 1. extract PipelineScheduler from Coordinator 2. move this framework into cascades and compute cost by dop 3. support StageScheduler, adaptive query execution and DynamicAssignedJob
dataroaring
pushed a commit
that referenced
this issue
Jun 30, 2024
## Proposed changes The legacy coordinator act not only scheduler but also distribute planner. The code is so complex to understand, and hard to extend, and exist many limitations. This pr extract and refine the computation of degree of parallel(dop) to a new DistributePlanner and resolve the limitations. ## How to use this function This function only use for nereids + pipelinex, and current only support query statement, and non cloud mode. Open this session variables to use this function: ```sql set enable_nereids_distribute_planner=true; -- default is false set enable_nereids_planner=true; -- default is true ``` ## Core process and concepts ``` ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌───────────────┐ ┌───────────────────┐ ┌─────────────────────────┐ │ │ Translate │ │ Typed │ │ Assign │ │ Wrap │ │ │ │ ──────────► │ PlanFragment │ ──────► │ UnassignedJob │ ───────► │ StaticAssignedJob │ ─────► │ PipelineDistributedPlan │ │ │ │ │ │ │ │ │ │ │ │ │ └──────────────┘ └───────────────┘ └───────────────────┘ └─────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ └──────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────┐ ┌─────────────────┐ ┌───────────────────┐ │ │ Distribute │ │ AdHoc │ │ │ PhysicalPlan │ ───────────► │ DistributedPlan │ ──────► │ PipelineScheduler │ │ │ │ │ │ │ └──────────────┘ └─────────────────┘ └───────────────────┘ ``` DistributePlanner is a new planner to compute dop and generate instances, it consume PlanFragment and do this tasks 1. Use PlanFragment to generate `UnassignedJob`, it's a **Typed Fragment**, decided how to calculate dop and how to select the datasource, but this fragment not yet assigned some backends and datasources. These are some unassignedJobs: UnassignedScanSingleOlapTableJob, UnassignedScanBucketOlapTableJob, UnassignedShuffleJob, UnassignedQueryConstantJob. Keep UnassignedJob different can decoupling unrelated logic, and easy to extend: just and a new type of UnassignedJob. 2. Use UnassignedJob to select datasource, compute dop, and generate `AssignedJob`, means a instance, which already assigned datasource and backend. There are StaticAssignedJob and LocalShuffleAssignedJob, we will add DynamicAssignedJob when support StageScheduler and adaptive query execution 3. Wrap PlanFragment, UnassignedJob and AssignedJob to `PipelineDistributedPlan`, the coordinator will consume the DistributedPlan and translate to TPlan and schedule instances ## Resolve limitations **1. left table shuffle to right table** if right table has distribution which distribute by `storage hash`, and left table has distribution which distribute by `compute hash`, we can shuffle left to right by `storage hash` to do shuffle bucket join, and keep right side not move. ```sql select * from ( select id2 from test_shuffle_left group by id2 ) a inner join [shuffle] test_shuffle_left b on a.id2=b.id; | PhysicalResultSink[288] ( outputExprs=[id2#1, id#2, id2#3] ) ... | +--PhysicalHashJoin[285]@4 ( type=INNER_JOIN, stats=3, hashCondition=[(id2#1 = id#2)], otherCondition=[], markCondition=[], hint=[shuffle] ) ... | |--PhysicalDistribute[281]@2 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=STORAGE_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=... | | +--PhysicalHashAggregate[278]@2 ( aggPhase=GLOBAL, aggMode=BUFFER_TO_RESULT, maybeUseStreaming=false, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], requir... | | +--PhysicalDistribute[275]@7 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, parti... | | +--PhysicalHashAggregate[272]@7 ( aggPhase=LOCAL, aggMode=INPUT_TO_BUFFER, maybeUseStreaming=true, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], req... | | +--PhysicalProject[269]@1 ( stats=3, projects=[id2#1] ) ... | | +--PhysicalOlapScan[test_shuffle_left]@0 ( stats=3 ) ... | +--PhysicalOlapScan[test_shuffle_left]@3 ( stats=3 ) ``` **2. support colocate union numbers function** support use one instance to union/join numbers, note this plan no any PhysicalDistribute plan: ```sql explain physical plan select * from numbers('number'='3')a union all select * from numbers('number'='4')b PhysicalResultSink[98] ( outputExprs=[number#2] ) +--PhysicalUnion@ ( qualifier=ALL, outputs=[number#2], regularChildrenOutputs=[[number#0], [number#1]], constantExprsList=[], stats=7 ) |--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#0], function=numbers('number' = '3') ) +--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#1], function=numbers('number' = '4') ) ``` **3. support bucket prune with right outer bucket shuffle join** left table prune some buckets, say [bucket 1, bucket 3] we should process the right bucket shuffle join like this ``` [ (left bucket 1) right outer join (exchange right table which should process by bucket 1), (empty bucket) right outer join (exchange right table which should process by bucket 2), (left bucket 3) right outer join (exchange right table which should process by bucket 3) ] ``` the left bucket 2 is pruned, so right table can not shuffle to left, because the left instance not exists, so bucket 2 will return empty rows and wrong. new DistributePlanner can fill up this instance. the case: ```sql explain physical plan SELECT * FROM (select * from test_outer_join1 where c0 =1)a RIGHT OUTER JOIN (select * from test_outer_join2)b ON a.c0 = b.c0 ``` ### New feature add an explain statement to show distributed plans ```sql explain distributed plan select ... ``` for example, you can use this function to check how many instances generated, how many bytes the instance will scan, which backend will process the instance: ```sql MySQL root@127.0.0.1:test> explain distributed plan select * from test_shuffle_left2 a join [shuffle] test_shuffle_left2 b on a.id2=b.id; Explain String(Nereids Planner) ------------------------------------------------------------------------------------------------------- PipelineDistributedPlan( id: 0, parallel: 2, fragmentJob: UnassignedScanSingleOlapTableJob, fragment: { OUTPUT EXPRS: id[#8] id2[#9] id[#10] id2[#11] PARTITION: HASH_PARTITIONED: id2[#3] HAS_COLO_PLAN_NODE: false VRESULT SINK MYSQL_PROTOCAL 3:VHASH JOIN(152) | join op: INNER JOIN(PARTITIONED)[] | equal join conjunct: (id2[#3] = id[#0]) | cardinality=3 | vec output tuple id: 3 | output tuple id: 3 | vIntermediate tuple ids: 2 | hash output slot ids: 0 1 2 3 | isMarkJoin: false | final projections: id[#4], id2[#5], id[#6], id2[#7] | final project output tuple id: 3 | distribute expr lists: id2[#3] | distribute expr lists: id[#0] | tuple ids: 1 0 | |----0:VOlapScanNode(149) | TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON | partitions=1/1 (test_shuffle_left2) | tablets=10/10, tabletList=22038,22040,22042 ... | cardinality=3, avgRowSize=0.0, numNodes=1 | pushAggOp=NONE | tuple ids: 0 | 2:VEXCHANGE offset: 0 distribute expr lists: id[#2] tuple ids: 1 }, instanceJobs: [ LocalShuffleAssignedJob( index: 0, worker: BackendWorker(id: 10095, address: 192.168.126.1:9050), shareScanIndex: 0, scanSource: [ { scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 400, ranges: [ tablet 22038, bytes: 0, tablet 22042, bytes: 0, tablet 22046, bytes: 0, tablet 22050, bytes: 400, tablet 22054, bytes: 0 ]) } ] ), LocalShuffleAssignedJob( index: 1, worker: BackendWorker(id: 10096, address: 192.168.126.2:9051), shareScanIndex: 1, scanSource: [ { scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 796, ranges: [ tablet 22040, bytes: 397, tablet 22044, bytes: 0, tablet 22048, bytes: 399, tablet 22052, bytes: 0, tablet 22056, bytes: 0 ]) } ] ) ] ) PipelineDistributedPlan( id: 1, parallel: 2, fragmentJob: UnassignedScanSingleOlapTableJob, fragment: { PARTITION: HASH_PARTITIONED: id[#2] HAS_COLO_PLAN_NODE: false STREAM DATA SINK EXCHANGE ID: 02 HASH_PARTITIONED: id2[#3] 1:VOlapScanNode(145) TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON partitions=1/1 (test_shuffle_left2) tablets=10/10, tabletList=22038,22040,22042 ... cardinality=3, avgRowSize=0.0, numNodes=1 pushAggOp=NONE tuple ids: 1 }, instanceJobs: [ LocalShuffleAssignedJob( index: 0, worker: BackendWorker(id: 10095, address: 192.168.126.1:9050), shareScanIndex: 0, scanSource: [ { scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 400, ranges: [ tablet 22038, bytes: 0, tablet 22042, bytes: 0, tablet 22046, bytes: 0, tablet 22050, bytes: 400, tablet 22054, bytes: 0 ]) } ] ), LocalShuffleAssignedJob( index: 1, worker: BackendWorker(id: 10096, address: 192.168.126.2:9051), shareScanIndex: 1, scanSource: [ { scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1}, scanRanges: ScanRanges(bytes: 796, ranges: [ tablet 22040, bytes: 397, tablet 22044, bytes: 0, tablet 22048, bytes: 399, tablet 22052, bytes: 0, tablet 22056, bytes: 0 ]) } ] ) ] ) Hint log: Used: [shuffle]_2 UnUsed: SyntaxError: ``` ## TODO 1. extract PipelineScheduler from Coordinator 2. move this framework into cascades and compute cost by dop 3. support StageScheduler, adaptive query execution and DynamicAssignedJob
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this issue
Jul 2, 2024
query like ``` select a, `__DORIS_DELETE_SIGN__` from `test_tbl` WHERE k = '1111111' ``` will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment. ``` +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[#0] | | k2[apache#1] | | k3[apache#2] | | k4[apache#3] | | k5[apache#4] | | k6[apache#5] | | k7[apache#6] | | k8[apache#7] | | k9[apache#8] | | k10[apache#9] | | k11[apache#10] | | k12[apache#11] | | new_column0[apache#12] | | new_column1[apache#13] | | __DORIS_DELETE_SIGN__[apache#14] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: k1[#0], k2[apache#1], k3[apache#2] | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[apache#1], k3[apache#2] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:VOlapScanNode(149) | | TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON | | PREDICATES: (k1[#0] = 1231) AND (k2[apache#1] = 119291.110000000) AND (k3[apache#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) | | partitions=1/1 (tbl_point_query0) | | tablets=1/1, tabletList=13203 | | cardinality=0, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | SHORT-CIRCUIT | | | | | | Statistics | | planed with unknown column statistics | +----------------------------------------------------------------------------------------------------------------------------+ ```
morningman
added a commit
that referenced
this issue
Jul 2, 2024
## Proposed changes This is just a workround try avoid coredump like this: ``` #0 0x56414f0e8ed1 in __asan::CheckUnwind() crtstuff.c #1 0x56414f1009a2 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) crtstuff.c #2 0x56414f0ecbf3 in __asan::AsanThread::GetStackFrameAccessByAddr(unsigned long, __asan::AsanThread::StackFrameAccess*) crtstuff.c #3 0x56414f050d87 in __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, bool) crtstuff.c #4 0x56414f052a73 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long) crtstuff.c #5 0x56414f0e6a9e in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) crtstuff.c #6 0x56414f066885 in gmtime_r (/mnt/hdd01/ci/branch21-deploy/be/lib/doris_be+0x17ef3885) (BuildId: f58eb5e327529636) #7 0x564177940521 in google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) crtstuff.c #8 0x564151de36fc in doris::Status doris::ThriftRpcHelper::rpc(std::__cxx11::basic_string, std::allocator> const&, int, std::function&)>, int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_rpc_helper.cpp:76:13 #9 0x56417603cda7 in doris::vectorized::VRowDistribution::automatic_create_partition() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/vrow_distribution.cpp:99:5 #10 0x56417614cffa in doris::vectorized::VTabletWriter::_send_new_partition_batch() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/writer/vtablet_writer.cpp:1346:9 .... ```
eldenmoon
added a commit
that referenced
this issue
Jul 3, 2024
query like ``` select a, `__DORIS_DELETE_SIGN__` from `test_tbl` WHERE k = '1111111' ``` will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment. ``` +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[#0] | | k2[#1] | | k3[#2] | | k4[#3] | | k5[#4] | | k6[#5] | | k7[#6] | | k8[#7] | | k9[#8] | | k10[#9] | | k11[#10] | | k12[#11] | | new_column0[#12] | | new_column1[#13] | | __DORIS_DELETE_SIGN__[#14] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: k1[#0], k2[#1], k3[#2] | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[#1], k3[#2] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:VOlapScanNode(149) | | TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON | | PREDICATES: (k1[#0] = 1231) AND (k2[#1] = 119291.110000000) AND (k3[#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[#14] = 0) | | partitions=1/1 (tbl_point_query0) | | tablets=1/1, tabletList=13203 | | cardinality=0, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | SHORT-CIRCUIT | | | | | | Statistics | | planed with unknown column statistics | +----------------------------------------------------------------------------------------------------------------------------+ ```
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this issue
Jul 9, 2024
query like ``` select a, `__DORIS_DELETE_SIGN__` from `test_tbl` WHERE k = '1111111' ``` will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment. ``` +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[#0] | | k2[apache#1] | | k3[apache#2] | | k4[apache#3] | | k5[apache#4] | | k6[apache#5] | | k7[apache#6] | | k8[apache#7] | | k9[apache#8] | | k10[apache#9] | | k11[apache#10] | | k12[apache#11] | | new_column0[apache#12] | | new_column1[apache#13] | | __DORIS_DELETE_SIGN__[apache#14] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: k1[#0], k2[apache#1], k3[apache#2] | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[apache#1], k3[apache#2] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:VOlapScanNode(149) | | TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON | | PREDICATES: (k1[#0] = 1231) AND (k2[apache#1] = 119291.110000000) AND (k3[apache#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) | | partitions=1/1 (tbl_point_query0) | | tablets=1/1, tabletList=13203 | | cardinality=0, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | SHORT-CIRCUIT | | | | | | Statistics | | planed with unknown column statistics | +----------------------------------------------------------------------------------------------------------------------------+ ```
morningman
pushed a commit
that referenced
this issue
Jul 9, 2024
## Proposed changes Currently, Arrow use separate Jemalloc, and use non-standard methods `mallocx`, `sdallocx`, `rallocx` in `memory_pool_jemalloc.cc` to optimize memory allocation. But this may be incompatible with older versions of the Linux kernel. when we use Arrow on Arm Kirin v10 or Centos 7.4, it will get stuck on the Jemalloc Lock, with the stack below, it will appear when calling `arrow::RecordBatch::MakeEmpty`. the kernel version of Arm Kylin v10 is 4.19.90, and the kernel version of Centos 7.4 is 4.14. ``` #0 0x0000ffffae3ceff8 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000ffffae3c9b50 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x0000ffffae61c834 in pthread_mutex_lock () from /lib64/libc.so.6 #3 0x0000aaaac99bc1e0 in je_arrow_private_je_malloc_mutex_lock_slow () #4 0x0000aaaac99af3a4 in ?? () #5 0x0000aaaac99b576c in je_arrow_mallocx () #6 0x0000aaaac99a8aec in ?? () #7 0x0000aaaac99a9858 in arrow::AllocateResizableBuffer(long, arrow::MemoryPool*) () #8 0x0000aaaac399f8b8 in arrow::BufferBuilder::Resize(long, bool) () #9 0x0000aaaac983715c in arrow::BaseBinaryBuilder<arrow::BinaryType>::Resize(long) () #10 0x0000aaaac39a47e0 in arrow::BaseBinaryBuilder<arrow::BinaryType>::Append(unsigned char const*, int) () ``` After disable separate Jemalloc when compiling Arrow, the above error disappeared, and Arrow will use the default memory allocator, which is Doris Jemalloc.
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this issue
Jul 16, 2024
query like ``` select a, `__DORIS_DELETE_SIGN__` from `test_tbl` WHERE k = '1111111' ``` will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment. ``` +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[#0] | | k2[apache#1] | | k3[apache#2] | | k4[apache#3] | | k5[apache#4] | | k6[apache#5] | | k7[apache#6] | | k8[apache#7] | | k9[apache#8] | | k10[apache#9] | | k11[apache#10] | | k12[apache#11] | | new_column0[apache#12] | | new_column1[apache#13] | | __DORIS_DELETE_SIGN__[apache#14] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: k1[#0], k2[apache#1], k3[apache#2] | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[apache#1], k3[apache#2] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | UNPARTITIONED | | | | 0:VOlapScanNode(149) | | TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON | | PREDICATES: (k1[#0] = 1231) AND (k2[apache#1] = 119291.110000000) AND (k3[apache#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) | | partitions=1/1 (tbl_point_query0) | | tablets=1/1, tabletList=13203 | | cardinality=0, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | SHORT-CIRCUIT | | | | | | Statistics | | planed with unknown column statistics | +----------------------------------------------------------------------------------------------------------------------------+ ```
morningman
pushed a commit
that referenced
this issue
Jul 16, 2024
## Proposed changes - Null pointer of type 'doris::StringRef' in orc reader. The root cause is error will throw when `num_values == 0` in `_decode_string_non_dict_encoded_column`. ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39 #2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16 #3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16 #4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 #5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9 #6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 ... ``` - Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') in parquet reader. The root cause is error will throw when `len == 0`. ``` /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') #0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27 #1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9 #2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24 #3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13 #4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9 #5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36 #6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13 #7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17 #8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 #9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12 #10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5 #11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25 #12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35 ... ```
seawinde
pushed a commit
to seawinde/doris
that referenced
this issue
Jul 17, 2024
## Proposed changes - Null pointer of type 'doris::StringRef' in orc reader. The root cause is error will throw when `num_values == 0` in `_decode_string_non_dict_encoded_column`. ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39 apache#2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16 apache#3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16 apache#4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 apache#5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9 apache#6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 ... ``` - Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') in parquet reader. The root cause is error will throw when `len == 0`. ``` /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') #0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27 #1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9 apache#2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24 apache#3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13 apache#4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9 apache#5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36 apache#6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13 apache#7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17 apache#8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 apache#9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12 apache#10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5 apache#11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25 apache#12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35 ... ```
dataroaring
pushed a commit
that referenced
this issue
Jul 17, 2024
## Proposed changes - Null pointer of type 'doris::StringRef' in orc reader. The root cause is error will throw when `num_values == 0` in `_decode_string_non_dict_encoded_column`. ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39 #2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16 #3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16 #4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 #5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9 #6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 ... ``` - Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') in parquet reader. The root cause is error will throw when `len == 0`. ``` /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') #0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27 #1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9 #2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24 #3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13 #4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9 #5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36 #6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13 #7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17 #8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 #9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12 #10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5 #11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25 #12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35 ... ```
yiguolei
pushed a commit
that referenced
this issue
Jul 18, 2024
…out range of _buf (#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 #5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 #6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 #7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 #10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 #11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 #12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 #13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 #18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 #19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 #22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 #23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 #28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) #29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 #5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
dataroaring
pushed a commit
that referenced
this issue
Jul 19, 2024
…out range of _buf (#37936) if we select nested type such as map/array/struct after large string , when string type in mysql_row_buf reserve make buffer size is not large enough , which will lead nested type open_dynamic_mode make _pos pointer out range of mysql_row_buf, then nested type call push_string, and reserve() will make heap_buffer_overflow ``` ==200769==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0051c12ba at pc 0x55a77788692d bp 0x7fb52f474a30 sp 0x7fb52f4741f8 READ of size 36541 at 0x62d0051c12ba thread T2309 (Pipe_normal [wo) #0 0x55a77788692c in __asan_memcpy (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c1c92c) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f622fa in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:140:5 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a21f16eb in doris::Status doris::vectorized::DataTypeMapSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:410:21 #4 0x55a7a21e4c1e in doris::vectorized::DataTypeMapSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_map_serde.cpp:478:12 #5 0x55a7a22070e6 in doris::Status doris::vectorized::DataTypeNullableSerDe::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:300:9 #6 0x55a7a21fbc5e in doris::vectorized::DataTypeNullableSerDe::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:317:12 #7 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #8 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #9 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 #10 0x55a7c99a4b27 in doris::pipeline::PipelineTask::execute(bool*) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:364:22 #11 0x55a7c9a23a2b in doris::pipeline::TaskScheduler::_do_work(unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:138:9 #12 0x55a7c9a269ca in doris::pipeline::TaskScheduler::start()::$_0::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/task_scheduler.cpp:64:9 #13 0x55a7c9a2694e in void std::__invoke_impl<void, doris::pipeline::TaskScheduler::start()::$_0&>(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #14 0x55a7c9a268ae in std::enable_if<is_invocable_r_v<void, doris::pipeline::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void, doris::pipeline::TaskScheduler::start()::$_0&>(doris::pipeline::TaskScheduler::start()::$_0&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #15 0x55a7c9a26635 in std::_Function_handler<void (), doris::pipeline::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #16 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #17 0x55a77e95ec94 in doris::FunctionRunnable::run() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:48:27 #18 0x55a77e941015 in doris::ThreadPool::dispatch_thread() /mnt/disk1/wangqiannan/amory/doris/be/src/util/threadpool.cpp:543:24 #19 0x55a77e97eb23 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #20 0x55a77e97e928 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #21 0x55a77e97e860 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11 #22 0x55a77e97e655 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17 #23 0x55a77e97e54e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 #24 0x55a77e97e48e in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 #25 0x55a77e97dd85 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9 #26 0x55a777b226da in std::function<void ()>::operator()() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 #27 0x55a77e8fb841 in doris::Thread::supervise_thread(void*) /mnt/disk1/wangqiannan/amory/doris/be/src/util/thread.cpp:498:5 #28 0x7fc1c3a111c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0) #29 0x7fc1c4400e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce) 0x62d0051c12ba is located 0 bytes after 36538-byte region [0x62d0051b8400,0x62d0051c12ba) allocated by thread T2309 (Pipe_normal [wo) here: #0 0x55a7778c20bd in operator new[](unsigned long) (/mnt/disk1/wangqiannan/amory/doris/output/be/lib/doris_be+0x60c580bd) (BuildId: 4513940b6b9e22fa) #1 0x55a7a1f621c1 in doris::MysqlRowBuffer<false>::reserve(long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:137:21 #2 0x55a7a1f638eb in doris::MysqlRowBuffer<false>::push_string(char const*, long) /mnt/disk1/wangqiannan/amory/doris/be/src/util/mysql_row_buffer.cpp:473:5 #3 0x55a7a1fd0d75 in doris::Status doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::_write_column_to_mysql<false>(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:260:16 #4 0x55a7a1fccc1e in doris::vectorized::DataTypeStringSerDeBase<doris::vectorized::ColumnStr<unsigned int>>::write_column_to_mysql(doris::vectorized::IColumn const&, doris::MysqlRowBuffer<false>&, int, bool, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_string_serde.h:215:16 #5 0x55a7c2e97e6c in doris::vectorized::VMysqlResultWriter<false>::write(doris::RuntimeState*, doris::vectorized::Block&) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/sink/vmysql_result_writer.cpp:216:17 #6 0x55a7c8031b83 in doris::pipeline::ResultSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/exec/result_sink_operator.cpp:142:5 #7 0x55a7c99a81d6 in doris::pipeline::PipelineTask::execute(bool*)::$_1::operator()() const /mnt/disk1/wangqiannan/amory/doris/be/src/pipeline/pipeline_task.cpp:361:38 ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
kaka11chen
added a commit
to kaka11chen/doris
that referenced
this issue
Jul 24, 2024
## Proposed changes - Null pointer of type 'doris::StringRef' in orc reader. The root cause is error will throw when `num_values == 0` in `_decode_string_non_dict_encoded_column`. ``` /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:9: runtime error: reference binding to null pointer of type 'doris::StringRef' #0 0x562516fa9770 in std::vector<doris::StringRef, std::allocator<doris::StringRef> >::operator[](unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1046:2 #1 0x562516fa9770 in doris::Status doris::vectorized::OrcReader::_decode_string_non_dict_encoded_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::EncodedStringVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1164:39 apache#2 0x562516f9c08b in doris::Status doris::vectorized::OrcReader::_decode_string_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> const&, orc::TypeKind const&, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1116:16 apache#3 0x562516f91d73 in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1357:16 apache#4 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 apache#5 0x562516f9339a in doris::Status doris::vectorized::OrcReader::_fill_doris_data_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1402:9 apache#6 0x562516c79a0c in doris::Status doris::vectorized::OrcReader::_orc_column_to_doris_column<false>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const> const&, orc::Type const*, orc::ColumnVectorBatch*, unsigned long) /root/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:1524:5 ... ``` - Shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') in parquet reader. The root cause is error will throw when `len == 0`. ``` /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27: runtime error: shift exponent 128 is too large for 128-bit type 'ValueCopyType' (aka '__int128') #0 0x56251760fbc7 in doris::vectorized::parquet::StringToDecimal<doris::vectorized::Decimal128V3, (doris::vectorized::DecimalScaleParams::ScaleType)1>::physical_convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:413:27 #1 0x562517290dc4 in doris::vectorized::parquet::PhysicalToLogicalConverter::convert(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, doris::TypeDescriptor, std::shared_ptr<doris::vectorized::IDataType const> const&, COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, bool) /root/doris/be/src/vec/exec/format/parquet/parquet_column_convert.h:209:9 apache#2 0x562517284a6d in doris::vectorized::ScalarColumnReader::read_column_data(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, unsigned long, unsigned long*, bool*, bool) /root/doris/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:569:24 apache#3 0x56251725ae7e in doris::vectorized::RowGroupReader::_read_column_data(doris::vectorized::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::vectorized::ColumnSelectVector&) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:421:13 apache#4 0x56251724d6d2 in doris::vectorized::RowGroupReader::next_batch(doris::vectorized::Block*, unsigned long, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:321:9 apache#5 0x56251708eb97 in doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:530:36 apache#6 0x56253036772d in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:311:13 apache#7 0x562530366549 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vfile_scanner.cpp:253:17 apache#8 0x5625176e79c8 in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:117:17 apache#9 0x5625176e6fc1 in doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscanner.cpp:84:12 apache#10 0x562517698047 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:250:5 apache#11 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:172:25 apache#12 0x56251769bc1f in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:171:35 ... ```
kaka11chen
added a commit
to kaka11chen/doris
that referenced
this issue
Jul 24, 2024
kaka11chen
added a commit
to kaka11chen/doris
that referenced
this issue
Jul 24, 2024
… min-max filtering).
morrySnow
pushed a commit
that referenced
this issue
Jul 26, 2024
Induced by #34196. In NormalizeRepeat, when NormalizeToSlot is called, aggregate function parameters, grouping scalar function parameters, and all expressions in grouping sets (including columns and column aliases) are pushed down to the lower-level project output. In the previous PR #34196, the context was split into two, but the two contexts were not consistent. It is possible that the triplets in one context save (id, c1, id as c1), and the triplets in the other context save (id, id, id). This causes id as c1 to be pushed down, but there is a reference to id in the upper-level LogicalRepeat, which causes the slot to be not found. This pr has been modified. If the same slot in the projection column has different aliases, for example, select id as c1, id, id as c3, grouping(id) from table1 group by grouping sets((id, value2),(id)); then id as c1 (using the first alias) will be pushed down to the project. In both the LogicalRepeat operator and the LogicalAggregate operator, c1 is referenced as the input slot, and id and c3 will not be used as input slots. before NormalizeRepeat: LogicalResultSink[32] ( outputExprs=[c1#3, id#0, c3#4, __grouping_3#5] ) +--LogicalRepeat ( groupingSets=[[id#0, value2#2], [id#0]], outputExpressions=[id#0 AS `c1`#3, id#0, id#0 AS `c3`#4, Grouping(id#0) AS `Grouping(id)`#5] ) +--LogicalOlapScan (qualified=table1) After NormalizeRepeat: LogicalResultSink[33] (outputExprs=[c1#3, id#0, c3#4, __grouping_3#5]) +--LogicalAggregate[30] (groupByExpr=[c1#3, value2#2, GROUPING_ID#7, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3)], outputExpr=[c1#3, c1#3 AS `id`#0, c1#3 AS `c3`#4, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3) AS `GROUPING_PREFIX_c1`#5], hasRepeat=true ) +--LogicalRepeat (groupingSets=[[c1#3, value2#2], [c1#3]], outputExpressions=[c1#3, value2#2, GROUPING_ID#7, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3)] ) +--LogicalProject[28] (projects=[id#0 AS `c1`#3, value2#2]) +--LogicalOlapScan (qualified=table1)
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this issue
Jul 29, 2024
Induced by apache#34196. In NormalizeRepeat, when NormalizeToSlot is called, aggregate function parameters, grouping scalar function parameters, and all expressions in grouping sets (including columns and column aliases) are pushed down to the lower-level project output. In the previous PR apache#34196, the context was split into two, but the two contexts were not consistent. It is possible that the triplets in one context save (id, c1, id as c1), and the triplets in the other context save (id, id, id). This causes id as c1 to be pushed down, but there is a reference to id in the upper-level LogicalRepeat, which causes the slot to be not found. This pr has been modified. If the same slot in the projection column has different aliases, for example, select id as c1, id, id as c3, grouping(id) from table1 group by grouping sets((id, value2),(id)); then id as c1 (using the first alias) will be pushed down to the project. In both the LogicalRepeat operator and the LogicalAggregate operator, c1 is referenced as the input slot, and id and c3 will not be used as input slots. before NormalizeRepeat: LogicalResultSink[32] ( outputExprs=[c1#3, id#0, c3#4, __grouping_3#5] ) +--LogicalRepeat ( groupingSets=[[id#0, value2#2], [id#0]], outputExpressions=[id#0 AS `c1`apache#3, id#0, id#0 AS `c3`apache#4, Grouping(id#0) AS `Grouping(id)`apache#5] ) +--LogicalOlapScan (qualified=table1) After NormalizeRepeat: LogicalResultSink[33] (outputExprs=[c1#3, id#0, c3#4, __grouping_3#5]) +--LogicalAggregate[30] (groupByExpr=[c1#3, value2#2, GROUPING_ID#7, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3)], outputExpr=[c1#3, c1#3 AS `id`#0, c1#3 AS `c3`apache#4, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3) AS `GROUPING_PREFIX_c1`apache#5], hasRepeat=true ) +--LogicalRepeat (groupingSets=[[c1#3, value2#2], [c1#3]], outputExpressions=[c1#3, value2#2, GROUPING_ID#7, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3)] ) +--LogicalProject[28] (projects=[id#0 AS `c1`apache#3, value2#2]) +--LogicalOlapScan (qualified=table1)
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this issue
Jul 30, 2024
Induced by apache#34196. In NormalizeRepeat, when NormalizeToSlot is called, aggregate function parameters, grouping scalar function parameters, and all expressions in grouping sets (including columns and column aliases) are pushed down to the lower-level project output. In the previous PR apache#34196, the context was split into two, but the two contexts were not consistent. It is possible that the triplets in one context save (id, c1, id as c1), and the triplets in the other context save (id, id, id). This causes id as c1 to be pushed down, but there is a reference to id in the upper-level LogicalRepeat, which causes the slot to be not found. This pr has been modified. If the same slot in the projection column has different aliases, for example, select id as c1, id, id as c3, grouping(id) from table1 group by grouping sets((id, value2),(id)); then id as c1 (using the first alias) will be pushed down to the project. In both the LogicalRepeat operator and the LogicalAggregate operator, c1 is referenced as the input slot, and id and c3 will not be used as input slots. before NormalizeRepeat: LogicalResultSink[32] ( outputExprs=[c1#3, id#0, c3#4, __grouping_3#5] ) +--LogicalRepeat ( groupingSets=[[id#0, value2#2], [id#0]], outputExpressions=[id#0 AS `c1`apache#3, id#0, id#0 AS `c3`apache#4, Grouping(id#0) AS `Grouping(id)`apache#5] ) +--LogicalOlapScan (qualified=table1) After NormalizeRepeat: LogicalResultSink[33] (outputExprs=[c1#3, id#0, c3#4, __grouping_3#5]) +--LogicalAggregate[30] (groupByExpr=[c1#3, value2#2, GROUPING_ID#7, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3)], outputExpr=[c1#3, c1#3 AS `id`#0, c1#3 AS `c3`apache#4, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3) AS `GROUPING_PREFIX_c1`apache#5], hasRepeat=true ) +--LogicalRepeat (groupingSets=[[c1#3, value2#2], [c1#3]], outputExpressions=[c1#3, value2#2, GROUPING_ID#7, GROUPING_PREFIX_c1#6 originExpression=Grouping(c1#3)] ) +--LogicalProject[28] (projects=[id#0 AS `c1`apache#3, value2#2]) +--LogicalOlapScan (qualified=table1)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
修改为maven就好了
The text was updated successfully, but these errors were encountered: