Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

heap-buffer-overflow when scanner the data #17822

Closed
chaoyli opened this issue Feb 14, 2023 · 3 comments · Fixed by #17879
Closed

heap-buffer-overflow when scanner the data #17822

chaoyli opened this issue Feb 14, 2023 · 3 comments · Fixed by #17879
Labels
type/bug Something isn't working

Comments

@chaoyli
Copy link
Contributor

chaoyli commented Feb 14, 2023 

start time: Tue Feb 14 23:36:51 UTC 2023
=================================================================
==41735==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61f000dc009e at pc 0x00000aa97d60 bp 0x7ffa6f4c7e50 sp 0x7ffa6f4c7e48
WRITE of size 1 at 0x61f000dc009e thread T504 (con_wg_scan_io)
    #0 0xaa97d5f in starrocks::vectorized::fill_null_column(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::FixedLengthColumn<unsigned char>*, unsigned long) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:69
    #1 0xaaa8174 in starrocks::vectorized::ArrowListConverter::convert_list(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, std::vector<unsigned char, std::allocator<unsigned char> >*, starrocks::vectorized::ArrowConvertContext*, starrocks::TypeDescriptor const*) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:900
    #2 0xaaa88b3 in starrocks::vectorized::ArrowListConverter::convert_list_with_null(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, std::vector<unsigned char, std::allocator<unsigned char> >*, starrocks::vectorized::ArrowConvertContext*, starrocks::TypeDescriptor const*) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:934
    #3 0xaaa898c in starrocks::vectorized::ArrowListConverter::apply(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, std::vector<unsigned char, std::allocator<unsigned char> >*, starrocks::vectorized::ArrowConvertContext*, starrocks::TypeDescriptor const*) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:942
    #4 0xa9f27d7 in starrocks::vectorized::ParquetScanner::convert_array_to_column(starrocks::Status (*)(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, unsigned char*, starrocks::vectorized::ArrowConvertContext*), unsigned long, arrow::Array const*, starrocks::TypeDescriptor const*, std::shared_ptr<starrocks::vectorized::Column> const&) /root/starrocks/be/src/exec/vectorized/parquet_scanner.cpp:241
    #5 0xa9ef3ac in starrocks::vectorized::ParquetScanner::append_batch_to_src_chunk(std::shared_ptr<starrocks::vectorized::Chunk>*) /root/starrocks/be/src/exec/vectorized/parquet_scanner.cpp:100
    #6 0xa9f33ff in starrocks::vectorized::ParquetScanner::get_next() /root/starrocks/be/src/exec/vectorized/parquet_scanner.cpp:298
    #7 0x1031e337 in starrocks::connector::FileDataSource::get_next(starrocks::RuntimeState*, std::shared_ptr<starrocks::vectorized::Chunk>*) /root/starrocks/be/src/connector/file_connector.cpp:96
    #8 0xa13c0a8 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::vectorized::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:200
    #9 0xa0d0e24 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:69
    #10 0x941fdc4 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:350
    #11 0x9425411 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:60
    #12 0x94252bf in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:110
    #13 0x9425134 in _M_invoke /usr/include/c++/10.3.0/bits/std_function.h:291
    #14 0x9009be1 in std::function<void ()>::operator()() const /usr/include/c++/10.3.0/bits/std_function.h:622
    #15 0x9481719 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:48
    #16 0x9480f63 in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:19
    #17 0x9482973 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:60
    #18 0x948263b in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:110
    #19 0x94821b0 in _M_invoke /usr/include/c++/10.3.0/bits/std_function.h:291
    #20 0x9009be1 in std::function<void ()>::operator()() const /usr/include/c++/10.3.0/bits/std_function.h:622
    #21 0xf4d22ad in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:44
    #22 0xf4cf0af in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:536
    #23 0xf4eacbb in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /usr/include/c++/10.3.0/bits/invoke.h:73
    #24 0xf4ea614 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /usr/include/c++/10.3.0/bits/invoke.h:95
    #25 0xf4e9a0b in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/10.3.0/functional:416
    #26 0xf4e836d in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /usr/include/c++/10.3.0/functional:499
    #27 0xf4e53d1 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /usr/include/c++/10.3.0/bits/invoke.h:60
    #28 0xf4e2d35 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /usr/include/c++/10.3.0/bits/invoke.h:110
    #29 0xf4ded9e in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /usr/include/c++/10.3.0/bits/std_function.h:291
    #30 0x9009be1 in std::function<void ()>::operator()() const /usr/include/c++/10.3.0/bits/std_function.h:622
    #31 0xf4b8c8e in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:326
    #32 0x7ffb94bce44a in start_thread (/lib64/libpthread.so.0+0x744a)
    #33 0x7ffb941bd40e in __GI___clone (/lib64/libc.so.6+0xed40e)

0x61f000dc009e is located 68 bytes to the right of 3034-byte region [0x61f000dbf480,0x61f000dc005a)
allocated by thread T504 (con_wg_scan_io) here:
    #0 0x8e82e77 in operator new(unsigned long) ../../.././libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x8ff1b4e in __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) /usr/include/c++/10.3.0/ext/new_allocator.h:115
    #2 0x8ff18a7 in std::allocator_traits<std::allocator<unsigned char> >::allocate(std::allocator<unsigned char>&, unsigned long) /home/work/tools/gcc-10.3.0/include/c++/10.3.0/bits/alloc_traits.h:460
    #3 0x8ff0ab5 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) /home/work/tools/gcc-10.3.0/include/c++/10.3.0/bits/stl_vector.h:346
    #4 0x900dc84 in std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_append(unsigned long) /usr/include/c++/10.3.0/bits/vector.tcc:635
    #5 0x9009f8c in std::vector<unsigned char, std::allocator<unsigned char> >::resize(unsigned long) /usr/include/c++/10.3.0/bits/stl_vector.h:940
    #6 0x900941e in starrocks::vectorized::FixedLengthColumnBase<unsigned char>::resize(unsigned long) /root/starrocks/be/src/column/fixed_length_column_base.h:81
    #7 0xaa97cc1 in starrocks::vectorized::fill_null_column(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::FixedLengthColumn<unsigned char>*, unsigned long) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:64
    #8 0xaaa8174 in starrocks::vectorized::ArrowListConverter::convert_list(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, std::vector<unsigned char, std::allocator<unsigned char> >*, starrocks::vectorized::ArrowConvertContext*, starrocks::TypeDescriptor const*) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:900
    #9 0xaaa88b3 in starrocks::vectorized::ArrowListConverter::convert_list_with_null(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, std::vector<unsigned char, std::allocator<unsigned char> >*, starrocks::vectorized::ArrowConvertContext*, starrocks::TypeDescriptor const*) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:934
    #10 0xaaa898c in starrocks::vectorized::ArrowListConverter::apply(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, std::vector<unsigned char, std::allocator<unsigned char> >*, starrocks::vectorized::ArrowConvertContext*, starrocks::TypeDescriptor const*) /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:942
    #11 0xa9f27d7 in starrocks::vectorized::ParquetScanner::convert_array_to_column(starrocks::Status (*)(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::Column*, unsigned long, unsigned char*, unsigned char*, starrocks::vectorized::ArrowConvertContext*), unsigned long, arrow::Array const*, starrocks::TypeDescriptor const*, std::shared_ptr<starrocks::vectorized::Column> const&) /root/starrocks/be/src/exec/vectorized/parquet_scanner.cpp:241
    #12 0xa9ef3ac in starrocks::vectorized::ParquetScanner::append_batch_to_src_chunk(std::shared_ptr<starrocks::vectorized::Chunk>*) /root/starrocks/be/src/exec/vectorized/parquet_scanner.cpp:100
    #13 0xa9f33ff in starrocks::vectorized::ParquetScanner::get_next() /root/starrocks/be/src/exec/vectorized/parquet_scanner.cpp:298
    #14 0x1031e337 in starrocks::connector::FileDataSource::get_next(starrocks::RuntimeState*, std::shared_ptr<starrocks::vectorized::Chunk>*) /root/starrocks/be/src/connector/file_connector.cpp:96
    #15 0xa13c0a8 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::vectorized::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:200
    #16 0xa0d0e24 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:69
    #17 0x941fdc4 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:350
    #18 0x9425411 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:60
    #19 0x94252bf in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:110
    #20 0x9425134 in _M_invoke /usr/include/c++/10.3.0/bits/std_function.h:291
    #21 0x9009be1 in std::function<void ()>::operator()() const /usr/include/c++/10.3.0/bits/std_function.h:622
    #22 0x9481719 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:48
    #23 0x9480f63 in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:19
    #24 0x9482973 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:60
    #25 0x948263b in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /usr/include/c++/10.3.0/bits/invoke.h:110
    #26 0x94821b0 in _M_invoke /usr/include/c++/10.3.0/bits/std_function.h:291
    #27 0x9009be1 in std::function<void ()>::operator()() const /usr/include/c++/10.3.0/bits/std_function.h:622
    #28 0xf4d22ad in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:44
    #29 0xf4cf0af in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:536

Thread T504 (con_wg_scan_io) created by T0 here:
    #0 0x8e2cb72 in __interceptor_pthread_create ../../.././libsanitizer/asan/asan_interceptors.cpp:214
    #1 0xf4b80e2 in starrocks::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&, unsigned long, scoped_refptr<starrocks::Thread>*) /root/starrocks/be/src/util/thread.cpp:281
    #2 0xf4d58ae in starrocks::Status starrocks::Thread::create<void (starrocks::ThreadPool::*)(), starrocks::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (starrocks::ThreadPool::* const&)(), starrocks::ThreadPool* const&, scoped_refptr<starrocks::Thread>*) /root/starrocks/be/src/util/thread.h:55
    #3 0xf4d034d in starrocks::ThreadPool::create_thread() /root/starrocks/be/src/util/threadpool.cpp:587
    #4 0xf4cc5aa in starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority) /root/starrocks/be/src/util/threadpool.cpp:423
    #5 0xf4cb081 in starrocks::ThreadPool::submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPool::Priority) /root/starrocks/be/src/util/threadpool.cpp:332
    #6 0xf4cb222 in starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority) /root/starrocks/be/src/util/threadpool.cpp:336
    #7 0x94810bd in starrocks::workgroup::ScanExecutor::initialize(int) /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:19
    #8 0xeeedccf in starrocks::ExecEnv::_init(std::vector<starrocks::StorePath, std::allocator<starrocks::StorePath> > const&) /root/starrocks/be/src/runtime/exec_env.cpp:222
    #9 0xeeeaf73 in starrocks::ExecEnv::init(starrocks::ExecEnv*, std::vector<starrocks::StorePath, std::allocator<starrocks::StorePath> > const&) /root/starrocks/be/src/runtime/exec_env.cpp:126
    #10 0x8ec39b0 in main /root/starrocks/be/src/service/starrocks_main.cpp:282
    #11 0x7ffb940f1139 in __libc_start_main (/lib64/libc.so.6+0x21139)

SUMMARY: AddressSanitizer: heap-buffer-overflow /root/starrocks/be/src/exec/vectorized/arrow_to_starrocks_converter.cpp:69 in starrocks::vectorized::fill_null_column(arrow::Array const*, unsigned long, unsigned long, starrocks::vectorized::FixedLengthColumn<unsigned char>*, unsigned long)
Shadow bytes around the buggy address:
  0x0c3e801affc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3e801affd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3e801affe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3e801afff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3e801b0000: 00 00 00 00 00 00 00 00 00 00 00 02 fa fa fa fa
=>0x0c3e801b0010: fa fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3e801b0020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3e801b0030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3e801b0040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3e801b0050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3e801b0060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==41735==ABORTING
@chaoyli chaoyli added the type/bug Something isn't working label Feb 14, 2023
@haosuinca
Copy link

haosuinca commented Feb 15, 2023 •
edited

schema in starrocks

| core_cs_ticket__snapshot | CREATE TABLE `core_cs_ticket__snapshot` (
  `id` bigint(20) NULL COMMENT "",
  `id_requester` bigint(20) NULL COMMENT "",
  `id_assignee` bigint(20) NULL COMMENT "",
  `id_group` bigint(20) NULL COMMENT "",
  `id_message_thread` bigint(20) NULL COMMENT "",
  `id_homes_reservation_code` bigint(20) NULL COMMENT "",
  `id_experiences_reservation_code` bigint(20) NULL COMMENT "",
  `id_listing` bigint(20) NULL COMMENT "",
  `id_experience` bigint(20) NULL COMMENT "",
  `id_topic` bigint(20) NULL COMMENT "",
  `ts_created_at_utc` varchar(1000000) NULL COMMENT "",
  `ts_assigned_at_utc` varchar(1000000) NULL COMMENT "",
  `ts_solved_at_utc` varchar(1000000) NULL COMMENT "",
  `ts_closed_at_utc` varchar(1000000) NULL COMMENT "",
  `ts_status_updated_utc` varchar(1000000) NULL COMMENT "",
  `ts_updated_at_utc` varchar(1000000) NULL COMMENT "",
  `dim_is_cs_ticket` tinyint(4) NULL COMMENT "",
  `dim_is_volume_service_group` tinyint(4) NULL COMMENT "",
  `dim_is_safety_service_group` tinyint(4) NULL COMMENT "",
  `dim_is_claims_service_group` tinyint(4) NULL COMMENT "",
  `dim_is_trust` tinyint(4) NULL COMMENT "",
  `dim_is_payments` tinyint(4) NULL COMMENT "",
  `dim_is_closed_by_merge` tinyint(4) NULL COMMENT "",
  `dim_is_requires_manual_close` tinyint(4) NULL COMMENT "",
  `dim_is_directly_routed` tinyint(4) NULL COMMENT "",
  `dim_is_directly_solved` tinyint(4) NULL COMMENT "",
  `dim_is_directly_escalated` tinyint(4) NULL COMMENT "",
  `dim_group` varchar(1000000) NULL COMMENT "",
  `dim_status` varchar(1000000) NULL COMMENT "",
  `dim_topic_service_group` varchar(1000000) NULL COMMENT "",
  `dim_created_via` varchar(1000000) NULL COMMENT "",
  `dim_selected_channel` varchar(1000000) NULL COMMENT "",
  `dim_requester_user_role` varchar(1000000) NULL COMMENT "",
  `dim_tags` varchar(1000000) NULL COMMENT "",
  `dim_inbound_or_outbound` varchar(1000000) NULL COMMENT "",
  `dim_homes_confirmation_code` varchar(1000000) NULL COMMENT "",
  `dim_experiences_confirmation_code` varchar(1000000) NULL COMMENT "",
  `dim_topic_path` varchar(1000000) NULL COMMENT "",
  `dim_topic_parent` varchar(1000000) NULL COMMENT "",
  `dim_topic_child` varchar(1000000) NULL COMMENT "",
  `dim_topic_grandchild` varchar(1000000) NULL COMMENT "",
  `dim_topic_terminal_node` varchar(1000000) NULL COMMENT "",
  `dim_user_issue` varchar(1000000) NULL COMMENT "",
  `dim_trust_tier` varchar(1000000) NULL COMMENT "",
  `dim_requester_language_preferred` varchar(1000000) NULL COMMENT "",
  `dim_business_unit` varchar(1000000) NULL COMMENT "",
  `m_groups` bigint(20) NULL COMMENT "",
  `m_assignees` bigint(20) NULL COMMENT "",
  `m_pendings` bigint(20) NULL COMMENT "",
  `m_reopens` bigint(20) NULL COMMENT "",
  `m_solves` bigint(20) NULL COMMENT "",
  `m_messages` bigint(20) NULL COMMENT "",
  `m_phone_calls` bigint(20) NULL COMMENT "",
  `m_phone_queue_attempts` bigint(20) NULL COMMENT "",
  `m_groups_transfers` bigint(20) NULL COMMENT "",
  `m_time_to_first_touch_sec` bigint(20) NULL COMMENT "",
  `m_total_resolution_time_sec` bigint(20) NULL COMMENT "",
  `ts_first_group_assigned_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_assigned_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_solved_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_message_utc` varchar(1000000) NULL COMMENT "",
  `ts_last_message_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_agent_message_utc` varchar(1000000) NULL COMMENT "",
  `ts_last_agent_message_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_customer_message_utc` varchar(1000000) NULL COMMENT "",
  `ts_last_customer_message_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_call_utc` varchar(1000000) NULL COMMENT "",
  `ts_last_call_utc` varchar(1000000) NULL COMMENT "",
  `id_assignee_first` varchar(1000000) NULL COMMENT "",
  `dim_group_name_first` varchar(1000000) NULL COMMENT "",
  `id_group_first` bigint(20) NULL COMMENT "",
  `dim_topic_service_group_first` varchar(1000000) NULL COMMENT "",
  `dim_topic_path_first` varchar(1000000) NULL COMMENT "",
  `dim_homes_confirmation_codes_all` ARRAY<varchar(1000000)> NULL COMMENT "",
  `dim_experiences_confirmation_codes_all` ARRAY<varchar(1000000)> NULL COMMENT "",
  `dim_phone_call_ids` ARRAY<varchar(1000000)> NULL COMMENT "",
  `dim_merged_into_id` bigint(20) NULL COMMENT "",
  `dim_merged_from_ids` ARRAY<bigint(20)> NULL COMMENT "",
  `uri_contact_reason` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_service_group` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_path` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_parent` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_child` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_grandchild` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_terminal_node` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_service_group_first` varchar(1000000) NULL COMMENT "",
  `dim_contact_reason_path_first` varchar(1000000) NULL COMMENT "",
  `id_requester_zendesk` bigint(20) NULL COMMENT "",
  `id_assignee_zendesk` bigint(20) NULL COMMENT "",
  `id_assignee_zendesk_first` varchar(1000000) NULL COMMENT "",
  `ds_created_at_utc` date NULL COMMENT "",
  `ds_assigned_at_utc` date NULL COMMENT "",
  `ds_solved_at_utc` date NULL COMMENT "",
  `ds_updated_at_utc` date NULL COMMENT "",
  `dim_selected_channel_name` varchar(1000000) NULL COMMENT "",
  `dim_is_solved_ticket` bigint(20) NULL COMMENT "",
  `dim_is_assigned` bigint(20) NULL COMMENT "",
  `ts_first_customer_contact_utc` varchar(1000000) NULL COMMENT "",
  `ts_last_customer_contact_utc` varchar(1000000) NULL COMMENT "",
  `ts_first_agent_contact_utc` varchar(1000000) NULL COMMENT "",
  `ts_last_agent_contact_utc` varchar(1000000) NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`id`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`) BUCKETS 32
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "DEFAULT",
"enable_persistent_index" = "false",
"compression" = "LZ4"
); |

@haosuinca
Copy link

haosuinca commented Feb 15, 2023 •
edited

parquet schema:

required group field_id=-1 hive_schema {
  optional int64 field_id=-1 id;
  optional int64 field_id=-1 id_requester;
  optional int64 field_id=-1 id_assignee;
  optional int64 field_id=-1 id_group;
  optional int64 field_id=-1 id_message_thread;
  optional int64 field_id=-1 id_homes_reservation_code;
  optional int64 field_id=-1 id_experiences_reservation_code;
  optional int64 field_id=-1 id_listing;
  optional int64 field_id=-1 id_experience;
  optional int64 field_id=-1 id_topic;
  optional binary field_id=-1 ts_created_at_utc (String);
  optional binary field_id=-1 ts_assigned_at_utc (String);
  optional binary field_id=-1 ts_solved_at_utc (String);
  optional binary field_id=-1 ts_closed_at_utc (String);
  optional binary field_id=-1 ts_status_updated_utc (String);
  optional binary field_id=-1 ts_updated_at_utc (String);
  optional int32 field_id=-1 dim_is_cs_ticket;
  optional int32 field_id=-1 dim_is_volume_service_group;
  optional int32 field_id=-1 dim_is_safety_service_group;
  optional int32 field_id=-1 dim_is_claims_service_group;
  optional int32 field_id=-1 dim_is_trust;
  optional int32 field_id=-1 dim_is_payments;
  optional int32 field_id=-1 dim_is_closed_by_merge;
  optional int32 field_id=-1 dim_is_requires_manual_close;
  optional int32 field_id=-1 dim_is_directly_routed;
  optional int32 field_id=-1 dim_is_directly_solved;
  optional int32 field_id=-1 dim_is_directly_escalated;
  optional binary field_id=-1 dim_group (String);
  optional binary field_id=-1 dim_status (String);
  optional binary field_id=-1 dim_topic_service_group (String);
  optional binary field_id=-1 dim_created_via (String);
  optional binary field_id=-1 dim_selected_channel (String);
  optional binary field_id=-1 dim_requester_user_role (String);
  optional binary field_id=-1 dim_tags (String);
  optional binary field_id=-1 dim_inbound_or_outbound (String);
  optional binary field_id=-1 dim_homes_confirmation_code (String);
  optional binary field_id=-1 dim_experiences_confirmation_code (String);
  optional binary field_id=-1 dim_topic_path (String);
  optional binary field_id=-1 dim_topic_parent (String);
  optional binary field_id=-1 dim_topic_child (String);
  optional binary field_id=-1 dim_topic_grandchild (String);
  optional binary field_id=-1 dim_topic_terminal_node (String);
  optional binary field_id=-1 dim_user_issue (String);
  optional binary field_id=-1 dim_trust_tier (String);
  optional binary field_id=-1 dim_requester_language_preferred (String);
  optional binary field_id=-1 dim_business_unit (String);
  optional int64 field_id=-1 m_groups;
  optional int64 field_id=-1 m_assignees;
  optional int64 field_id=-1 m_pendings;
  optional int64 field_id=-1 m_reopens;
  optional int64 field_id=-1 m_solves;
  optional int64 field_id=-1 m_messages;
  optional int64 field_id=-1 m_phone_calls;
  optional int64 field_id=-1 m_phone_queue_attempts;
  optional int64 field_id=-1 m_groups_transfers;
  optional int64 field_id=-1 m_time_to_first_touch_sec;
  optional int64 field_id=-1 m_total_resolution_time_sec;
  optional binary field_id=-1 ts_first_group_assigned_utc (String);
  optional binary field_id=-1 ts_first_assigned_utc (String);
  optional binary field_id=-1 ts_first_solved_utc (String);
  optional binary field_id=-1 ts_first_message_utc (String);
  optional binary field_id=-1 ts_last_message_utc (String);
  optional binary field_id=-1 ts_first_agent_message_utc (String);
  optional binary field_id=-1 ts_last_agent_message_utc (String);
  optional binary field_id=-1 ts_first_customer_message_utc (String);
  optional binary field_id=-1 ts_last_customer_message_utc (String);
  optional binary field_id=-1 ts_first_call_utc (String);
  optional binary field_id=-1 ts_last_call_utc (String);
  optional binary field_id=-1 id_assignee_first (String);
  optional binary field_id=-1 dim_group_name_first (String);
  optional int64 field_id=-1 id_group_first;
  optional binary field_id=-1 dim_topic_service_group_first (String);
  optional binary field_id=-1 dim_topic_path_first (String);
  optional group field_id=-1 dim_homes_confirmation_codes_all (List) {
    repeated group field_id=-1 bag {
      optional binary field_id=-1 array_element (String);
    }
  }
  optional group field_id=-1 dim_experiences_confirmation_codes_all (List) {
    repeated group field_id=-1 bag {
      optional binary field_id=-1 array_element (String);
    }
  }
  optional group field_id=-1 dim_phone_call_ids (List) {
    repeated group field_id=-1 bag {
      optional binary field_id=-1 array_element (String);
    }
  }
  optional int64 field_id=-1 dim_merged_into_id;
  optional group field_id=-1 dim_merged_from_ids (List) {
    repeated group field_id=-1 bag {
      optional int64 field_id=-1 array_element;
    }
  }
  optional binary field_id=-1 uri_contact_reason (String);
  optional binary field_id=-1 dim_contact_reason_service_group (String);
  optional binary field_id=-1 dim_contact_reason_path (String);
  optional binary field_id=-1 dim_contact_reason_parent (String);
  optional binary field_id=-1 dim_contact_reason_child (String);
  optional binary field_id=-1 dim_contact_reason_grandchild (String);
  optional binary field_id=-1 dim_contact_reason_terminal_node (String);
  optional binary field_id=-1 dim_contact_reason_service_group_first (String);
  optional binary field_id=-1 dim_contact_reason_path_first (String);
  optional int64 field_id=-1 id_requester_zendesk;
  optional int64 field_id=-1 id_assignee_zendesk;
  optional binary field_id=-1 id_assignee_zendesk_first (String);
  optional binary field_id=-1 ds_created_at_utc (String);
  optional binary field_id=-1 ds_assigned_at_utc (String);
  optional binary field_id=-1 ds_solved_at_utc (String);
  optional binary field_id=-1 ds_updated_at_utc (String);
  optional binary field_id=-1 dim_selected_channel_name (String);
  optional int64 field_id=-1 dim_is_solved_ticket;
  optional int64 field_id=-1 dim_is_assigned;
  optional binary field_id=-1 ts_first_customer_contact_utc (String);
  optional binary field_id=-1 ts_last_customer_contact_utc (String);
  optional binary field_id=-1 ts_first_agent_contact_utc (String);
  optional binary field_id=-1 ts_last_agent_contact_utc (String);
}

@xiaoyong-z xiaoyong-z mentioned this issue Feb 15, 2023
Merged
16 tasks
@xiaoyong-z xiaoyong-z mentioned this issue Feb 17, 2023
Merged
16 tasks
@chaoyli
Copy link
Contributor Author

chaoyli commented Mar 21, 2023 

query_id:3bf726e6-a43f-4fa7-a205-3e428f6edd27, fragment_instance:3bf726e6-a43f-4fa7-a205-3e428f6edd34
*** Aborted at 1675895492 (unix time) try "date -d @1675895492" if you are using GNU date ***
PC: @          0x7ae9c72 jemalloc_usable_size
*** SIGSEGV (@0x2a450) received by PID 47600 (TID 0x7fd6133ee700) from PID 173136; stack trace: ***
    @          0x57bd8c2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fd7792b88e0 (unknown)
    @          0x7ae9c72 jemalloc_usable_size
    @          0x47abd99 free
    @          0x2cec0d5 std::vector<>::_M_default_append()
    @          0x31ccdd6 starrocks::vectorized::ArrowListConverter::convert_list()
    @          0x31cd9a8 starrocks::vectorized::ArrowListConverter::apply()
    @          0x319c0a9 starrocks::vectorized::ParquetScanner::convert_array_to_column()
    @          0x319dc0f starrocks::vectorized::ParquetScanner::append_batch_to_src_chunk()
    @          0x319ebb5 starrocks::vectorized::ParquetScanner::get_next()
    @          0x4dc8131 starrocks::connector::FileDataSource::get_next()
    @          0x2fdb005 starrocks::pipeline::ConnectorChunkSource::_read_chunk()
    @          0x2fc4e6c starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking()
    @          0x2d47b04 _ZZN9starrocks8pipeline12ScanOperator18_trigger_next_scanEPNS_12RuntimeStateEiENKUlvE_clEv
    @          0x2d58b3d starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x488224d starrocks::ThreadPool::dispatch_thread()
    @          0x487cfda starrocks::Thread::supervise_thread()
    @     0x7fd7792ae44b start_thread
    @     0x7fd77889d40f __GI___clone
    @                0x0 (unknown)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[BugFix] fix parquet array load crash bug xiaoyong-z/starrocks
2 participants
@haosuinca @chaoyli