[fix] Fixed length error in compress.cpp#48139
Closed
lzyy2024 wants to merge 487 commits intoapache:masterfrom
Closed
[fix] Fixed length error in compress.cpp#48139lzyy2024 wants to merge 487 commits intoapache:masterfrom
lzyy2024 wants to merge 487 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31473 ms |
TPC-DS: Total hot run time: 183675 ms |
ClickBench: Total hot run time: 30.05 s |
|
TeamCity be ut coverage result: |
yiguolei
reviewed
Feb 21, 2025
| // Z_MEM_ERROR and Z_BUF_ERROR are already handled in compress, making sure st is always Z_OK | ||
| RETURN_IF_ERROR(compression_codec->compress(data, &compressed_str)); | ||
| col_data.resize(col_data.size() + 4 + compressed_str.size()); | ||
| col_data.resize(col_data.size() + compressed_str_length + compressed_str.size()); |
Contributor
There was a problem hiding this comment.
please add some SQL based groovy test and beut for this function.
…arameter does not work (apache#47578) ### What problem does this PR solve? Related PR: apache#47262 Problem Summary: Starting thirdpaty script with only the reserve-ports parameter does not work ```bash ./run-thirdparties-docker.sh --reserve-ports ```
### What problem does this PR solve? Use prometheus to calculate average value is better. Related PR: apache#43144 For example, we use `task_execution_time_ns_avg_in_last_1000_times` which is equal to `SUM(cost 0, ... cost 999) / 1000` to represent average execution time, it has two problems: 1. Update of its data source `_task_execution_time_ns_statistic` acquires lock. 2. Result of `task_execution_time_ns_avg_in_last_1000_times` is not zero if we just finished a set of tasks and no more tasks to run. For example, we have a continuous straight line after all tasks have finished for a while. <img width="416" alt="image" src="https://github.com/user-attachments/assets/e874a077-1e74-4700-9dd8-4cf9625bc8f8" /> The problem can be fixed by: 1. Using `task_execution_time_ns_total` an atomic counter to store total sum of execution time of each iteration. 2. With the help of `irate` function of prometheus, we can have an equivalent substitution like `irate(doris_be_task_execution_time_ns_total[$__rate_interval])/doris_be_thread_pool_active_threads` <img width="349" alt="image" src="https://github.com/user-attachments/assets/2b0b41bd-8709-4cab-84c1-cf11c4cc3ac9" /> After all tasks finished, the curve will be zero, this is more reasonable.
…d_on_limit (apache#46993) ### What problem does this PR solve? adaptive_pipeline_task_serial_read_on_limit is unstable, since profile is too much. check profile content after query finished instead of checking after all test sql finished.
For a scan operator in a query, FE will assign a tablet id to a specific scan operator and then it will be search by storage engine during execution. But this searching process will be done after runtime filter reached. However, if we wait for a runtime filter for a long time and compaction / balance tasks is done at the same time, a tablet / rowset will be lost and we got an error.
… 1 (apache#47586) ### What problem does this PR solve? fix wrong result when percentile's second argument is 1 related with apache#34382 ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…apache#47565) Use single tablet table to verify compaction score action, to ensure compaction score on tablets is predictable.
…paction by default" (apache#47521) Reverts apache#47377
evict in advance if current cache size is over threshold to avoid sync evict during query, which may affect query performance.
apache#47588) …RGlCA: No such file or directory"
selectdb/ccr-syncer#396 force_replace flag will only replace table with different schema except for non-OLAP tables
…t crash (apache#47610) ### What problem does this PR solve? *** Query id: 5447701417c13e4e-cea25b10f284c6a5 *** *** is nereids: 0 *** *** tablet id: 1738818748602 *** *** Aborted at 1738820047 (unix time) try "date -d @1738820047" if you are using GNU date *** *** Current BE git commitID: 512681c *** *** SIGSEGV invalid permissions for mapped object (@0x7f112a5df53f) received by PID 6310 (TID 6765 OR 0x7f1384ed3640) from PID 710800703; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F14815CC520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::vectorized::ColumnVector<unsigned char>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/columns/column_vector.cpp:323 5# doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*, std::vector<int, std::allocator<int> > const*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/core/block.cpp:1036 6# doris::MemTable::_put_into_output(doris::vectorized::Block&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/memtable.cpp:257 7# doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/memtable.cpp:513 8# doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/memtable.cpp:532 9# doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/memtable_flush_executor.cpp:144 10# doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, int, long) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 11# doris::MemtableFlushTask::run() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/memtable_flush_executor.cpp:60 12# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 13# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499 14# start_thread at ./nptl/pthread_create.c:442 15# 0x00007F14816B0850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 Problem Summary: - When memtable insert fails (e.g., due to memory allocation failure during add_rows), the memtable is left in an inconsistent state - Under memory pressure, the system might trigger a flush operation on this failed memtable, leading to crashes Solution: - Reset memtable immediately after insert failure
) ### What problem does this PR solve? Related PR: apache#45255
…ties backup and restore (apache#41925) ## Proposed changes Issue Number: close #xxx When backup and restore, the database properties remain
…view alter on transparent rewriting (apache#47310)
…group_by_key_by_uniform (apache#48119) Related PR: apache#46225 apache#46223
…#48107) Introduced by apache#47702 F20250218 08:55:18.762200 10135 assert_cast.h:57] Bad cast from type:doris::vectorized::ColumnNullable to doris::vectorized::ColumnStr *** Check failure stack trace: *** @ 0x55bad64c6436 google::LogMessage::SendToLog() @ 0x55bad64c2e80 google::LogMessage::Flush() @ 0x55bad64c6c79 google::LogMessageFatal::~LogMessageFatal() @ 0x55bacb081956 ZZ11assert_castIRKN5doris10vectorized9ColumnStrIjEEL18TypeCheckOnRelease1ERKNS1_7IColumnEET_OT1_ENKUlOSA_E_clIS9_EES5_SD @ 0x55bacb081797 assert_cast<>() @ 0x55bacb07f4ef doris::vectorized::ColumnStr<>::insert_from() @ 0x55bad00b7317 doris::vectorized::IColumn::insert_from_multi_column() @ 0x55bad5555bc1 doris::vectorized::VSortedRunMerger::get_next()::$_1::operator()() @ 0x55bad55551ef doris::vectorized::VSortedRunMerger::get_next() @ 0x55bad552fbc8 doris::vectorized::VDataStreamRecvr::get_next() @ 0x55bad626bf83 doris::pipeline::ExchangeSourceOperatorX::get_block() @ 0x55bad55fba4e doris::pipeline::OperatorXBase::get_block_after_projects() @ 0x55bad63ea221 doris::pipeline::PipelineTask::execute() @ 0x55bad63fa3fd doris::pipeline::TaskScheduler::_do_work() @ 0x55bacbffc90a doris::ThreadPool::dispatch_thread() @ 0x55bacbff0ea1 doris::Thread::supervise_thread() @ 0x7f945ba99ac3 (unknown) @ 0x7f945bb2b850 (unknown) @ (nil) (unknown) *** Query id: dac13a78b39642cb-9218eca2d5a11638 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1739840118 (unix time) try "date -d @1739840118" if you are using GNU date *** *** Current BE git commitID: apache@a8e18b0 *** *** SIGABRT unknown detail explain (@0x1cb6) received by PID 7350 (TID 10135 OR 0x7f909bbf6640) from PID 7350; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# 0x00007F945BA47520 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill at ./nptl/pthread_kill.c:89 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# 0x000055BAD64D0D0D in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 6# 0x000055BAD64C334A in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 8# google::LogMessage::Flush() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 10# doris::vectorized::ColumnStr const& assert_cast const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&)::{lambda(auto:1&&)https://github.com/apache/doris/issues/1}::operator()(doris::vectorized::IColumn const&) const at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/common/assert_cast.h:57 11# doris::vectorized::ColumnStr const& assert_cast const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/common/assert_cast.h:72 12# doris::vectorized::ColumnStr::insert_from(doris::vectorized::IColumn const&, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/columns/column_string.h:174 13# doris::vectorized::IColumn::insert_from_multi_column(std::vector > const&, std::vector >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/columns/column.cpp:51 14# doris::vectorized::VSortedRunMerger::get_next(doris::vectorized::Block*, bool*)::$_1::operator()() const at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vsorted_run_merger.cpp:172 15# doris::vectorized::VSortedRunMerger::get_next(doris::vectorized::Block*, bool*) in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 16# doris::vectorized::VDataStreamRecvr::get_next(doris::vectorized::Block*, bool*) in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 17# doris::pipeline::ExchangeSourceOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/exec/exchange_source_operator.cpp:160 18# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 19# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/pipeline_task.cpp:376 20# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/pipeline/task_scheduler.cpp:138 21# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be 22# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499 23# start_thread at ./nptl/pthread_create.c:442 24# 0x00007F945BB2B850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
…#48133) ### What problem does this PR solve? 1. Some aggregate functions, such as array_union, require array sort to ensure order in the output. 2. For the foreach function, if the input is an array, it is difficult to guarantee the order of the output. Therefore, some unstable cases have been removed, leaving only those related to scalars. Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
Remove the hash slot output judge empty logic, unless in nereids
…che#48137) Introduced a new local variable `escaped_suffix` to store the result of `escape_for_path_name(suffix_path)`. This change improves performance by avoiding repeated function calls.
### What problem does this PR solve? [fix](regression) Spark reads Doris data error
… organization and maintainability. (apache#48079) Problem Summary: This pull request includes several changes to the ICU (International Components for Unicode) integration within the project. The primary focus is on updating the build configuration, refactoring the ICU analyzer, and adding new ICU-related files.
…n inverted index filters (apache#47504) Problem Summary: select count() from httplogs where clientip match '232.71.0.0' and request match 'images'; IndexFilter: - HitRows: 0ns - fr_clientip: 10.392K (10392) - fr_request: 28.601172M (28601172) - ExecTime: 0ns - ft_clientip: 2.65ms - ft_request: 201.778ms FilteredRows: Represents the count of rows that met the filtering conditions post-index filtering. FilteredTime: Represents the time taken to complete the filtering operation.
apache#48005) Reverts apache#40245 since it's not useful
WRITE of size 1 at 0x6160007e86f0 thread T1983 (Pipe_normal [wo) #0 0x55fc8065b975 in std::__atomic_base<bool>::store(bool, std::memory_order) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:457:2 apache#1 0x55fc8065b975 in std::__atomic_base<bool>::operator=(bool) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:349:2 apache#2 0x55fc8065b975 in std::atomic<bool>::operator=(bool) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:80:22 apache#3 0x55fc8065b975 in doris::pipeline::PipelineTask::set_running(bool) /root/doris/be/src/pipeline/pipeline_task.h:192:47 apache#4 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int)::$_0::operator()() const /root/doris/be/src/pipeline/task_scheduler.cpp:121:23 apache#5 0x55fc8065b975 in doris::Defer<doris::pipeline::TaskScheduler::_do_work(int)::$_0>::~Defer() /root/doris/be/src/util/defer_op.h:37:16 apache#6 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:162:5 apache#7 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:608:24 apache#8 0x55fc4c55395e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#9 0x55fc4c55395e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#10 0x7f9ee3d25608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#11 0x7f9ee3fd2132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x6160007e86f0 is located 624 bytes inside of 632-byte region [0x6160007e8480,0x6160007e86f8) freed by thread T1981 (Pipe_normal [wo) here: #0 0x55fc47aa680d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3376e80d) (BuildId: 865149e62959581e) apache#1 0x55fc8059db84 in std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2 apache#2 0x55fc8059db84 in std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4 apache#3 0x55fc8059db84 in void std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#4 0x55fc8059db84 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 apache#5 0x55fc8059db84 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 apache#6 0x55fc8059db84 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*>(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 apache#7 0x55fc8059db84 in void std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >(std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >*, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 apache#8 0x55fc8059db84 in std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2 apache#9 0x55fc8052571c in void std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15 apache#10 0x55fc8052571c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7 apache#11 0x55fc8052571c in void std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6 apache#12 0x55fc8052571c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7 apache#13 0x55fc8052571c in void std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7 apache#14 0x55fc8052571c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6 apache#15 0x55fc8052571c in std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > >, std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> >, std::allocator<std::unique_ptr<doris::pipeline::PipelineTask, std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9 apache#16 0x55fc8052571c in doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext() /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:142:12 apache#17 0x55fc47ad30cc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6 apache#18 0x55fc80658d57 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11 apache#19 0x55fc80658d57 in std::__shared_ptr<doris::TaskExecutionContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31 apache#20 0x55fc80658d57 in doris::pipeline::close_task(doris::pipeline::PipelineTask*, doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:100:1 apache#21 0x55fc8065aa17 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:160:36 apache#22 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:608:24 apache#23 0x55fc4c55395e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9 apache#24 0x55fc4c55395e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5 apache#25 0x7f9ee3d25608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Previously, queries like SELECT COUNT(*) FROM table WHERE date='2017-10-01' required reading the date column in the first read phase, even though it was only used for filtering and not in the aggregation. This PR optimizes the execution plan to eliminate unnecessary column reads, improving performance.
### What problem does this PR solve? ``` [==========] Running 4 tests from 1 test suite. [----------] Global test environment set-up. [----------] 4 tests from WorkloadSchedPolicyTest [ RUN ] WorkloadSchedPolicyTest.one_policy_one_condition [ OK ] WorkloadSchedPolicyTest.one_policy_one_condition (51 ms) [ RUN ] WorkloadSchedPolicyTest.one_policy_mutl_condition [ OK ] WorkloadSchedPolicyTest.one_policy_mutl_condition (50 ms) [ RUN ] WorkloadSchedPolicyTest.test_operator [ OK ] WorkloadSchedPolicyTest.test_operator (0 ms) [ RUN ] WorkloadSchedPolicyTest.test_wg_id_set [ OK ] WorkloadSchedPolicyTest.test_wg_id_set (0 ms) [----------] 4 tests from WorkloadSchedPolicyTest (103 ms total) [----------] Global test environment tear-down [==========] 4 tests from 1 test suite ran. (104 ms total) [ PASSED ] 4 tests. === Finished. Gtest output: /mnt/disk2/zouxinyi/doris/core/be/ut_build_ASAN/gtest_output ```
lzyy2024
force-pushed
the
debug-compress_function
branch
from
668ad16 to
4591515
Compare
lzyy2024
requested review from
CalvinKirs,
dataroaring,
gavinchou,
morningman and
w41ter
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Fixed length error in compress.cpp
Issue Number: close #xxx
Related PR: #47307
Problem Summary:
The compressed string length should be represented by 4 bytes instead of 10, and I replaced the magic value with a constant
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)