New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support bulk loading from S3 compatible distributed storage #3
Labels
Milestone
Comments
good idea |
这个在下一个版本,我们会开发引入 |
morningman
pushed a commit
to morningman/doris
that referenced
this issue
Dec 10, 2019
Seek block when starts a ScanKey (apache#1828)
This is supported through #3003 |
wangbo
pushed a commit
to wangbo/incubator-doris
that referenced
this issue
Apr 24, 2020
Spark etl fe dev fix bitmap union bug
Closed
3 tasks
2 tasks
3 tasks
songchuangyuan
pushed a commit
to songchuangyuan/incubator-doris
that referenced
this issue
Mar 8, 2022
morrySnow
pushed a commit
to morrySnow/incubator-doris
that referenced
this issue
Mar 16, 2022
[test] add join add group-by case from trino
Henry2SS
added a commit
to Henry2SS/incubator-doris
that referenced
this issue
Mar 16, 2022
[JD][Release] Release version of jd-0.15-1.0.5 of Doris
yiguolei
pushed a commit
that referenced
this issue
Oct 23, 2023
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> : class MinmaxFunctionTraits { public: using BasePtr = MinMaxFuncBase*; template <PrimitiveType type, size_t N> static BasePtr get_function() { return new MinMaxNumFunc<std::conditional_t< type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I, vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>, typename PrimitiveTypeTraits<type>::CppType>>(); } }; (gdb) bt #0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 #1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 #2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61 #3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492 #4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504 #5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060 #6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188 #7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> ( this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
dutyu
pushed a commit
to dutyu/doris
that referenced
this issue
Oct 28, 2023
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> : class MinmaxFunctionTraits { public: using BasePtr = MinMaxFuncBase*; template <PrimitiveType type, size_t N> static BasePtr get_function() { return new MinMaxNumFunc<std::conditional_t< type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I, vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>, typename PrimitiveTypeTraits<type>::CppType>>(); } }; (gdb) bt #0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 apache#1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 apache#2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61 apache#3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492 apache#4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504 apache#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060 apache#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188 apache#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> ( this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
wsjz
pushed a commit
to wsjz/incubator-doris
that referenced
this issue
Nov 19, 2023
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> : class MinmaxFunctionTraits { public: using BasePtr = MinMaxFuncBase*; template <PrimitiveType type, size_t N> static BasePtr get_function() { return new MinMaxNumFunc<std::conditional_t< type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I, vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>, typename PrimitiveTypeTraits<type>::CppType>>(); } }; (gdb) bt #0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 #1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 apache#2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61 apache#3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492 apache#4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504 apache#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060 apache#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188 apache#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> ( this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
3 tasks
XuJianxu
pushed a commit
to XuJianxu/doris
that referenced
this issue
Dec 14, 2023
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> : class MinmaxFunctionTraits { public: using BasePtr = MinMaxFuncBase*; template <PrimitiveType type, size_t N> static BasePtr get_function() { return new MinMaxNumFunc<std::conditional_t< type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I, vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>, typename PrimitiveTypeTraits<type>::CppType>>(); } }; (gdb) bt #0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 apache#1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381 apache#2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61 apache#3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492 apache#4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504 apache#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060 apache#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188 apache#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> ( this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this issue
Dec 17, 2023
…_in_blocks` Otherwise using the stale `_row_in_blocks` will result in heap-use-after-free ``` ==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7 0 sp 0x7f62c595ef68 READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh) #0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33 apache#1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25 apache#2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14 apache#3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27 apache#4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13 apache#5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62 apache#6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16 apache#7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23 ```
eldenmoon
added a commit
that referenced
this issue
Dec 18, 2023
…_in_blocks` (#28536) Otherwise using the stale `_row_in_blocks` will result in heap-buffer-overflow ``` ==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7 0 sp 0x7f62c595ef68 READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh) #0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33 #1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25 #2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14 #3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27 #4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13 #5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62 #6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16 #7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23 ```
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this issue
Dec 19, 2023
…_in_blocks` (apache#28536) Otherwise using the stale `_row_in_blocks` will result in heap-buffer-overflow ``` ==2695213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62900122e210 at pc 0x56524744aecf bp 0x7f62c595ef7 0 sp 0x7f62c595ef68 READ of size 8 at 0x62900122e210 thread T1627 (MemTableFlushTh) #0 0x56524744aece in doris::vectorized::ColumnVector<long>::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_vector.cpp:378:33 apache#1 0x5652472a7538 in doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/columns/column_nullable.cpp:310:25 apache#2 0x56524782a62a in doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, unsigned int const*, unsigned int const*) /mnt/disk2/lihangyu/doris/be/src/vec/core/block.cpp:961:14 apache#3 0x565233f187ae in doris::MemTable::_put_into_output(doris::vectorized::Block&) /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:248:27 apache#4 0x565233f1db66 in doris::MemTable::to_block() /mnt/disk2/lihangyu/doris/be/src/olap/memtable.cpp:496:13 apache#5 0x565233efae60 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:121:62 apache#6 0x565233efc8d6 in doris::FlushToken::_flush_memtable(doris::MemTable*, int, long) /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:150:16 apache#7 0x565233f0c5eb in doris::MemtableFlushTask::run() /mnt/disk2/lihangyu/doris/be/src/olap/memtable_flush_executor.cpp:58:23 ```
3 tasks
Open
3 tasks
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
S3 compatible is more and more prevalent especially for cloud native applications, as a result, it's reasonable to encapsulate a S3 compatible wrapper to fs_broker to make Palo more popular on multiple clouds.
The text was updated successfully, but these errors were encountered: