[BugFix] fix large binary column overflow when column writer append #36281

luohaha · 2023-12-01T18:23:36Z

Why I'm doing:
In BinaryColumn, we use vector<uint32_t> _offset to store each rows offset in vector<uint8_t> _bytes. But if _bytes is large than 4GB, which means that offset will large than UINT32_MAX, offset overflow in uint32. To prevent this overflow, we add this check in pr #33521, and because overflow happens, check fail with BE crash.

template <typename T>
void BinaryColumnBase<T>::_build_slices() const {
    if constexpr (std::is_same_v<T, uint32_t>) {
        CHECK_LT(_bytes.size(), (size_t)UINT32_MAX) << "BinaryColumn size overflow";
    }

BE crash:

   @          0x5e50559 google::LogMessageFatal::~LogMessageFatal()
    @          0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices()
    @          0x2757dd5 starrocks::BinaryColumnBase<>::raw_data()
    @          0x480a0da starrocks::ScalarColumnWriter::append()
    @          0x480a33c starrocks::StringColumnWriter::append()
    @          0x43a2806 starrocks::SegmentWriter::append_chunk()
    @          0x4a98da0 starrocks::VerticalRowsetWriter::add_columns()
    @          0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data()
    @          0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group()
    @          0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data()
    @          0x4a76129 starrocks::VerticalCompactionTask::run_impl()
    @          0x4a6b6fc starrocks::CompactionTask::run()
    @          0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x4cc33d2 starrocks::ThreadPool::dispatch_thread()
    @          0x4cbde6a starrocks::Thread::supervise_thread()
    @     0x7f4658c6644b start_thread
    @     0x7f465803f40f __GI___clone
    @                0x0 (unknown)

What I'm doing:
Add byte size limit to dictionary speculate, and make sure dictionary speculate will not batch too large column and cause overflow.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

be/src/common/constexpr.h

be/src/storage/rowset/column_writer.cpp

be/src/common/constexpr.h

Signed-off-by: luohaha <18810541851@163.com>

github-actions · 2023-12-02T08:18:59Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2023-12-02T08:20:32Z

[BE Incremental Coverage Report]

✅ pass : 8 / 8 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/column/binary_column.cpp	1	1	100.00%	[]
🔵	src/storage/rowset/column_writer.cpp	7	7	100.00%	[]

github-actions · 2023-12-02T19:01:27Z

@Mergifyio backport branch-3.2

github-actions · 2023-12-02T19:01:28Z

@Mergifyio backport branch-3.1

mergify · 2023-12-02T19:01:51Z

backport branch-3.2

✅ Backports have been created

#36300 [BugFix] fix large binary column overflow when column writer append (backport #36281) has been created for branch branch-3.2

mergify · 2023-12-02T19:01:52Z

backport branch-3.1

✅ Backports have been created

#36299 [BugFix] fix large binary column overflow when column writer append (backport #36281) has been created for branch branch-3.1

…36281) Why I'm doing: In BinaryColumn, we use `vector<uint32_t> _offset` to store each rows offset in `vector<uint8_t> _bytes`. But if `_bytes` is large than 4GB, which means that offset will large than UINT32_MAX, offset overflow in `uint32`. To prevent this overflow, we add this check in pr #33521, and because overflow happens, check fail with BE crash. ``` template <typename T> void BinaryColumnBase<T>::_build_slices() const { if constexpr (std::is_same_v<T, uint32_t>) { CHECK_LT(_bytes.size(), (size_t)UINT32_MAX) << "BinaryColumn size overflow"; } ``` BE crash: ``` @ 0x5e50559 google::LogMessageFatal::~LogMessageFatal() @ 0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices() @ 0x2757dd5 starrocks::BinaryColumnBase<>::raw_data() @ 0x480a0da starrocks::ScalarColumnWriter::append() @ 0x480a33c starrocks::StringColumnWriter::append() @ 0x43a2806 starrocks::SegmentWriter::append_chunk() @ 0x4a98da0 starrocks::VerticalRowsetWriter::add_columns() @ 0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data() @ 0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group() @ 0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data() @ 0x4a76129 starrocks::VerticalCompactionTask::run_impl() @ 0x4a6b6fc starrocks::CompactionTask::run() @ 0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data @ 0x4cc33d2 starrocks::ThreadPool::dispatch_thread() @ 0x4cbde6a starrocks::Thread::supervise_thread() @ 0x7f4658c6644b start_thread @ 0x7f465803f40f __GI___clone @ 0x0 (unknown) ``` What I'm doing: Add byte size limit to dictionary speculate, and make sure dictionary speculate will not batch too large column and cause overflow. Signed-off-by: luohaha <18810541851@163.com> (cherry picked from commit f06e2c8)

…backport #36281) (#36299) Co-authored-by: Yixin Luo <18810541851@163.com>

…backport #36281) (#36300) Co-authored-by: Yixin Luo <18810541851@163.com>

…tarRocks#36281) Why I'm doing: In BinaryColumn, we use `vector<uint32_t> _offset` to store each rows offset in `vector<uint8_t> _bytes`. But if `_bytes` is large than 4GB, which means that offset will large than UINT32_MAX, offset overflow in `uint32`. To prevent this overflow, we add this check in pr StarRocks#33521, and because overflow happens, check fail with BE crash. ``` template <typename T> void BinaryColumnBase<T>::_build_slices() const { if constexpr (std::is_same_v<T, uint32_t>) { CHECK_LT(_bytes.size(), (size_t)UINT32_MAX) << "BinaryColumn size overflow"; } ``` BE crash: ``` @ 0x5e50559 google::LogMessageFatal::~LogMessageFatal() @ 0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices() @ 0x2757dd5 starrocks::BinaryColumnBase<>::raw_data() @ 0x480a0da starrocks::ScalarColumnWriter::append() @ 0x480a33c starrocks::StringColumnWriter::append() @ 0x43a2806 starrocks::SegmentWriter::append_chunk() @ 0x4a98da0 starrocks::VerticalRowsetWriter::add_columns() @ 0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data() @ 0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group() @ 0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data() @ 0x4a76129 starrocks::VerticalCompactionTask::run_impl() @ 0x4a6b6fc starrocks::CompactionTask::run() @ 0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data @ 0x4cc33d2 starrocks::ThreadPool::dispatch_thread() @ 0x4cbde6a starrocks::Thread::supervise_thread() @ 0x7f4658c6644b start_thread @ 0x7f465803f40f __GI___clone @ 0x0 (unknown) ``` What I'm doing: Add byte size limit to dictionary speculate, and make sure dictionary speculate will not batch too large column and cause overflow. Signed-off-by: luohaha <18810541851@163.com> Signed-off-by: 鼎昕 <zhangdingxin.zdx@alibaba-inc.com>

…backport #36281) (#36299) Co-authored-by: Yixin Luo <18810541851@163.com> (cherry picked from commit 2af81c3)

luohaha · 2023-12-12T19:15:30Z

https://github.com/Mergifyio backport branch-3.0

luohaha · 2023-12-12T19:15:36Z

https://github.com/Mergifyio backport branch-2.5

mergify · 2023-12-12T19:16:32Z

backport branch-3.0

✅ Backports have been created

#36901 [BugFix] fix large binary column overflow when column writer append (backport #36281) has been created for branch branch-3.0

mergify · 2023-12-12T19:16:34Z

backport branch-2.5

✅ Backports have been created

#36902 [BugFix] fix large binary column overflow when column writer append (backport #36281) has been created for branch branch-2.5

…36281) Why I'm doing: In BinaryColumn, we use `vector<uint32_t> _offset` to store each rows offset in `vector<uint8_t> _bytes`. But if `_bytes` is large than 4GB, which means that offset will large than UINT32_MAX, offset overflow in `uint32`. To prevent this overflow, we add this check in pr #33521, and because overflow happens, check fail with BE crash. ``` template <typename T> void BinaryColumnBase<T>::_build_slices() const { if constexpr (std::is_same_v<T, uint32_t>) { CHECK_LT(_bytes.size(), (size_t)UINT32_MAX) << "BinaryColumn size overflow"; } ``` BE crash: ``` @ 0x5e50559 google::LogMessageFatal::~LogMessageFatal() @ 0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices() @ 0x2757dd5 starrocks::BinaryColumnBase<>::raw_data() @ 0x480a0da starrocks::ScalarColumnWriter::append() @ 0x480a33c starrocks::StringColumnWriter::append() @ 0x43a2806 starrocks::SegmentWriter::append_chunk() @ 0x4a98da0 starrocks::VerticalRowsetWriter::add_columns() @ 0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data() @ 0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group() @ 0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data() @ 0x4a76129 starrocks::VerticalCompactionTask::run_impl() @ 0x4a6b6fc starrocks::CompactionTask::run() @ 0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data @ 0x4cc33d2 starrocks::ThreadPool::dispatch_thread() @ 0x4cbde6a starrocks::Thread::supervise_thread() @ 0x7f4658c6644b start_thread @ 0x7f465803f40f __GI___clone @ 0x0 (unknown) ``` What I'm doing: Add byte size limit to dictionary speculate, and make sure dictionary speculate will not batch too large column and cause overflow. Signed-off-by: luohaha <18810541851@163.com> (cherry picked from commit f06e2c8)

…backport #36281) (#36901) Co-authored-by: Yixin Luo <18810541851@163.com>

…backport #36281) (backport #36299) (#36900) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…36281) Why I'm doing: In BinaryColumn, we use `vector<uint32_t> _offset` to store each rows offset in `vector<uint8_t> _bytes`. But if `_bytes` is large than 4GB, which means that offset will large than UINT32_MAX, offset overflow in `uint32`. To prevent this overflow, we add this check in pr #33521, and because overflow happens, check fail with BE crash. ``` template <typename T> void BinaryColumnBase<T>::_build_slices() const { if constexpr (std::is_same_v<T, uint32_t>) { CHECK_LT(_bytes.size(), (size_t)UINT32_MAX) << "BinaryColumn size overflow"; } ``` BE crash: ``` @ 0x5e50559 google::LogMessageFatal::~LogMessageFatal() @ 0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices() @ 0x2757dd5 starrocks::BinaryColumnBase<>::raw_data() @ 0x480a0da starrocks::ScalarColumnWriter::append() @ 0x480a33c starrocks::StringColumnWriter::append() @ 0x43a2806 starrocks::SegmentWriter::append_chunk() @ 0x4a98da0 starrocks::VerticalRowsetWriter::add_columns() @ 0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data() @ 0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group() @ 0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data() @ 0x4a76129 starrocks::VerticalCompactionTask::run_impl() @ 0x4a6b6fc starrocks::CompactionTask::run() @ 0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data @ 0x4cc33d2 starrocks::ThreadPool::dispatch_thread() @ 0x4cbde6a starrocks::Thread::supervise_thread() @ 0x7f4658c6644b start_thread @ 0x7f465803f40f __GI___clone @ 0x0 (unknown) ``` What I'm doing: Add byte size limit to dictionary speculate, and make sure dictionary speculate will not batch too large column and cause overflow. Signed-off-by: luohaha <18810541851@163.com> (cherry picked from commit f06e2c8)

…backport #36281) (#36902) Signed-off-by: Yixin Luo <18810541851@163.com> Co-authored-by: Yixin Luo <18810541851@163.com>

…backport StarRocks#36281) (StarRocks#36902) Signed-off-by: Yixin Luo <18810541851@163.com> Co-authored-by: Yixin Luo <18810541851@163.com>

github-actions bot added 3.2 3.1 labels Dec 1, 2023

mergify bot assigned luohaha Dec 1, 2023

imay reviewed Dec 1, 2023

View reviewed changes

be/src/common/constexpr.h Outdated Show resolved Hide resolved

be/src/storage/rowset/column_writer.cpp Outdated Show resolved Hide resolved

luohaha force-pushed the fix-bc-overflow branch 2 times, most recently from 9ead095 to 8758ff6 Compare December 1, 2023 22:24

imay reviewed Dec 2, 2023

View reviewed changes

be/src/common/constexpr.h Outdated Show resolved Hide resolved

luohaha force-pushed the fix-bc-overflow branch from 8758ff6 to 90a4fa5 Compare December 2, 2023 05:35

[BugFix] fix large binary column overflow when column writer append

1a0594d

Signed-off-by: luohaha <18810541851@163.com>

luohaha force-pushed the fix-bc-overflow branch from 90a4fa5 to 1a0594d Compare December 2, 2023 06:09

imay approved these changes Dec 2, 2023

View reviewed changes

imay merged commit f06e2c8 into StarRocks:main Dec 2, 2023
38 checks passed

github-actions bot removed the 3.2 label Dec 2, 2023

github-actions bot removed the 3.1 label Dec 2, 2023

mergify bot mentioned this pull request Dec 2, 2023

[BugFix] fix large binary column overflow when column writer append (backport #36281) #36299

Merged

mergify bot mentioned this pull request Dec 2, 2023

[BugFix] fix large binary column overflow when column writer append (backport #36281) #36300

Merged

imay pushed a commit that referenced this pull request Dec 2, 2023

[BugFix] fix large binary column overflow when column writer append (…

2af81c3

…backport #36281) (#36299) Co-authored-by: Yixin Luo <18810541851@163.com>

imay pushed a commit that referenced this pull request Dec 2, 2023

[BugFix] fix large binary column overflow when column writer append (…

5387efd

…backport #36281) (#36300) Co-authored-by: Yixin Luo <18810541851@163.com>

wanpengfei-git added the backport_ok label Dec 4, 2023

mergify bot added a commit that referenced this pull request Dec 12, 2023

[BugFix] fix large binary column overflow when column writer append (…

de29297

…backport #36281) (#36299) Co-authored-by: Yixin Luo <18810541851@163.com> (cherry picked from commit 2af81c3)

mergify bot mentioned this pull request Dec 12, 2023

[BugFix] fix large binary column overflow when column writer append (backport #36281) #36901

Merged

mergify bot mentioned this pull request Dec 12, 2023

[BugFix] fix large binary column overflow when column writer append (backport #36281) #36902

Merged

11 tasks

wanpengfei-git pushed a commit that referenced this pull request Dec 12, 2023

[BugFix] fix large binary column overflow when column writer append (…

3847833

…backport #36281) (#36901) Co-authored-by: Yixin Luo <18810541851@163.com>

luohaha pushed a commit that referenced this pull request Dec 12, 2023

[BugFix] fix large binary column overflow when column writer append (…

970ff44

…backport #36281) (backport #36299) (#36900) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

luohaha mentioned this pull request Jan 5, 2024

[Feature] Support dict encoding for non-string column #36211

Merged

22 tasks

github-actions bot added 3.2 3.1 3.0 2.5 labels Mar 22, 2024

wanpengfei-git pushed a commit that referenced this pull request Mar 22, 2024

[BugFix] fix large binary column overflow when column writer append (…

cde5108

…backport #36281) (#36902) Signed-off-by: Yixin Luo <18810541851@163.com> Co-authored-by: Yixin Luo <18810541851@163.com>

github-actions bot added the 2.5-merged label Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] fix large binary column overflow when column writer append #36281

[BugFix] fix large binary column overflow when column writer append #36281

luohaha commented Dec 1, 2023 •

edited by trueeyu

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 2, 2023

mergify bot commented Dec 2, 2023 •

edited

mergify bot commented Dec 2, 2023 •

edited

luohaha commented Dec 12, 2023

luohaha commented Dec 12, 2023

mergify bot commented Dec 12, 2023 •

edited

mergify bot commented Dec 12, 2023 •

edited

[BugFix] fix large binary column overflow when column writer append #36281

[BugFix] fix large binary column overflow when column writer append #36281

Conversation

luohaha commented Dec 1, 2023 • edited by trueeyu

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

github-actions bot commented Dec 2, 2023

[FE Incremental Coverage Report]

github-actions bot commented Dec 2, 2023

[BE Incremental Coverage Report]

file detail

github-actions bot commented Dec 2, 2023

github-actions bot commented Dec 2, 2023

mergify bot commented Dec 2, 2023 • edited

✅ Backports have been created

mergify bot commented Dec 2, 2023 • edited

✅ Backports have been created

luohaha commented Dec 12, 2023

luohaha commented Dec 12, 2023

mergify bot commented Dec 12, 2023 • edited

✅ Backports have been created

mergify bot commented Dec 12, 2023 • edited

✅ Backports have been created

luohaha commented Dec 1, 2023 •

edited by trueeyu

mergify bot commented Dec 2, 2023 •

edited

mergify bot commented Dec 2, 2023 •

edited

mergify bot commented Dec 12, 2023 •

edited

mergify bot commented Dec 12, 2023 •

edited