[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size by airborne12 · Pull Request #43480 · apache/doris

airborne12 · 2024-11-08T02:25:00Z

What problem does this PR solve?

Problem Summary:
Fix ngram bloomfilter index coredump as below

*** SIGFPE integer divide by zero (@0x56294f026472) received by PID 4016941 (TID 4019213 OR 0x7f294eb4d640) from PID 1325556850; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at be/src/common/signal_handler.h:421
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in bin/jdk-17.0.2/lib/server/libjvm.so
 2# JVM_handle_linux_signal in bin/jdk-17.0.2/lib/server/libjvm.so
 3# 0x00007F3071042520 in /lib/x86_64-linux-gnu/libc.so.6
 4# doris::segment_v2::NGramBloomFilter::add_bytes(char const*, unsigned int) at be/src/olap/rowset/segment_v2/ngram_bloom_filter.cpp:61
 5# doris::ITokenExtractorHelper<doris::NgramTokenExtractor>::string_to_bloom_filter(char const*, unsigned long, doris::segment_v2::BloomFilter&) const at be/src/olap/itoken_extractor.h:61
 6# doris::segment_v2::NGramBloomFilterIndexWriterImpl::add_values(void const*, unsigned long) at be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:250

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-08T02:25:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2024-11-08T02:25:08Z

run buildall

github-actions · 2024-11-08T02:29:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-08T03:16:59Z

TeamCity be ut coverage result:
Function Coverage: 37.90% (9864/26026)
Line Coverage: 29.07% (82064/282344)
Region Coverage: 28.28% (42260/149434)
Branch Coverage: 24.86% (21439/86250)
Coverage Report: http://coverage.selectdb-in.cc/coverage/58d386df8a7122e86d2f8f74205e088a19096de4_58d386df8a7122e86d2f8f74205e088a19096de4/report/index.html

fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java

airborne12 · 2024-11-11T09:01:58Z

run buildall

github-actions · 2024-11-11T09:05:59Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-11T09:08:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-11T09:08:23Z

PR approved by anyone and no changes requested.

be/src/olap/rowset/segment_v2/segment_writer.cpp

airborne12 · 2024-11-11T09:25:41Z

run buildall

github-actions · 2024-11-11T09:31:19Z

clang-tidy review says "All clean, LGTM! 👍"

qidaye

LGTM

github-actions · 2024-11-11T09:37:41Z

PR approved by at least one committer and no changes requested.

csun5285

LGTM

doris-robot · 2024-11-11T09:51:41Z

TeamCity be ut coverage result:
Function Coverage: 37.79% (9846/26054)
Line Coverage: 29.08% (82367/283227)
Region Coverage: 28.19% (42325/150148)
Branch Coverage: 24.77% (21455/86620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9497e00f2c31ee9220b730f3c762b925a9567f52_9497e00f2c31ee9220b730f3c762b925a9567f52/report/index.html

eldenmoon

LGTM

) ### What problem does this PR solve? Problem Summary: Fix ngram bloomfilter index coredump as below ``` *** SIGFPE integer divide by zero (@0x56294f026472) received by PID 4016941 (TID 4019213 OR 0x7f294eb4d640) from PID 1325556850; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in bin/jdk-17.0.2/lib/server/libjvm.so 2# JVM_handle_linux_signal in bin/jdk-17.0.2/lib/server/libjvm.so 3# 0x00007F3071042520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::segment_v2::NGramBloomFilter::add_bytes(char const*, unsigned int) at be/src/olap/rowset/segment_v2/ngram_bloom_filter.cpp:61 5# doris::ITokenExtractorHelper<doris::NgramTokenExtractor>::string_to_bloom_filter(char const*, unsigned long, doris::segment_v2::BloomFilter&) const at be/src/olap/itoken_extractor.h:61 6# doris::segment_v2::NGramBloomFilterIndexWriterImpl::add_values(void const*, unsigned long) at be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:250 ``` Co-authored-by: airborne12 <jiangkai@selectdb.com>

…che#43480) Problem Summary: Fix ngram bloomfilter index coredump as below ``` *** SIGFPE integer divide by zero (@0x56294f026472) received by PID 4016941 (TID 4019213 OR 0x7f294eb4d640) from PID 1325556850; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in bin/jdk-17.0.2/lib/server/libjvm.so 2# JVM_handle_linux_signal in bin/jdk-17.0.2/lib/server/libjvm.so 3# 0x00007F3071042520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::segment_v2::NGramBloomFilter::add_bytes(char const*, unsigned int) at be/src/olap/rowset/segment_v2/ngram_bloom_filter.cpp:61 5# doris::ITokenExtractorHelper<doris::NgramTokenExtractor>::string_to_bloom_filter(char const*, unsigned long, doris::segment_v2::BloomFilter&) const at be/src/olap/itoken_extractor.h:61 6# doris::segment_v2::NGramBloomFilterIndexWriterImpl::add_values(void const*, unsigned long) at be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:250 ``` Co-authored-by: airborne12 <jiangkai@selectdb.com>

…bf_size (#43645) Cherry-picked from #43480 Co-authored-by: airborne12 <jiangkai@selectdb.com>

… (#43655) cherry pick from #43480

…che#43480) ### What problem does this PR solve? Problem Summary: Fix ngram bloomfilter index coredump as below ``` *** SIGFPE integer divide by zero (@0x56294f026472) received by PID 4016941 (TID 4019213 OR 0x7f294eb4d640) from PID 1325556850; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in bin/jdk-17.0.2/lib/server/libjvm.so 2# JVM_handle_linux_signal in bin/jdk-17.0.2/lib/server/libjvm.so 3# 0x00007F3071042520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::segment_v2::NGramBloomFilter::add_bytes(char const*, unsigned int) at be/src/olap/rowset/segment_v2/ngram_bloom_filter.cpp:61 5# doris::ITokenExtractorHelper<doris::NgramTokenExtractor>::string_to_bloom_filter(char const*, unsigned long, doris::segment_v2::BloomFilter&) const at be/src/olap/itoken_extractor.h:61 6# doris::segment_v2::NGramBloomFilterIndexWriterImpl::add_values(void const*, unsigned long) at be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:250 ``` Co-authored-by: airborne12 <jiangkai@selectdb.com>

… (#43654) cherry pick from #43480

…che#43480) ### What problem does this PR solve? Problem Summary: Fix ngram bloomfilter index coredump as below ``` *** SIGFPE integer divide by zero (@0x56294f026472) received by PID 4016941 (TID 4019213 OR 0x7f294eb4d640) from PID 1325556850; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in bin/jdk-17.0.2/lib/server/libjvm.so 2# JVM_handle_linux_signal in bin/jdk-17.0.2/lib/server/libjvm.so 3# 0x00007F3071042520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::segment_v2::NGramBloomFilter::add_bytes(char const*, unsigned int) at be/src/olap/rowset/segment_v2/ngram_bloom_filter.cpp:61 5# doris::ITokenExtractorHelper<doris::NgramTokenExtractor>::string_to_bloom_filter(char const*, unsigned long, doris::segment_v2::BloomFilter&) const at be/src/olap/itoken_extractor.h:61 6# doris::segment_v2::NGramBloomFilterIndexWriterImpl::add_values(void const*, unsigned long) at be/src/olap/rowset/segment_v2/bloom_filter_index_writer.cpp:250 ``` Co-authored-by: airborne12 <jiangkai@selectdb.com>

[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size

58d386d

eldenmoon reviewed Nov 8, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java Outdated Show resolved Hide resolved

eldenmoon reviewed Nov 8, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java Show resolved Hide resolved

gavinchou added the dev/3.0.x label Nov 8, 2024

add test case

30bdb69

gavinchou previously approved these changes Nov 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 11, 2024

github-actions bot added the reviewed label Nov 11, 2024

csun5285 reviewed Nov 11, 2024

View reviewed changes

be/src/olap/rowset/segment_v2/segment_writer.cpp Show resolved Hide resolved

add vertical segment writer

9497e00

airborne12 dismissed gavinchou’s stale review via 9497e00 November 11, 2024 09:25

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 11, 2024

qidaye approved these changes Nov 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 11, 2024

csun5285 approved these changes Nov 11, 2024

View reviewed changes

eldenmoon approved these changes Nov 11, 2024

View reviewed changes

gavinchou added p0_c cir labels Nov 11, 2024

airborne12 merged commit 2218e09 into apache:master Nov 12, 2024

airborne12 deleted the fix-bloomfilter branch November 12, 2024 01:10

airborne12 added dev/2.1.x dev/2.0.x labels Nov 12, 2024

github-actions bot mentioned this pull request Nov 12, 2024

branch-3.0: [fix](ngram bloomfilter) fix narrow conversion for ngram bf_size #43645

Merged

airborne12 mentioned this pull request Nov 12, 2024

[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size #43480 #43654

Merged

airborne12 mentioned this pull request Nov 12, 2024

[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size #43480 #43655

Merged

airborne12 added a commit that referenced this pull request Nov 12, 2024

branch-3.0: [fix](ngram bloomfilter) fix narrow conversion for ngram …

f5f1f8c

…bf_size (#43645) Cherry-picked from #43480 Co-authored-by: airborne12 <jiangkai@selectdb.com>

airborne12 added dev/3.0.3-merged and removed dev/3.0.x labels Nov 12, 2024

airborne12 added a commit that referenced this pull request Nov 12, 2024

[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size #43480…

514e79c

… (#43655) cherry pick from #43480

airborne12 added dev/2.0.16-merged-deprecated 2.0.16 is NOT and will NOT be released and removed dev/2.0.x labels Nov 12, 2024

airborne12 added a commit that referenced this pull request Nov 12, 2024

[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size #43480…

3e098e1

… (#43654) cherry pick from #43480

airborne12 added dev/2.1.8-merged and removed dev/2.1.x labels Nov 12, 2024

Conversation

airborne12 commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 8, 2024

Uh oh!

airborne12 commented Nov 8, 2024

Uh oh!

github-actions bot commented Nov 8, 2024

Uh oh!

doris-robot commented Nov 8, 2024

Uh oh!

Uh oh!

Uh oh!

airborne12 commented Nov 11, 2024

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

Uh oh!

airborne12 commented Nov 11, 2024

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Nov 11, 2024

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

airborne12 commented Nov 8, 2024 •

edited

Loading