[Bug] All BE crashed after run 'build index' command on a ngram bloomfilter index

[zhaopinglu]

Search before asking

• I had searched in the issues and found no similar issues.

Version

2.0.2-bin-x64

What's Wrong?

I created a ngram bloomfilter index then ran build command on it. Then all be crashed.

What You Expected?

All BE can startup and work as normal.

How to Reproduce?

step 1: create a table say lineitem as below, then insert some lines
Note: Actually this table was created by tpch-tool. But I don't think its relevant.

CREATE TABLE lineitem (
l_shipdate date NOT NULL,
l_orderkey bigint(20) NOT NULL,
l_linenumber int(11) NOT NULL,
l_partkey int(11) NOT NULL,
l_suppkey int(11) NOT NULL,
l_quantity DECIMAL(15, 2) NOT NULL,
l_extendedprice DECIMAL(15, 2) NOT NULL,
l_discount DECIMAL(15, 2) NOT NULL,
l_tax DECIMAL(15, 2) NOT NULL,
l_returnflag varchar(1) NOT NULL,
l_linestatus varchar(1) NOT NULL,
l_commitdate date NOT NULL,
l_receiptdate date NOT NULL,
l_shipinstruct varchar(25) NOT NULL,
l_shipmode varchar(10) NOT NULL,
l_comment varchar(44) NOT NULL
) ENGINE=OLAP
UNIQUE KEY(l_shipdate, l_orderkey)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(l_orderkey) BUCKETS 96

step 2: create a ngram bloomfilter on lineitem
alter table lineitem add index idx_ngram_(l_comment) using NGRAM_BF PROPERTIES("gram_size"="3", "bf_size"="512");

step 3: build the index which created in previous step.
build index idx_ngram_ on lineitem;

Anything Else?

I know I probably should not run 'build index' on an ngram bloomfilter index.
But Doris didn't reject it when run the command.

Crash stack:

start time: 2023年 12月 07日 星期四 18:24:09 CST
INFO: java_cmd /usr/lib/jvm/openlogic-openjdk-8-hotspot/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
OpenJDK 64-Bit Server VM warning: You have loaded library /data/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
*** Query id: 0-0 ***
*** tablet id: 0 ***
*** Aborted at 1701944660 (unix time) try "date -d @1701944660" if you are using GNU date ***
*** Current BE git commitID: ae923f7 ***
*** SIGSEGV address not mapped to object (@0x30) received by PID 247774 (TID 248522 OR 0x7e67a769f700) from PID 48; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417
1# os::Linux::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/openlogic-openjdk-8-hotspot/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/openlogic-openjdk-8-hotspot/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo_t*, void*) in /usr/lib/jvm/openlogic-openjdk-8-hotspot/jre/lib/amd64/server/libjvm.so
4# 0x00007F091D6ECB80 in /usr/lib64/libc.so.6
5# doris::vectorized::OlapBlockDataConvertor::convert_column_data(unsigned long) at /root/src/doris-2.0/be/src/vec/olap/olap_data_convertor.cpp:222
6# doris::IndexBuilder::_write_inverted_index_data(std::shared_ptrdoris::TabletSchema, int, doris::vectorized::Block*) at /root/src/doris-2.0/be/src/olap/task/index_builder.cpp:267
7# doris::IndexBuilder::handle_single_rowset(std::shared_ptrdoris::RowsetMeta, std::vector<std::shared_ptrdoris::segment_v2::Segment, std::allocator<std::shared_ptrdoris::segment_v2::Segment > >&) at /root/src/doris-2.0/be/src/olap/task/index_builder.cpp:230
8# doris::IndexBuilder::handle_inverted_index_data() at /root/src/doris-2.0/be/src/olap/task/index_builder.cpp:397
9# doris::IndexBuilder::do_build_inverted_index() in /root/doris/be/lib/doris_be
10# doris::StorageEngine::_handle_index_change(std::shared_ptrdoris::IndexBuilder) at /root/src/doris-2.0/be/src/olap/olap_server.cpp:1039
11# doris::StorageEngine::process_index_change_task(doris::TAlterInvertedIndexReq const&) at /root/src/doris-2.0/be/src/olap/olap_server.cpp:1033
12# doris::EngineIndexChangeTask::execute() at /root/src/doris-2.0/be/src/olap/task/engine_index_change_task.cpp:43
13# doris::StorageEngine::execute_task(doris::EngineTask*) at /root/src/doris-2.0/be/src/olap/storage_engine.cpp:1168
14# doris::TaskWorkerPool::_alter_inverted_index_worker_thread_callback() at /root/src/doris-2.0/be/src/agent/task_worker_pool.cpp:354
15# doris::ThreadPool::dispatch_thread() in /root/doris/be/lib/doris_be
16# doris::Thread::supervise_thread(void*) at /root/src/doris-2.0/be/src/util/thread.cpp:470
17# start_thread in /usr/lib64/libpthread.so.0
18# clone in /usr/lib64/libc.so.6

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[niomac]

This is my first time starting BE and reporting this error

start time: Wed Jan 24 16:20:03 CST 2024
INFO: java_cmd /usr/java/jdk1.8.0_192/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLogger
Binder.class]
SLF4J: Found binding in [jar:file:/usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard
. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
F0124 16:20:07.932852 22939 backend_options.cpp:108] wrong cidr format. cidr_str=
*** Check failure stack trace: ***
@ 0x5571ba226e99 google::LogMessageFatal::~LogMessageFatal()
@ 0x5571b2b4ef2a doris::BackendOptions::analyze_priority_cidrs()
@ 0x5571b2b4e9fe doris::BackendOptions::init()
@ 0x5571b20a07a7 main
@ 0x7f3e565d9555 __libc_start_main
@ 0x5571b209f02a _start
@ (nil) (unknown)
*** Query id: 0-0 ***
*** tablet id: 0 ***
*** Aborted at 1706084408 (unix time) try "date -d @1706084408" if you are using GNU date ***
*** Current BE git commitID: 37d31a5 ***
*** SIGABRT unknown detail explain (@0x599b) received by PID 22939 (TID 22939 OR 0x7f3e56fb9e00) from PID 22939; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/src/doris-2.0/be/src/common/signal_handler.h:417
1# 0x00007F3E565ED400 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# abort in /lib64/libc.so.6
4# 0x00005571BA22CE79 in /usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/doris_be
5# google::LogMessage::SendToLog() in /usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/doris_be
6# google::LogMessage::Flush() in /usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/doris_be
7# google::LogMessageFatal::~LogMessageFatal() in /usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/doris_be
8# doris::BackendOptions::analyze_priority_cidrs(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<doris::CIDR, std::allocatordoris::CIDR >*)
at /root/src/doris-2.0/be/src/service/backend_options.cpp:111
9# doris::BackendOptions::init() at /root/src/doris-2.0/be/src/service/backend_options.cpp:44
10# main at /root/src/doris-2.0/be/src/service/doris_main.cpp:458
11# __libc_start_main in /lib64/libc.so.6
12# _start in /usr/local/apache-doris-2.0.3-bin-x64-noavx2/be/lib/doris_be

[xiaokang]

@zhaopinglu It was fixed in 2.0.4. You can upgrade to 2.0.10 or higher version.

[github-actions]

We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!