Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Doris Be node crash once in a while ,version:doris-1.2.7.1-rc01 #26600

Open
2 of 3 tasks
mnloveyx opened this issue Nov 8, 2023 · 1 comment
Open
2 of 3 tasks

Comments

@mnloveyx
Copy link

mnloveyx commented Nov 8, 2023

Search before asking

  • I had searched in the issues and found no similar issues.

Version

doris-1.2.7.1-rc01

What's Wrong?

*** Query id: 9e0aa771264c415e-bc67205f9cdd857e ***
*** Aborted at 1699411494 (unix time) try "date -d @1699411494" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGSEGV address not mapped to object (@0x7f2aa2000000) received by PID 28566 (TID 0x7f3c6596a700) from PID 18446744072132493312; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/release/doris-branch-1.2-lts/be/src/common/signal_handler.h:420
1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/jdk/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /opt/jdk/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /opt/jdk/jre/lib/amd64/server/libjvm.so
4# 0x00007F4D7BC33400 in /lib64/libc.so.6
5# memcpy at /root/doris/release/doris-branch-1.2-lts/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# doris::vectorized::DataTypeString::deserialize(char const*, doris::vectorized::IColumn*, int) const at /root/doris/release/doris-branch-1.2-lts/be/src/vec/data_types/data_type_string.cpp:184
7# doris::vectorized::Block::Block(doris::PBlock const&) at /root/doris/release/doris-branch-1.2-lts/be/src/vec/core/block.cpp:106
8# doris::stream_load::VNodeChannel::try_send_block(doris::RuntimeState*) at /root/doris/release/doris-branch-1.2-lts/be/src/vec/sink/vtablet_sink.cpp:305
9# doris::ThreadPool::dispatch_thread() at /root/doris/release/doris-branch-1.2-lts/be/src/util/threadpool.cpp:543
10# doris::Thread::supervise_thread(void*) at /root/doris/release/doris-branch-1.2-lts/be/src/util/thread.cpp:455
11# start_thread in /lib64/libpthread.so.0
12# clone in /lib64/libc.so.6

Cpu Info:
Model: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
Cores: 32
Max Possible Cores: 32
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 1.00 MB (Line: 64.00 B)
L3 Cache: 35.75 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
avx
avx2
Numa Nodes: 1
Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 | 4->0 | 5->0 | 6->0 | 7->0 | 8->0 | 9->0 | 10->0 | 11->0 | 12->0 | 13->0 | 14->0 | 15->0 | 16->0 | 17->0 | 18->0 | 19->0 | 20->0 | 21->0 | 22->0 | 23->0 | 24->0 | 25->0 | 26->0 | 27->0 | 28->0 | 29->0 | 30->0 | 31->0 |
I1108 10:46:00.625185 13181 daemon.cpp:466] Disk Info:
Num disks 7: vda, vdb, vdc, vdd, vde, vdf, vdg
I1108 10:46:00.625190 13181 daemon.cpp:467] Physical Memory: 247.79 GB
Memory Limt: 198.23 GB
CGroup Info: Process CGroup Info: memory.limit_in_bytes=9223372036854771712, cpu cfs limits: unlimited

SYSTME INFO:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

What You Expected?

sql 是定时调度的,问题偶尔出现。

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@mnloveyx mnloveyx changed the title [Bug] Doris Be node crash once in a while ,version:doris-1.2.7.01 [Bug] Doris Be node crash once in a while ,version:doris-1.2.7.1-rc01 Nov 8, 2023
@lide-reed
Copy link
Contributor

lide-reed commented Nov 26, 2023

Core was generated by `/usr/local/service/doris/lib/be/doris_be'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055de8d7a4199 in _mm_loadu_si128(long long __vector(2) const*) (__P=0x7f294c1ffffa)
at /var/local/ldb-toolchain/lib/gcc/x86_64-linux-gnu/11/include/emmintrin.h:703
703 /var/local/ldb-toolchain/lib/gcc/x86_64-linux-gnu/11/include/emmintrin.h: No such file or directory.
[Current thread is 1 (Thread 0x7f3c504ee700 (LWP 326148))]
(gdb) bt
#0 0x000055de8d7a4199 in _mm_loadu_si128(long long __vector(2) const*) (P=0x7f294c1ffffa)
at /var/local/ldb-toolchain/lib/gcc/x86_64-linux-gnu/11/include/emmintrin.h:703
#1 inline_memcpy (size=17194771064, src
=0x7f285c6a12da, dst
=0x7ebc391c7010)
at /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:189
#2 memcpy (dst=0x7ebc391c7010, src=src@entry=0x7f285c6a12da, size=size@entry=21216448376)
at /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
#3 0x000055de8f2d2fe3 in memcpy (__len=, __src=0x7f285c6a12da, __dest=)
at /var/local/ldb-toolchain/usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#4 doris::vectorized::DataTypeNumberBase::deserialize (this=,
buf=0x7f285c6a12da "\311\371\240\326u\223P(\232Y\330\t\r\212Kj\354\354\017mp=\257$"\024Hɔ\246\265<?\261\361\322\016t>5\252\237\213p\005\222&\177\246\003¤a\320ŕN\232w", column=0x7f36217b6150, be_exec_version=)
at /data/doris-1.x/be/src/vec/data_types/data_type_number_base.cpp:149
#5 0x000055de8f29bfe3 in doris::vectorized::Block::Block (this=this@entry=0x7f3c504e6400, pblock=...)
at /data/doris-1.x/be/src/vec/common/cow.h:190
#6 0x000055de92ef39f4 in doris::stream_load::VNodeChannel::try_send_block (this=0x7f39144a4010, state=0x7f390813a600)
at /data/doris-1.x/gensrc/build/gen_cpp/internal_service.pb.h:16387
#7 0x000055de8e399b15 in std::function<void ()>::operator()() const (this=)
at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:556
#8 doris::FunctionRunnable::run (this=) at /data/doris-1.x/be/src/util/threadpool.cpp:46
#9 doris::ThreadPool::dispatch_thread (this=0x7f3c990871c0) at /data/doris-1.x/be/src/util/threadpool.cpp:535
#10 0x000055de8e38ef6f in std::function<void ()>::operator()() const (this=0x7f3c9906c458)
at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:556
#11 doris::Thread::supervise_thread (arg=0x7f3c9906c440) at /data/doris-1.x/be/src/util/thread.cpp:454
#12 0x00007f3ccb04bea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3ccb35e9fd in clone () from /lib64/libc.so.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants