-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BE occasionally crashes in DataStreamSender #2466
Comments
@gaodayue Hi, have you found where the problem is? |
Not yet. |
@gaodayue Does this problem occured frequently in your cluster? Our cluster may occur this problem once a day. |
Not as high as yours, we run into this problem about once a week |
I double that memory is written dirty, you can can ASAN sanitizer to found the problem. |
The resource of our cluster is very nervous, so I can't use ASAN to analysis this problem. If the memory in your cluster is ample, hope u can help to find the problem. Thanks very much! |
We got brpc cores in DataStreamSender today again. And in our case, the queried tables are all |
Describe the bug
Recently we found BE occasionally crash inside brpc with the following stacktrace.
Content of the problematic
butil::IOBuf
is pasted below, both0x2000000000134
and0x30000000002a6
are invalid memory locations causing the segment fault we saw.The context of
DataStreamSender::Channel::send_batch
is that_packet_seq == 1
which means theDataStreamSender::Channel
is closed right after init.What's also strange is that
_closure->_refs == 1
, does is mean the response of the RPC has been received?To Reproduce
Right row I can't find a stable way to reproduce the bug. The version we use is 0.10 with #1830 patched.
The text was updated successfully, but these errors were encountered: