Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tidb panic with error “[ERROR] [client_batch.go:337] [batchSendLoop] [r="runtime error: slice bounds out of range [-1:]"]” #51921

Closed
Lily2025 opened this issue Mar 20, 2024 · 1 comment · Fixed by tikv/client-go#1242 or #52026

Comments

@Lily2025
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run mussel workload
2、inject network partition between pd leader and pd followers

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

tidb panic

logs:
tidb-0.tar.gz

[2024/03/20 03:29:52.792 +08:00] [ERROR] [client_batch.go:337] [batchSendLoop] [r="runtime error: slice bounds out of range [-1:]"] [stack="github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop.func1\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240316105842-98a7df8f413d/internal/client/client_batch.go:339\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.goPanicSliceB\n\t/usr/local/go/src/runtime/panic.go:154\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request).MarshalToSizedBuffer\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20240227073058-929ab83f9754/pkg/tikvpb/tikvpb.pb.go:4478\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).MarshalToSizedBuffer\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20240227073058-929ab83f9754/pkg/tikvpb/tikvpb.pb.go:4436\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).Marshal\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20240227073058-929ab83f9754/pkg/tikvpb/tikvpb.pb.go:4394\ngoogle.golang.org/protobuf/internal/impl.legacyMarshal\n\t/go/pkg/mod/google.golang.org/protobuf@v1.33.0/internal/impl/legacy_message.go:407\ngoogle.golang.org/protobuf/proto.MarshalOptions.marshal\n\t/go/pkg/mod/google.golang.org/protobuf@v1.33.0/proto/encode.go:166\ngoogle.golang.org/protobuf/proto.Marshal\n\t/go/pkg/mod/google.golang.org/protobuf@v1.33.0/proto/encode.go:80\ngoogle.golang.org/grpc/encoding/proto.codec.Marshal\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/encoding/proto/proto.go:47\ngoogle.golang.org/grpc.encode\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/rpc_util.go:647\ngoogle.golang.org/grpc.prepareMsg\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/stream.go:1771\ngoogle.golang.org/grpc.(*clientStream).SendMsg\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/stream.go:893\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*tikvBatchCommandsClient).Send\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20240227073058-929ab83f9754/pkg/tikvpb/tikvpb.pb.go:2293\ngithub.com/tikv/client-go/v2/internal/client.(*batchCommandsClient).send\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240316105842-98a7df8f413d/internal/client/client_batch.go:599\ngithub.com/tikv/client-go/v2/internal/client.(*batchConn).getClientAndSend\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240316105842-98a7df8f413d/internal/client/client_batch.go:435\ngithub.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.8-0.20240316105842-98a7df8f413d/internal/client/client_batch.go:378"]
[2024/03/20 03:29:52.792 +08:00] [INFO] [client_batch.go:341] ["restart batchSendLoop"] [count=2]

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: 02be83a
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-03-17 12:14:34
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-03-20T03:14:17.205+0800

@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Mar 20, 2024
@zyguan
Copy link
Contributor

zyguan commented Mar 20, 2024

It seems to be caused by the data race on XxxRequest.Context

2024-03-20_123605

zyguan added a commit to zyguan/tidb that referenced this issue Mar 22, 2024
Signed-off-by: zyguan <zhongyangguan@gmail.com>
ti-chi-bot bot pushed a commit that referenced this issue Mar 22, 2024
ti-chi-bot bot pushed a commit that referenced this issue Mar 25, 2024
ti-chi-bot bot pushed a commit that referenced this issue May 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants