Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TiKV crash on startup #10495

Closed
aphyr opened this issue May 15, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@aphyr
Copy link

commented May 15, 2019

Yes, it's another crash-on-startup bug! This one looks like it might occur because PD hasn't elected a leader yet?

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?

With Jepsen fecb635cdb7a164dcad523a42a79fadedf5316d5, some attempts to start up the cluster fail because TiKV can crash on startup. See 20190515T121527.000Z.zip for full logs.

lein run test-all --only-workloads-expected-to-pass --time-limit 30 --username admin --nodes-file ~/nodes --tarball-url https://download.pingcap.org/tidb-cluster-for-jepsen.tar.gz
  1. What did you expect to see?

I expected TiKV to start up.

  1. What did you see instead?

TiKV exits after what look like some RPC requests to a non-leader node:

[2019/05/15 12:15:37.672 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.673 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.673 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.673 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.673 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.674 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.674 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.674 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.674 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.675 +00:00] [ERROR] [util.rs:338] ["request failed"] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(\"rpc error: code = Unavailable desc = not leader\") }))"]
[2019/05/15 12:15:37.675 +00:00] [ERROR] [tikv-server.rs:276] ["failed to start node: Other(StringError(\"[src/pd/util.rs:346]: fail to request\"))"]
  1. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

Release Version: v3.0.0-beta.1-203-g412869121a
Git Commit Hash: 4128691
Git Branch: HEAD
UTC Build Time: 2019-05-05 11:26:47
GoVersion: go version go1.12 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false

@overvenus

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

Hi @aphyr, TiKV does not exit anymore in 3.0rc2, tikv/tikv#4502 has resolved this issue.

@aphyr

This comment has been minimized.

Copy link
Author

commented May 31, 2019

Excellent, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.