Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tidb-server refuses to start if no pump is registered with pd #10240

Open
kolbe opened this issue Apr 23, 2019 · 4 comments

Comments

Projects
None yet
5 participants
@kolbe
Copy link
Contributor

commented Apr 23, 2019

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
    When trying to restart all TiDB cluster components simultaneously, tidb-server will exit if binlog is enabled but no pump instances are yet registered with pd.

  2. What did you expect to see?
    tidb-server should continue looking for a pump instance either indefinitely or until some timeout, depending on prevailing wisdom and consensus about startup behavior of other components.

  3. What did you see instead?

2019/04/23 13:10:36.520 terror.go:329: [fatal] no pump found in pd
github.com/pingcap/tidb-tools/tidb-binlog/pump_client.NewPumpsClient
        /home/jenkins/workspace/build_tidb_master/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190321065848-1e8b48f5c168+incompatible/tidb-binlog/pump_client/client.go:168
main.setupBinlogClient
        /home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/tidb-server/main.go:227
main.main
        /home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/tidb-server/main.go:166
runtime.main
        /usr/local/go/src/runtime/proc.go:200
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
[4]+  Exit 1                  ./bin/tidb-server --config=tidb.toml &>tidb.out
  1. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
Release Version: v3.0.0-beta.1-150-g9c8396e78
Git Commit Hash: 9c8396e789dfa8681ae290a3f1f96cdedffc320e
Git Branch: master
UTC Build Time: 2019-04-23 03:11:16
GoVersion: go version go1.12 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false
@zz-jason

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

@amyangfei PTAL

@lonng

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

Hi, @kolbe This error occurred because of your deployment and configuration inconsistent. Pump instances must be available if enable-binlog is true.

Maybe we should refine the error message and make it more clear.

@amyangfei

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

yes, if we configure TiDB with enable-binlog = true, the recommended startup sequence is: PD -> TiKV -> Pump -> TiDB -> Drainer

@morgo

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

yes, if we configure TiDB with enable-binlog = true, the recommended startup sequence is: PD -> TiKV -> Pump -> TiDB -> Drainer

TiKV and TiDB have been improved so that they will retry connecting to pd if they are first to start. This helps simplify deployment since guaranteeing order adds complexity to systems. For example see: tikv/tikv#4500

The behavior change that @kolbe is suggesting makes sense to me. The TiDB server should block in a retry loop waiting for a pump server to be available instead of hard failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.