pd server报错

[soniazhai]

2016/12/29 17:04:09 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 99
2016/12/29 17:04:09 conn.go:102: [error] handle request header:<uuid:"\364\276E\355\034j@\320\225\307\221g\007"\350u" cluster_id:6369440677371556373 > cmd_type:Tso tso:<count:1 > err /home/jenkins/workspace/BUILD_PD_WORKFLOW/go/src/github.com/pingcap/pd/server/tso.go:169: can not get timestamp
/home/jenkins/workspace/BUILD_PD_WORKFLOW/go/src/github.com/pingcap/pd/server/command.go:33:
2016/12/29 17:04:09 conn.go:75: [error] read request message err EOF

该服务器时钟服务是正常的，与集群里其他机器也是同步的，这个错误是怎么导致的？该如何解决？

[shenli]

@soniazhai 请问你启动了几个 pd-server?

[soniazhai]

三个

[siddontang]

please give us the whole pd log.

[soniazhai]

邮件发还是都贴出来？

[huachaohuang]

可以 attach 到 github 上

[soniazhai]

# 我又重启了所有的机器，重新试了一次，就是在运行tidb时，pd server leader 开始报错
2016/12/29 18:15:42 conn.go:75: [error] read request message err EOF
2016/12/29 18:15:42 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 0
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 1
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 2
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 3
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 4
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 5
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 6
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 7
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 8
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 9
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 10
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 11
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 12
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 13
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 14
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 15
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 16
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 17
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 18
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 19
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 20
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 21
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 22
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 23
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 24
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 25
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 26
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 27
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 28
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 29
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 30
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 31
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 32
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 33
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 34
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 35
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 36
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 37
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 38
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 39
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 40
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 41
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 42
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 43
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 44
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 45
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 46
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 47
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 48
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 49
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 50
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 51
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 52
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 53
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 54
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 55
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 56
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 57
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 58
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 59
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 60
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 61
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 62
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 63
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 64
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 65
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 66
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 67
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 68
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 69
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 70
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 71
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 72
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 73
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 74
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 75
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 76
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 77
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 78
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 79
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 80
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 81
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 82
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 83
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 84
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 85
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 86
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 87
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 88
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 89
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 90
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 91
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 92
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 93
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 94
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 95
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 96
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 97
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 98
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 99
2016/12/29 18:16:03 conn.go:102: [error] handle request header:<uuid:"\227\231\222\034R\214B\370\220\223\220\256\274L\221\330" cluster_id:6369440677371556373 > cmd_type:Tso tso:<count:1 > err /home/jenkins/workspace/BUILD_PD_WORKFLOW/go/src/github.com/pingcap/pd/server/tso.go:169: can not get timestamp
/home/jenkins/workspace/BUILD_PD_WORKFLOW/go/src/github.com/pingcap/pd/server/command.go:33:
2016/12/29 18:16:03 conn.go:75: [error] read request message err EOF

同时，tidb-server也没运行成功
2016/12/29 18:15:43 printer.go:31: [info] Welcome to TiDB.
2016/12/29 18:15:43 printer.go:32: [info] Version:
2016/12/29 18:15:43 printer.go:33: [info] Git Commit Hash: f1aaeafade7c78f1ca6ee7edec395bc3306282df
2016/12/29 18:15:43 printer.go:34: [info] UTC Build Time: 2016-12-13 06:50:46
2016/12/29 18:15:43 client.go:52: [info] [pd] create pd client with endpoints [10.5.77.11:22379 10.5.77.12:22379 10.5.77.13:22379]
2016/12/29 18:15:43 rpc_worker.go:82: [info] [pd] init cluster id 6369440677371556373
2016/12/29 18:15:43 rpc_worker.go:113: [info] [pd] connect to pd server [http://10.5.77.11:22379 http://10.5.77.12:22379 http://10.5.77.13:22379]
2016/12/29 18:15:43 rpc_worker.go:118: [info] [pd] connected to 10.5.77.12:22379
2016/12/29 18:15:44 rpc_worker.go:196: [error] [pd] rpc failed: read tcp 10.5.77.11:37702->10.5.77.12:22379: i/o timeout
2016/12/29 18:15:44 rpc_worker.go:113: [info] [pd] connect to pd server [http://10.5.77.11:22379 http://10.5.77.12:22379 http://10.5.77.13:22379]
2016/12/29 18:15:44 main.go:156: [fatal] github.com/pingcap/pd/pd-client/rpc_worker.go:371: [pd] rpc failed: read tcp 10.5.77.11:37702->10.5.77.12:22379: i/o timeout
github.com/pingcap/pd/pd-client/rpc_worker.go:279:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/oracle/oracles/pd.go:86:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/oracle/oracles/pd.go:75:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/oracle/oracles/pd.go:57:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/kv.go:95:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/kv.go:73:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/tidb.go:248:

[huachaohuang]

请把三个 PD 的完整日志发到 huanghuachao@pingcap.com

[shenli]

@soniazhai 从 log 来看，我们猜测是两种可能，1. PD 向磁盘写东西写不进去，比如磁盘慢了， 2. PD 之间的通讯有问题，比如某个端口被防火墙block了。请帮忙把三个pd完整的log贴在这里，我们确认一下问题。
多谢！

[soniazhai]

不好意思，放假几天，没及时跟进。问题已经解决了，还是时钟同步有问题。重新调用了ntp服务，已经正常了。

[huachaohuang]

好的，欢迎继续试用，有问题随时反馈，多谢！