Add timeout when updatetopology

[Win-Man]

tiup cluster start will statll at updatetopology stage and don't exit when server can't connect to pd server.
Suggest to add timeout at updatetopology.

[lucklove]

Can you provide the version of tiup?

tiup --version

[Win-Man]

The version of tiup cluster is:

[tidb@sj-db-mysql-dev002 package]$ tiup --binary cluster
/home/tidb/.tiup/components/cluster/v0.6.2/cluster

But the version of tiup I've forgot.

[lucklove]

We did have --ssh-timeout and --wait-timeout now (v1.x). Is this what you want?

[Win-Man]

It seems --ssh-timeout and --wait-timeout didn't work in this case. The user waited for a long time, that has already exceed the default value of two options, but the updatetopology didn't exit.
The reason is tiup server couldn't connection to the pd server, so updatetopology can't success.

[lucklove]

Do you know how to reproduce this?

I have checked the code and found that it's impossible to hang on the UpdateTopology stage. Because in UpdateTopology, tiup doesn't access network.

[lucklove]

Reproduce action I have tried:

• Deploy a normal cluster
• Modify meta.yaml, change pd's IP to 1.1.1.1, which can't be connect
• Start the cluster

The result is that it will fail on start operation, the update topology action is after start operation.

[AstroProfundis]

Could you confirm if this issue still exist on v1.0.8 or later?

[mengling2012]

It's the same phenomenon

[tidb@ip-172-16-4-31 ~]$ tiup cluster start tidb-vivo -R grafana,prometheus Starting component cluster`: /home/tidb/.tiup/components/cluster/v1.3.2/tiup-cluster start tidb-vivo -R grafana,prometheus
Starting cluster tidb-vivo...

• [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-vivo/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-vivo/ssh/id_rsa.pub
• [Parallel] - UserSSH: user=tidb, host=172.16.4.31
• [Parallel] - UserSSH: user=tidb, host=172.16.4.31
• [Parallel] - UserSSH: user=tidb, host=172.16.4.32
• [Parallel] - UserSSH: user=tidb, host=172.16.4.35
• [Parallel] - UserSSH: user=tidb, host=172.16.4.33
• [Parallel] - UserSSH: user=tidb, host=172.16.4.33
• [Parallel] - UserSSH: user=tidb, host=172.16.4.31
• [Parallel] - UserSSH: user=tidb, host=172.16.4.34
• [Parallel] - UserSSH: user=tidb, host=172.16.4.33
• [Parallel] - UserSSH: user=tidb, host=172.16.4.31
• [Parallel] - UserSSH: user=tidb, host=172.16.4.31
• [Parallel] - UserSSH: user=tidb, host=172.16.4.32
• [Parallel] - UserSSH: user=tidb, host=172.16.4.34
• [Parallel] - UserSSH: user=tidb, host=172.16.4.35
• [ Serial ] - StartCluster
  Starting component prometheus
  Starting instance prometheus 172.16.4.31:29090
  Start prometheus 172.16.4.31:29090 success
  Starting component node_exporter
  Starting instance 172.16.4.31
  Start 172.16.4.31 success
  Starting component blackbox_exporter
  Starting instance 172.16.4.31
  Start 172.16.4.31 success
  Starting component grafana
  Starting instance grafana 172.16.4.31:23000
  Start grafana 172.16.4.31:23000 success
• [ Serial ] - UpdateTopology: cluster=tidb-vivo`

Pause here,block shell.

thanks.

[mengling2012]

open new issue
#1141

thanks.

[AstroProfundis]

Hi @mengling2012, could you share with us how to reproduce it? The exact command you executed in each step, the system environment and network status, etc.

[mengling2012]

any cluster tidb.
1.tiup cluster stop tidb-vivo
2.tiup cluster start tidb-vivo -R grafana,prometheus

then,you can reproduce it.

thanks.

[mengling2012]

Only start grafana and prometheus,others roles are down or inacitve.

thanks.

[9547]

Only start grafana and prometheus,others roles are down or inacitve.

thanks.

Thanks for your reproduce instruction, I might have found something wrong, If all PD servers were down, the UpdateTopology will hang forever, the stack looks below:

#	0xb8a94f	google.golang.org/grpc.(*pickerWrapper).pick+0x78f				/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/picker_wrapper.go:152
#	0xb854e4	google.golang.org/grpc.(*ClientConn).getTransport+0x84				/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/clientconn.go:882
#	0xb9d827	google.golang.org/grpc.(*clientStream).newAttemptLocked+0x187			/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/stream.go:351
#	0xb9c699	google.golang.org/grpc.newClientStream+0x999					/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/stream.go:284
#	0xb7fe58	google.golang.org/grpc.invoke+0x98						/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/call.go:66
#	0xc7114d	go.etcd.io/etcd/clientv3.(*Client).unaryClientInterceptor.func1+0x48d		/root/.go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/clientv3/retry_interceptor.go:58
#	0xb7fbe8	google.golang.org/grpc.(*ClientConn).Invoke+0x108				/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/call.go:35
#	0xbccf6e	google.golang.org/grpc.Invoke+0xce						/root/.go/pkg/mod/google.golang.org/grpc@v1.26.0/call.go:60
#	0xbccedb	go.etcd.io/etcd/etcdserver/etcdserverpb.(*kVClient).Txn+0x3b			/root/.go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/etcdserver/etcdserverpb/rpc.pb.go:3497
#	0xc6395b	go.etcd.io/etcd/clientv3.(*retryKVClient).Txn+0x7b				/root/.go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/clientv3/retry.go:117
#	0xc68dd7	go.etcd.io/etcd/clientv3.(*txn).Commit+0x157					/root/.go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/clientv3/txn.go:146
#	0x1271206	github.com/pingcap/tiup/pkg/cluster/task.(*UpdateTopology).Execute+0x786	/root/.go/src/github.com/pingcap/tiup/pkg/cluster/task/update_topology.go:76
#	0x126c354	github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute+0x154		/root/.go/src/github.com/pingcap/tiup/pkg/cluster/task/task.go:85
#	0x127533a	github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).StartCluster+0x5da	/root/.go/src/github.com/pingcap/tiup/pkg/cluster/manager/basic.go:105
#	0x131d5a4	github.com/pingcap/tiup/components/cluster/command.newStartCmd.func1+0x204	/root/.go/src/github.com/pingcap/tiup/components/cluster/command/start.go:38
#	0x82c27b	github.com/spf13/cobra.(*Command).execute+0x47b					/root/.go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852
#	0x82cdf4	github.com/spf13/cobra.(*Command).ExecuteC+0x374				/root/.go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960
#	0x1313c16	github.com/spf13/cobra.(*Command).Execute+0x476					/root/.go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
#	0x1313c07	github.com/pingcap/tiup/components/cluster/command.Execute+0x467		/root/.go/src/github.com/pingcap/tiup/components/cluster/command/root.go:260
#	0x131f47c	main.main+0x5c									/root/.go/src/github.com/pingcap/tiup/components/cluster/main.go:30
#	0x43a2a8	runtime.main+0x208								/usr/local/go/src/runtime/proc.go:204

We need to check the PD status before UpdateTopology is invoked, or add timeout as @Win-Man suggested.