flushWatcher loop may call refreshPluginState frequently when watch chan closed

[lcwangchao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb/pkg/plugin/plugin.go

Lines 279 to 289 in 90e272a

	func (w *flushWatcher) watchLoop() {
	watchChan := w.etcd.Watch(w.ctx, w.path)
	for {
	select {
	case <-w.ctx.Done():
	return
	case <-watchChan:
	_ = w.refreshPluginState()
	}
	}
	}

This above code does not handle case watchChan closed, so refreshPluginState will be called frequently when this happens.

2. What did you expect to see? (Required)

When watchChan closed , we should close watch chan

3. What did you see instead (Required)

closed chan is not handled correctly

4. What is your TiDB version? (Required)

[siddontang]

@kevin-xianliu, please sort out the code that calls etcd directly, change them one by one, this is prohibited from now on.

[siddontang]

@kevin-xianliu
another thing you need to know is that one of the best way to avoid misconnecting etcd is to open Authentication in etcd and only pd-client can be allowed to connect it. please add an issue to track this.

[niubell]

@kevin-xianliu another thing you need to know is that one of the best way to avoid misconnecting etcd is to open Authentication in etcd and only pd-client can be allowed to connect it. please add an issue to track this.

we don't provide uniform pd-client for Etcd now, but I think introducing the authentication mechanism for embedded Etcd is a good idea, will open an issue to track it after we sort out this module.