TestEtcdScaleInAndOut failed

[rleungx]

Flaky Test

Which jobs are failing

--- FAIL: TestEtcdScaleInAndOut (0.45s)
    etcdutil_test.go:310: 
        	Error Trace:	/home/runner/work/pd/pd/pkg/utils/etcdutil/etcdutil_test.go:310
        	Error:      	Received unexpected error:
        	            	[PD:etcd:ErrEtcdMemberRemove]etcdserver: server stopped: etcdserver: server stopped
        	            	github.com/pingcap/errors.AddStack
        	            		/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174
        	            	github.com/pingcap/errors.(*Error).GenWithStackByCause
        	            		/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:307
        	            	github.com/tikv/pd/pkg/utils/etcdutil.RemoveEtcdMember
        	            		/home/runner/work/pd/pd/pkg/utils/etcdutil/etcdutil.go:124
        	            	github.com/tikv/pd/pkg/utils/etcdutil.TestEtcdScaleInAndOut
        	            		/home/runner/work/pd/pd/pkg/utils/etcdutil/etcdutil_test.go:309
        	            	testing.tRunner
        	            		/opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1576
        	            	runtime.goexit
        	            		/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/asm_amd64.s:1598
        	Test:       	TestEtcdScaleInAndOut

CI link

https://github.com/tikv/pd/actions/runs/5748065431/job/15580210773?pr=6891

Reason for failure (if possible)

Anything else

[lhy1024]

It met server stopped before we removed this server from the cluster.

There is only a log with it 2023-08-03T07:55:34.9662606Z [2023/08/03 07:54:49.387 +00:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=endpoint://client-aa84e8be-f741-46c2-b8ce-334a8bf853e9/127.0.0.1:33107] [attempt=0] [error="rpc error: code = Unavailable desc = etcdserver: server stopped"]

[lhy1024]

2023-09-04T09:50:39.4700298Z --- FAIL: TestEtcdClientSync (0.67s)
2023-09-04T09:50:39.4700670Z     etcdutil_test.go:186: 
2023-09-04T09:50:39.4701471Z         	Error Trace:	/home/runner/work/pd/pd/pkg/utils/etcdutil/etcdutil_test.go:186
2023-09-04T09:50:39.4702048Z         	Error:      	Received unexpected error:
2023-09-04T09:50:39.4703120Z         	            	[PD:etcd:ErrEtcdMemberRemove]etcdserver: server stopped: etcdserver: server stopped
2023-09-04T09:50:39.4703885Z         	            	github.com/pingcap/errors.AddStack
2023-09-04T09:50:39.4705351Z         	            		/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174
2023-09-04T09:50:39.4706239Z         	            	github.com/pingcap/errors.(*Error).GenWithStackByCause
2023-09-04T09:50:39.4707747Z         	            		/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:307
2023-09-04T09:50:39.4708657Z         	            	github.com/tikv/pd/pkg/utils/etcdutil.RemoveEtcdMember
2023-09-04T09:50:39.4709654Z         	            		/home/runner/work/pd/pd/pkg/utils/etcdutil/etcdutil.go:129
2023-09-04T09:50:39.4710540Z         	            	github.com/tikv/pd/pkg/utils/etcdutil.TestEtcdClientSync
2023-09-04T09:50:39.4711589Z         	            		/home/runner/work/pd/pd/pkg/utils/etcdutil/etcdutil_test.go:185
2023-09-04T09:50:39.4712122Z         	            	testing.tRunner
2023-09-04T09:50:39.4713095Z         	            		/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595
2023-09-04T09:50:39.4713647Z         	            	runtime.goexit
2023-09-04T09:50:39.4714535Z         	            		/opt/hostedtoolcache/go/1.21.0/x64/src/runtime/asm_amd64.s:1650
2023-09-04T09:50:39.4715157Z         	Test:       	TestEtcdClientSync

https://github.com/tikv/pd/actions/runs/6071808375/job/16470578332?pr=7040