cdc: client_test.TestHandleError is not stable #177
Description
Bug Report
1. Describe the bug
client_test.TestHandleError is not stable.
CI link: https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true
From the stack, goroutine raise a exception from here, and stuck at here.
goroutine 2021 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc0008bb358?)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc0008bb350)
/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func1()
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:534 +0x85
runtime.Goexit()
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:500 +0x1be
github.com/pingcap/check.(*C).stopNow(...)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:108
github.com/pingcap/check.(*C).FailNow(0xc00055d4a0)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/helpers.go:37 +0x45
github.com/pingcap/check.(*C).Fatalf(0xc000b114a0?, {0x5b19d2c, 0x25}, {0x0, 0x0, 0x0})
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/helpers.go:151 +0x157
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError(0x3600000000000000?, 0xc00055d4a0)
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:677 +0x2809
reflect.Value.call({0x5a59ee0?, 0xc000711300?, 0xc000bdccd0?}, {0x5aa2b5e, 0x4}, {0xc000879eb8, 0x1, 0xc000bdccf0?})
/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:556 +0xd9d
reflect.Value.Call({0x5a59ee0?, 0xc000711300?, 0xc000481a40?}, {0xc000bdceb8, 0x1, 0x1})
/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:[339](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:340) +0xd8
github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc00055d4a0)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:850 +0xa72
github.com/pingcap/check.(*suiteRunner).forkCall.func1()
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:739 +0x11f
created by github.com/pingcap/check.(*suiteRunner).forkCall
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:734 +0x5de
Full stack:
=== FAIL: cdc/kv Test (unknown)
Error: 7/22 08:56:34.747 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75"]
Error: 7/22 08:56:35.295 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75"]
Error: 7/22 08:56:36.557 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:33771, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75"]
Error: 7/22 08:56:44.541 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75"]
coverage: 63.9% of statements
panic: test timed out after 30m0s
goroutine 2208 [running]:
testing.(*M).startAlarm.func1()
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:2029 +0xbb
created by time.goFunc
/opt/hostedtoolcache/go/1.18.3/x64/src/time/sleep.go:176 +0x48
goroutine 1 [chan receive, 30 minutes]:
testing.(*T).Run(0xc000b78b60, {0x5aa2f56, 0x4}, 0x5bd2c78)
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1487 +0x750
testing.runTests.func1(0x0?)
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1839 +0x9a
testing.tRunner(0xc000b78b60, 0xc0008dfb68)
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
testing.runTests(0xc000a790e0?, {0x8a65680, 0x2, 0x2}, {0x40?, 0x55ed100?, 0x91ae5a0?})
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1837 +0x7e5
testing.(*M).Run(0xc000a790e0)
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1719 +0xa72
main.main()
_testmain.go:125 +0x3aa
goroutine 1966 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/raft.go:174 +0x259
created by go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/raft.go:169 +0xed
goroutine 2021 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc0008bb358?)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc0008bb350)
/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func1()
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:534 +0x85
runtime.Goexit()
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:500 +0x1be
github.com/pingcap/check.(*C).stopNow(...)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:108
github.com/pingcap/check.(*C).FailNow(0xc00055d4a0)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/helpers.go:37 +0x45
github.com/pingcap/check.(*C).Fatalf(0xc000b114a0?, {0x5b19d2c, 0x25}, {0x0, 0x0, 0x0})
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/helpers.go:151 +0x157
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError(0x3600000000000000?, 0xc00055d4a0)
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:677 +0x2809
reflect.Value.call({0x5a59ee0?, 0xc000711300?, 0xc000bdccd0?}, {0x5aa2b5e, 0x4}, {0xc000879eb8, 0x1, 0xc000bdccf0?})
/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:556 +0xd9d
reflect.Value.Call({0x5a59ee0?, 0xc000711300?, 0xc000481a40?}, {0xc000bdceb8, 0x1, 0x1})
/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:339 +0xd8
github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc00055d4a0)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:850 +0xa72
github.com/pingcap/check.(*suiteRunner).forkCall.func1()
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:739 +0x11f
created by github.com/pingcap/check.(*suiteRunner).forkCall
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:734 +0x5de
goroutine 2032 [select]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncWatchersLoop(0xc000202960)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/mvcc/watchable_store.go:233 +0x249
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/mvcc/watchable_store.go:95 +0xb75
goroutine 26 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
/home/runner/go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
/home/runner/go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:410 +0x265
goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001c3e80)
/home/runner/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0x177
created by go.opencensus.io/stats/view.init.0
/home/runner/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0xf7
goroutine 1965 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000a9a180)
/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/v3@v3.5.4/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/v3@v3.5.4/schedule/schedule.go:70 +0x2f7
goroutine 2042 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).linearizableReadLoop(0xc0002e3c00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/v3_server.go:738 +0x178
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2055 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16048?, 0xc00147e4c0?}, 0xc0006b5f80?, 0xc000a20000?})
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000bb9c00, {0x6f18c88, 0xc000d9e000})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:780 +0x5cb
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func2()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:120 +0x66
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:120 +0x8d8
goroutine 2080 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc000b94d50?)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000b94d48)
/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000b94d40)
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed(0xc0001b7a40, {0x6f19750, 0xc000b94540}, 0x64)
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:590 +0xaef
github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).EventFeed(0xc000a2b9a0, {0x6f19750, 0xc000b94540}, {{0xc0008bb8ff, 0x1, 0x1}, {0xc0008bb90f, 0x1, 0x1}}, 0x64, ...)
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:405 +0x179
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func2()
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:568 +0x27c
created by github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:566 +0xe3f
goroutine 2098 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16048?, 0xc00147e4c0?}, 0xc0006b4d20?, 0xc000c304e0?})
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc000c0c000, {0x6f18c88, 0xc000d9f280})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:[303](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:304)9 +0x5a7
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func3()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:137 +0x59
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:137 +0xf58
goroutine 33 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e3f0, {0x6f19750, 0xc000c04140})
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 32 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e360, {0x6f19750, 0xc000c04140})
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 31 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e2d0, {0x6f19750, 0xc000c04140})
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 30 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e240, {0x6f19750, 0xc000c04140})
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 29 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc000c04190?)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000c04188)
/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000c04180)
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run(0xc00077c150, {0x6f19750, 0xc000c040c0})
/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:75 +0x2c8
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool.func1()
/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:827 +0x72
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 27 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc000c04110?)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000c04108)
/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000c04100)
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool({0x6f19788, 0xc000054128})
/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:829 +0x16f
github.com/tikv/migration/cdc/cdc/kv.Test.func1()
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:62 +0x3d
created by github.com/tikv/migration/cdc/cdc/kv.Test
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:61 +0x66
goroutine 153 [select, 29 minutes]:
github.com/pingcap/check.(*suiteRunner).doRun(0xc000b1cbd0, 0x0)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:667 +0x2a5
github.com/pingcap/check.(*suiteRunner).run(0xc000b1cbd0)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:696 +0xe6
github.com/pingcap/check.Run({0x5a59ee0, 0xc000711300}, 0xc000c00248?)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/run.go:150 +0x4a
github.com/pingcap/check.RunAll(0xc000c13d02?)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/run.go:117 +0x259
github.com/pingcap/check.TestingT(0xc000b78d00)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/run.go:99 +0x647
github.com/tikv/migration/cdc/cdc/kv.Test(0x0?)
/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:64 +0x70
testing.tRunner(0xc000b78d00, 0x5bd2c78)
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
created by testing.(*T).Run
/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1486 +0x725
goroutine 2087 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d318, 0x72)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0013de218, 0xc0008c2000?, 0x0)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0013de200, {0xc0008c2000, 0x8000, 0x8000})
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc0013de200, {0xc0008c2000, 0x8000, 0x8000})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc0004d6038, {0xc0008c2000, 0x8000, 0x8000})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0007a61e0, {0xc000af6040, 0x9, 0x9})
/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6d00, 0xc0007a61e0}, {0xc000af6040, 0x9, 0x9}, 0x9)
/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000af6040?, 0x9?, 0x9?}, {0x6ef6d00?, 0xc0007a61e0?})
/home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20220607020251-c690dde0001d/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000af6000)
/home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20220607020251-c690dde0001d/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001660000)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_client.go:1498 +0x585
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_client.go:365 +0x2772
goroutine 2040 [select, 11 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.monitorFileDescriptor(0xc0002e3c00?, 0xc000bf7aa0)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/metrics.go:237 +0x3f9
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Start.func3()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:802 +0x65
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2024 [select]:
go.etcd.io/etcd/server/v3/mvcc/backend.(*backend).run(0xc000b13b00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/mvcc/backend/backend.go:421 +0x235
created by go.etcd.io/etcd/server/v3/mvcc/backend.newBackend
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/mvcc/backend/backend.go:229 +0xd36
goroutine 1969 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/fileutil/purge.go:44 +0x4f2
goroutine 1967 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/fileutil/purge.go:44 +0x4f2
goroutine 2085 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*sizedRegionRouter).Run(0xc000dab9b0, {0x6f19750, 0xc000b94d00})
/home/runner/work/migration/migration/cdc/cdc/kv/token_region.go:151 +0x1fb
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func5()
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:584 +0x82
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 1968 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/fileutil/purge.go:44 +0x4f2
goroutine 2081 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).dispatchRequest(0xc0001b7a40, {0x6f19750, 0xc000b94d00})
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:870 +0x4d6
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func1()
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:511 +0x5e
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 226 [select, 29 minutes]:
github.com/pingcap/check.(*resultTracker)._loopRoutine(0xc000b1cb40)
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:464 +0x153
created by github.com/pingcap/check.(*resultTracker).start
/home/runner/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:444 +0x8f
goroutine 2048 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d7c8, 0x72)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000b19918, 0x1?, 0x0)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000b19900)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc000b19900)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc001347770)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc001347770)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*rwTimeoutListener).Accept(0xc000ad4f20)
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/transport/timeout_listener.go:36 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc000a1c640)
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:557 +0x178
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc000ad4f00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:585 +0x176
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:579 +0xe32
goroutine 2041 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorVersions(0xc0002e3c00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2465 +0x149
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2090 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000a2a410, 0x1)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007a65a0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_server.go:332 +0x1cd
created by google.golang.org/grpc/internal/transport.NewServerTransport
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_server.go:329 +0x269b
goroutine 2092 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d138, 0x72)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0013de598, 0xc0008d0000?, 0x0)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0013de580, {0xc0008d0000, 0x8000, 0x8000})
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc0013de580, {0xc0008d0000, 0x8000, 0x8000})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000415860, {0xc0008d0000, 0x8000, 0x8000})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
github.com/soheilhy/cmux.(*bufferedReader).Read(0xc0009f88d0, {0xc0008d0000, 0x8000, 0x8000})
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/buffer.go:53 +0x2f8
github.com/soheilhy/cmux.(*MuxConn).Read(0xc0009f88c0, {0xc0008d0000, 0x8000, 0x8000})
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:297 +0x51
bufio.(*Reader).Read(0xc0007a62a0, {0xc000af6120, 0x9, 0x9})
/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6d00, 0xc0007a62a0}, {0xc000af6120, 0x9, 0x9}, 0x9)
/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000af6120?, 0x9?, 0x9?}, {0x6ef6d00?, 0xc0007a62a0?})
/home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20220607020251-c690dde0001d/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000af60e0)
/home/runner/go/pkg/mod/golang.org/x/net@v0.0.0-20220607020251-c690dde0001d/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000cbba0, 0x4ed04d6?, 0xc000b06f80?)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_server.go:611 +0x145
google.golang.org/grpc.(*Server).serveStreams(0xc000bb9c00, {0x6f31410?, 0xc0000cbba0})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:906 +0x267
google.golang.org/grpc.(*Server).handleRawConn.func1()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:848 +0x65
created by google.golang.org/grpc.(*Server).handleRawConn
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:847 +0x245
goroutine 2065 [chan receive, 29 minutes]:
go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:252 +0x71
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:251 +0x65f
goroutine 2044 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorDowngrade(0xc0002e3c00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2586 +0x156
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2028 [select]:
go.etcd.io/etcd/server/v3/lease.(*lessor).runLoop(0xc000b13c80)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/lease/lessor.go:619 +0x195
created by go.etcd.io/etcd/server/v3/lease.newLessor
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/lease/lessor.go:237 +0x5f7
goroutine 2047 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f160a8?, 0xc000ad4f20?}, 0xc0006e5e60?, 0xc000bf7da0?})
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc000af6c40, {0x6f18c88, 0xc000ad53a0})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x5a7
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:551 +0xa4f
goroutine 2046 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f160a8?, 0xc000ad4f20?}, 0xc0006e5e00?, 0xc000bf7d40?})
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000bb9880, {0x6f18c88, 0xc000ad5340})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:780 +0x5cb
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:545 +0x63f
goroutine 2039 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile(0xc0002e3c00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:876 +0x519
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2025 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/wal.(*filePipeline).run(0xc000b3c9c0)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/wal/file_pipeline.go:97 +0x1e5
created by go.etcd.io/etcd/server/v3/wal.newFilePipeline
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/wal/file_pipeline.go:55 +0x4af
goroutine 2045 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2033 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncVictimsLoop(0xc000202960)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/mvcc/watchable_store.go:259 +0x209
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/mvcc/watchable_store.go:96 +0xbf2
goroutine 2036 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run(0xc0002e3c00)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:1117 +0xb65
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).start
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:861 +0x159e
goroutine 2026 [select]:
go.etcd.io/etcd/raft/v3.(*node).run(0xc0006e56e0)
/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/v3@v3.5.4/node.go:341 +0xc1d
created by go.etcd.io/etcd/raft/v3.StartNode
/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/v3@v3.5.4/node.go:230 +0x31e
goroutine 2054 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/server.go:2688 +0x1d2
goroutine 2029 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc0006e5a40)
/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/v3@v3.5.4/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/v3@v3.5.4/schedule/schedule.go:70 +0x2f7
goroutine 2088 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000a2a320, 0x1)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007a6240)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_client.go:415 +0x106
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_client.go:413 +0x[312](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:313)9
goroutine 2091 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000cbba0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_server.go:1131 +0x3cf
created by google.golang.org/grpc/internal/transport.NewServerTransport
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/internal/transport/http2_server.go:[341](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:342) +0x271a
goroutine 2082 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore(0xc0001b7a40, {0x6f19750?, 0xc000b94d00?}, 0xc000b94d40)
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:690 +0x21c
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2()
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515 +0x70
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 2052 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d5e8, 0x72)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000b19998, 0xc000817501?, 0x0)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000b19980)
/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc000b19980)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc001[347](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:348)7d0)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc0013477d0)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*limitListener).Accept(0xc0013477e8)
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/transport/limit_listen.go:46 +0x6c
go.etcd.io/etcd/client/pkg/v3/transport.(*keepaliveListener).Accept(0xc00147e4c0)
/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/v3@v3.5.4/transport/keepalive_listener.go:49 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc000b6ba90)
/home/runner/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve(0xc0009f0000, 0xc0002e3c00, 0xc0004bb960, {0x6efd080, 0xc000b47980}, 0xc00147f080, {0xc000ad5540, 0x2, 0x2})
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/serve.go:195 +0x1f65
go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1(0x0?)
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:734 +0x145
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients
/home/runner/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/embed/etcd.go:733 +0x126f
goroutine 2056 [select, 29 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000b980c0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/balancer_conn_wrappers.go:112 +0x150
created by google.golang.org/grpc.newCCBalancerWrapper
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.2/balancer_conn_wrappers.go:73 +0x4b7
goroutine 2084 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4()
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:546 +0x289
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:72 +0x12f
goroutine 2083 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func3()
/home/runner/work/migration/migration/cdc/cdc/kv/client.go:520 +0x169
golang.org/x/sync/errgroup.(*Group).Go.func1()
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-202206011[502](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:503)17-0de[741](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:742)cfad7f/errgroup/errgroup.go:72 +0x12f
2. Minimal reproduce step (Required)
3. What did you see instead (Required)
4. What did you expect to see? (Required)
5. What is your migration tool and TiKV version? (Required)
- TiKV:
- TiKV CDC:
- TiKV BR:
- TiKV Online Bulk Load: