Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data race in hotregion #4159

Closed
nolouch opened this issue Sep 26, 2021 · 0 comments · Fixed by #4167
Closed

data race in hotregion #4159

nolouch opened this issue Sep 26, 2021 · 0 comments · Fixed by #4167
Assignees
@rleungx
Labels
status/TODO The issue will be done in the future. type/bug The issue is confirmed as a bug.

Comments

@nolouch
Copy link
Contributor

nolouch commented Sep 26, 2021

Bug Report

What did you do?

ci failed: https://ci2.pingcap.net/blue/organizations/jenkins/pd_test/detail/pd_test/2574/pipeline

What did you expect to see?

ci pass

What did you see instead?

[2021-09-26T06:27:47.246Z] ==================

[2021-09-26T06:27:47.246Z] WARNING: DATA RACE

[2021-09-26T06:27:47.246Z] Write at 0x00c00073a1e0 by goroutine 64:

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotRegionSchedulerConfig).checkQuerySupport()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region_config.go:353 +0x5dc

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*balanceSolver).getPriorities()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:414 +0xcd

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*balanceSolver).init()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:406 +0xa79

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.newBalanceSolver()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:439 +0xcc

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotScheduler).balanceHotReadRegions()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:262 +0x45

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotScheduler).dispatch()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:167 +0x1b0

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotScheduler).Schedule()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:156 +0x26e

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/cluster.(*scheduleController).Schedule()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster/coordinator.go:855 +0x196

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/cluster.(*coordinator).runScheduler()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster/coordinator.go:778 +0x376

[2021-09-26T06:27:47.246Z] 

[2021-09-26T06:27:47.246Z] Previous read at 0x00c00073a1e0 by goroutine 34:

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotRegionSchedulerConfig).getValidConf()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region_config.go:103 +0x64

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotRegionSchedulerConfig).handleGetConfig()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region_config.go:292 +0x173

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotRegionSchedulerConfig).handleGetConfig-fm()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region_config.go:288 +0x68

[2021-09-26T06:27:47.246Z]   net/http.HandlerFunc.ServeHTTP()

[2021-09-26T06:27:47.246Z]       /usr/local/go/src/net/http/server.go:2069 +0x51

[2021-09-26T06:27:47.246Z]   github.com/gorilla/mux.(*Router).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x132

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotRegionSchedulerConfig).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region_config.go:285 +0x2ab

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/schedulers.(*hotScheduler).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/schedulers/hot_region.go:136 +0x77

[2021-09-26T06:27:47.246Z]   net/http.StripPrefix.func1()

[2021-09-26T06:27:47.246Z]       /usr/local/go/src/net/http/server.go:2112 +0x535

[2021-09-26T06:27:47.246Z]   net/http.HandlerFunc.ServeHTTP()

[2021-09-26T06:27:47.246Z]       /usr/local/go/src/net/http/server.go:2069 +0x51

[2021-09-26T06:27:47.246Z]   net/http.(*ServeMux).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /usr/local/go/src/net/http/server.go:2448 +0xaf

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/server/api.(*schedulerConfigHandler).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/api/scheduler.go:343 +0xa6

[2021-09-26T06:27:47.246Z]   github.com/gorilla/mux.(*Router).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x132

[2021-09-26T06:27:47.246Z]   github.com/urfave/negroni.Wrap.func1()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x68

[2021-09-26T06:27:47.246Z]   github.com/urfave/negroni.HandlerFunc.ServeHTTP()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x5b

[2021-09-26T06:27:47.246Z]   github.com/urfave/negroni.middleware.ServeHTTP()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x116

[2021-09-26T06:27:47.246Z]   github.com/urfave/negroni.middleware.ServeHTTP-fm()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:37 +0x7e

[2021-09-26T06:27:47.246Z]   github.com/tikv/pd/pkg/apiutil/serverapi.(*redirector).ServeHTTP()

[2021-09-26T06:27:47.246Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/pkg/apiutil/serverapi/middleware.go:98 +0xb4b

[2021-09-26T06:27:47.246Z]   github.com/urfave/negroni.middleware.ServeHTTP()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x116

[2021-09-26T06:27:47.246Z]   github.com/urfave/negroni.middleware.ServeHTTP-fm()

[2021-09-26T06:27:47.246Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:37 +0x7e

[2021-09-26T06:27:47.247Z]   github.com/tikv/pd/pkg/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/pkg/apiutil/serverapi/middleware.go:55 +0x11e

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.middleware.ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x116

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.(*Negroni).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x186

[2021-09-26T06:27:47.247Z]   github.com/gorilla/mux.(*Router).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x132

[2021-09-26T06:27:47.247Z]   github.com/gorilla/mux.(*Router).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x132

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.Wrap.func1()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x68

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.HandlerFunc.ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x5b

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.middleware.ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x116

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.middleware.ServeHTTP-fm()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:37 +0x7e

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.(*Recovery).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/recovery.go:193 +0xd6

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.middleware.ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x116

[2021-09-26T06:27:47.247Z]   github.com/urfave/negroni.(*Negroni).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x186

[2021-09-26T06:27:47.247Z]   net/http.(*ServeMux).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /usr/local/go/src/net/http/server.go:2448 +0xaf

[2021-09-26T06:27:47.247Z]   go.etcd.io/etcd/embed.(*accessController).ServeHTTP()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/embed/serve.go:359 +0x2ec

[2021-09-26T06:27:47.247Z]   net/http.serverHandler.ServeHTTP()

[2021-09-26T06:27:47.247Z]       /usr/local/go/src/net/http/server.go:2887 +0xca

[2021-09-26T06:27:47.247Z]   net/http.(*conn).serve()

[2021-09-26T06:27:47.247Z]       /usr/local/go/src/net/http/server.go:1952 +0x87d

[2021-09-26T06:27:47.247Z] 

[2021-09-26T06:27:47.247Z] Goroutine 64 (running) created at:

[2021-09-26T06:27:47.247Z]   github.com/tikv/pd/server/cluster.(*coordinator).addScheduler()

[2021-09-26T06:27:47.247Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster/coordinator.go:625 +0x2d6

[2021-09-26T06:27:47.247Z]   github.com/tikv/pd/server/cluster.(*coordinator).run()

[2021-09-26T06:27:47.247Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster/coordinator.go:370 +0x1e04

[2021-09-26T06:27:47.247Z]   github.com/tikv/pd/server/cluster.(*RaftCluster).runCoordinator()

[2021-09-26T06:27:47.247Z]       /home/jenkins/agent/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster/cluster.go:368 +0x12c

[2021-09-26T06:27:47.247Z] 

[2021-09-26T06:27:47.247Z] Goroutine 34 (running) created at:

[2021-09-26T06:27:47.247Z]   net/http.(*Server).Serve()

[2021-09-26T06:27:47.247Z]       /usr/local/go/src/net/http/server.go:3013 +0x644

[2021-09-26T06:27:47.247Z]   go.etcd.io/etcd/embed.(*serveCtx).serve.func3()

[2021-09-26T06:27:47.247Z]       /nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/embed/serve.go:135 +0x4c

[2021-09-26T06:27:47.247Z] ==================

What version of PD are you using (pd-server -V)?

master
@nolouch nolouch added the type/bug The issue is confirmed as a bug. label Sep 26, 2021
@nolouch nolouch mentioned this issue Sep 26, 2021
Closed
5 tasks
@nolouch nolouch added the status/TODO The issue will be done in the future. label Sep 26, 2021
@rleungx rleungx assigned rleungx and unassigned lhy1024 Sep 27, 2021
@rleungx rleungx mentioned this issue Sep 27, 2021
Merged
@ti-chi-bot ti-chi-bot mentioned this issue Sep 27, 2021
Merged
@HunDunDM HunDunDM mentioned this issue Sep 28, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rleungx rleungx

Labels
status/TODO The issue will be done in the future. type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: fix race in hot region config rleungx/pd
3 participants
@nolouch @lhy1024 @rleungx