agent: Fix data race when accessing d.monitorAgent #11823

tgraf · 2020-06-02T13:01:17Z

Fixes:

2020-04-15T08:20:51.171924491Z WARNING: DATA RACE
2020-04-15T08:20:51.171929471Z Write at 0x00c0002d89c0 by main goroutine:
2020-04-15T08:20:51.171932068Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171934565Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:521 +0x3811
2020-04-15T08:20:51.171937059Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).init()
2020-04-15T08:20:51.171939762Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:210 +0x4a4
2020-04-15T08:20:51.171942223Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171944604Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:508 +0x2fa5
2020-04-15T08:20:51.171947111Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171949522Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:411 +0x463e
2020-04-15T08:20:51.171951975Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-04-15T08:20:51.171954378Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1194 +0x345
2020-04-15T08:20:51.171956805Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-04-15T08:20:51.171959248Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:107 +0x108
2020-04-15T08:20:51.171961654Z   github.com/cilium/cilium/daemon/cmd.initEnv()
2020-04-15T08:20:51.171964065Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1073 +0x2c56
2020-04-15T08:20:51.171966471Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-04-15T08:20:51.171968921Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:105 +0xee
2020-04-15T08:20:51.171971339Z   github.com/spf13/cobra.(*Command).execute()
2020-04-15T08:20:51.171973711Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:830 +0x8e0
2020-04-15T08:20:51.171983921Z   github.com/spf13/cobra.(*Command).ExecuteC()
2020-04-15T08:20:51.17198653Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:914 +0x41a
2020-04-15T08:20:51.171989059Z   github.com/spf13/cobra.(*Command).Execute()
2020-04-15T08:20:51.171991461Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:864 +0x251
2020-04-15T08:20:51.171993892Z   github.com/cilium/cilium/daemon/cmd.Execute()
2020-04-15T08:20:51.171996301Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:138 +0x232
2020-04-15T08:20:51.171998765Z   main.main()
2020-04-15T08:20:51.172001107Z       /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f
2020-04-15T08:20:51.172003564Z
2020-04-15T08:20:51.172005844Z Previous read at 0x00c0002d89c0 by goroutine 82:
2020-04-15T08:20:51.172008245Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).SendNotification()
2020-04-15T08:20:51.172010648Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:633 +0xc0
2020-04-15T08:20:51.172013085Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).notifyMonitor()
2020-04-15T08:20:51.172015524Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:113 +0x288
2020-04-15T08:20:51.172019826Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).OnIPIdentityCacheChange()
2020-04-15T08:20:51.172022388Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:142 +0x21e
2020-04-15T08:20:51.172024963Z   github.com/cilium/cilium/pkg/ipcache.(*IPCache).Upsert()
2020-04-15T08:20:51.172027361Z       /go/src/github.com/cilium/cilium/pkg/ipcache/ipcache.go:278 +0xaa0
2020-04-15T08:20:51.172029767Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).syncEndpointsAndHostIPs()
2020-04-15T08:20:51.17203218Z       /go/src/github.com/cilium/cilium/daemon/cmd/datapath.go:275 +0xcd0
2020-04-15T08:20:51.172034647Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).init.func2()
2020-04-15T08:20:51.17203703Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:224 +0x41
2020-04-15T08:20:51.172039455Z   github.com/cilium/cilium/pkg/controller.(*Controller).runController()
2020-04-15T08:20:51.172041885Z       /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xc71

Fixes: #10987

Fixes: ``` 2020-04-15T08:20:51.171924491Z WARNING: DATA RACE 2020-04-15T08:20:51.171929471Z Write at 0x00c0002d89c0 by main goroutine: 2020-04-15T08:20:51.171932068Z github.com/cilium/cilium/daemon/cmd.NewDaemon() 2020-04-15T08:20:51.171934565Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:521 +0x3811 2020-04-15T08:20:51.171937059Z github.com/cilium/cilium/daemon/cmd.(*Daemon).init() 2020-04-15T08:20:51.171939762Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:210 +0x4a4 2020-04-15T08:20:51.171942223Z github.com/cilium/cilium/daemon/cmd.NewDaemon() 2020-04-15T08:20:51.171944604Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:508 +0x2fa5 2020-04-15T08:20:51.171947111Z github.com/cilium/cilium/daemon/cmd.NewDaemon() 2020-04-15T08:20:51.171949522Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:411 +0x463e 2020-04-15T08:20:51.171951975Z github.com/cilium/cilium/daemon/cmd.runDaemon() 2020-04-15T08:20:51.171954378Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1194 +0x345 2020-04-15T08:20:51.171956805Z github.com/cilium/cilium/daemon/cmd.glob..func1() 2020-04-15T08:20:51.171959248Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:107 +0x108 2020-04-15T08:20:51.171961654Z github.com/cilium/cilium/daemon/cmd.initEnv() 2020-04-15T08:20:51.171964065Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1073 +0x2c56 2020-04-15T08:20:51.171966471Z github.com/cilium/cilium/daemon/cmd.glob..func1() 2020-04-15T08:20:51.171968921Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:105 +0xee 2020-04-15T08:20:51.171971339Z github.com/spf13/cobra.(*Command).execute() 2020-04-15T08:20:51.171973711Z /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:830 +0x8e0 2020-04-15T08:20:51.171983921Z github.com/spf13/cobra.(*Command).ExecuteC() 2020-04-15T08:20:51.17198653Z /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:914 +0x41a 2020-04-15T08:20:51.171989059Z github.com/spf13/cobra.(*Command).Execute() 2020-04-15T08:20:51.171991461Z /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:864 +0x251 2020-04-15T08:20:51.171993892Z github.com/cilium/cilium/daemon/cmd.Execute() 2020-04-15T08:20:51.171996301Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:138 +0x232 2020-04-15T08:20:51.171998765Z main.main() 2020-04-15T08:20:51.172001107Z /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f 2020-04-15T08:20:51.172003564Z 2020-04-15T08:20:51.172005844Z Previous read at 0x00c0002d89c0 by goroutine 82: 2020-04-15T08:20:51.172008245Z github.com/cilium/cilium/daemon/cmd.(*Daemon).SendNotification() 2020-04-15T08:20:51.172010648Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:633 +0xc0 2020-04-15T08:20:51.172013085Z github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).notifyMonitor() 2020-04-15T08:20:51.172015524Z /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:113 +0x288 2020-04-15T08:20:51.172019826Z github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).OnIPIdentityCacheChange() 2020-04-15T08:20:51.172022388Z /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:142 +0x21e 2020-04-15T08:20:51.172024963Z github.com/cilium/cilium/pkg/ipcache.(*IPCache).Upsert() 2020-04-15T08:20:51.172027361Z /go/src/github.com/cilium/cilium/pkg/ipcache/ipcache.go:278 +0xaa0 2020-04-15T08:20:51.172029767Z github.com/cilium/cilium/daemon/cmd.(*Daemon).syncEndpointsAndHostIPs() 2020-04-15T08:20:51.17203218Z /go/src/github.com/cilium/cilium/daemon/cmd/datapath.go:275 +0xcd0 2020-04-15T08:20:51.172034647Z github.com/cilium/cilium/daemon/cmd.(*Daemon).init.func2() 2020-04-15T08:20:51.17203703Z /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:224 +0x41 2020-04-15T08:20:51.172039455Z github.com/cilium/cilium/pkg/controller.(*Controller).runController() 2020-04-15T08:20:51.172041885Z /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xc71 ``` Fixes: #10987 Signed-off-by: Thomas Graf <thomas@cilium.io>

borkmann

Thanks for fixing!

aanm · 2020-06-02T13:21:27Z

test-me-please

coveralls · 2020-06-02T13:43:15Z

Coverage increased (+0.03%) to 36.913% when pulling 717d083 on pr/tgraf/fix-monitor-agent-race into e57ae1c on master.

tgraf added kind/bug This is a bug in the Cilium logic. priority/release-blocker release-note/misc This PR makes changes that have no direct user impact. labels Jun 2, 2020

tgraf requested review from a team June 2, 2020 13:01

maintainer-s-little-helper bot added this to In progress in 1.8.0 Jun 2, 2020

maintainer-s-little-helper bot added this to Needs backport from master in 1.8.0 Jun 2, 2020

maintainer-s-little-helper bot added this to Needs backport from master in 1.7.5 Jun 2, 2020

borkmann approved these changes Jun 2, 2020

View reviewed changes

maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jun 2, 2020

borkmann merged commit 870894a into master Jun 2, 2020

1.8.0 automation moved this from In progress to Merged Jun 2, 2020

borkmann deleted the pr/tgraf/fix-monitor-agent-race branch June 2, 2020 20:46

tklauser mentioned this pull request Jun 3, 2020

v1.7 backports 2020-06-03 #11855

Merged

tklauser added backport-pending/1.7 and removed needs-backport/1.7 labels Jun 3, 2020

maintainer-s-little-helper bot moved this from Needs backport from master to Backport pending to v1.7 in 1.7.5 Jun 3, 2020

tklauser mentioned this pull request Jun 3, 2020

v1.8 backports 2020-06-03 #11856

Merged

tklauser added backport-pending/1.8 and removed needs-backport/1.8 labels Jun 3, 2020

maintainer-s-little-helper bot moved this from Needs backport from master to Backport pending to v1.8 in 1.8.0 Jun 3, 2020

joestringer added backport-done/1.7 and removed backport-pending/1.7 labels Jun 3, 2020

joestringer moved this from Backport pending to v1.7 to Backport done to v1.7 in 1.7.5 Jun 3, 2020

tklauser removed the backport-pending/1.8 label Jun 4, 2020

tklauser added the backport-done/1.8 label Jun 4, 2020

maintainer-s-little-helper bot moved this from Backport pending to v1.8 to Backport done to v1.8 in 1.8.0 Jun 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

agent: Fix data race when accessing d.monitorAgent #11823

agent: Fix data race when accessing d.monitorAgent #11823

tgraf commented Jun 2, 2020

borkmann left a comment

aanm commented Jun 2, 2020

coveralls commented Jun 2, 2020

agent: Fix data race when accessing d.monitorAgent #11823

agent: Fix data race when accessing d.monitorAgent #11823

Conversation

tgraf commented Jun 2, 2020

borkmann left a comment

Choose a reason for hiding this comment

aanm commented Jun 2, 2020

coveralls commented Jun 2, 2020