Skip to content

Commit

Permalink
pkg/node: fix concurrent access of entry node
Browse files Browse the repository at this point in the history 
Fixes the following data race:

```
Read at 0x00c00079f200 by goroutine 43:
  github.com/cilium/cilium/pkg/node/manager.(*Manager).StartNeighborRefresh.func1.1()
      /go/src/github.com/cilium/cilium/pkg/node/manager/manager.go:581 +0xc4

Previous write at 0x00c00079f200 by goroutine 166:
  github.com/cilium/cilium/pkg/node/manager.(*Manager).NodeUpdated()
      /go/src/github.com/cilium/cilium/pkg/node/manager/manager.go:414 +0x824
  github.com/cilium/cilium/pkg/k8s/watchers.(*K8sWatcher).ciliumNodeInit.func2()
      /go/src/github.com/cilium/cilium/pkg/k8s/watchers/cilium_node.go:71 +0x2d4
  k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:238 +0xa5
  k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate()
      <autogenerated>:1 +0x2e
  github.com/cilium/cilium/pkg/k8s/informer.NewInformerWithStore.func1()
      /go/src/github.com/cilium/cilium/pkg/k8s/informer/informer.go:114 +0x389
  k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:507 +0x50d
  k8s.io/client-go/tools/cache.(*controller).processLoop()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:183 +0x83
  k8s.io/client-go/tools/cache.(*controller).processLoop-fm()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:181 +0x44
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba
  k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114
  k8s.io/apimachinery/pkg/util/wait.Until()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x507
  k8s.io/client-go/tools/cache.(*controller).Run()
      /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:154 +0x4a9

Goroutine 43 (running) created at:
  github.com/cilium/cilium/pkg/node/manager.(*Manager).StartNeighborRefresh.func1()
      /go/src/github.com/cilium/cilium/pkg/node/manager/manager.go:578 +0x368
  github.com/cilium/cilium/pkg/controller.(*Controller).runController()
      /go/src/github.com/cilium/cilium/pkg/controller/controller.go:208 +0xd10

Goroutine 166 (running) created at:
  github.com/cilium/cilium/pkg/k8s/watchers.(*K8sWatcher).ciliumNodeInit()
      /go/src/github.com/cilium/cilium/pkg/k8s/watchers/cilium_node.go:101 +0x184
```

Fixes: 5ec4d51 ("daemon, node: refresh neighbor by sending arping periodically")
Signed-off-by: André Martins <andre@cilium.io>
  • Loading branch information
@aanm
aanm committed Jan 14, 2021
1 parent ce5e705 commit a544c51
Showing 1 changed file with 7 additions and 4 deletions.
11 changes: 7 additions & 4 deletions pkg/node/manager/manager.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -572,14 +572,17 @@ func (m *Manager) StartNeighborRefresh(nh datapath.NodeHandler) {
m.mutex.RLock()
defer m.mutex.RUnlock()
for _, entry := range m.nodes {
if entry.node.IsLocal() {
entry.mutex.Lock()
entryNode := entry.node
entry.mutex.Unlock()
if entryNode.IsLocal() {
continue
}
go func(c context.Context, e *nodeEntry) {
go func(c context.Context, e nodeTypes.Node) {
n := randGen.Int63n(int64(interval / 2))
time.Sleep(interval/2 + time.Duration(n))
nh.NodeNeighborRefresh(c, e.node)
}(ctx, entry)
nh.NodeNeighborRefresh(c, e)
}(ctx, entryNode)
}
return nil
},
Expand Down

0 comments on commit a544c51

Please sign in to comment.