concurrent map iteration and map write in protokube gossip mesh

[jacksontj]

1. What kops version are you running? The command kops version, will display
this information.

$ kops version
Version 8b22e4ef9 (git-8b22e4ef9)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

v1.10.11

3. What cloud provider are you using?
AWS

One one of the running clusters I saw the following log on one of the masters:

docker[49495]: fatal error: concurrent map iteration and map write
docker[49495]: goroutine 1500 [running]:
docker[49495]: runtime.throw(0x2ddf79f, 0x26)
docker[49495]:         /usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc4208dfb70 sp=0xc4208dfb50 pc=0x42cb05
docker[49495]: runtime.mapiternext(0xc4208dfcb0)
docker[49495]:         /usr/local/go/src/runtime/hashmap.go:778 +0x6f1 fp=0xc4208dfc08 sp=0xc4208dfb70 pc=0x40b671
docker[49495]: runtime.mapiterinit(0x273f760, 0xc4205e74d0, 0xc4208dfcb0)
docker[49495]:         /usr/local/go/src/runtime/hashmap.go:768 +0x270 fp=0xc4208dfc70 sp=0xc4208dfc08 pc=0x40ad20
docker[49495]: k8s.io/kops/protokube/pkg/gossip/mesh.(*KVState).Size(0xc420c20208, 0x40f0fb)
docker[49495]:         /go/src/k8s.io/kops/protokube/pkg/gossip/mesh/mesh.pb.go:197 +0x8d fp=0xc4208dfd20 sp=0xc4208dfc70 pc=0x9bd14d
docker[49495]: k8s.io/kops/protokube/pkg/gossip/mesh.(*KVState).Marshal(0xc420c20208, 0x44d8400, 0xc420c20208, 0x7f73238f99e0, 0xc420c20208, 0xc4206b0e01)
docker[49495]:         /go/src/k8s.io/kops/protokube/pkg/gossip/mesh/mesh.pb.go:85 +0x2f fp=0xc4208dfd78 sp=0xc4208dfd20 pc=0x9bc78f
docker[49495]: k8s.io/kops/vendor/github.com/gogo/protobuf/proto.Marshal(0x44d8400, 0xc420c20208, 0xc4207fc000, 0xc4203727e8, 0xc420372868, 0x273d0c0, 0xc4214041b0)
docker[49495]:         /go/src/k8s.io/kops/vendor/github.com/gogo/protobuf/proto/encode.go:233 +0x12c fp=0xc4208dfdc0 sp=0xc4208dfd78 pc=0x97c05c
docker[49495]: k8s.io/kops/protokube/pkg/gossip/mesh.(*KVState).Encode(0xc420c20208, 0x44d0580, 0xc420c20208, 0xc4204cc140)
docker[49495]:         /go/src/k8s.io/kops/protokube/pkg/gossip/mesh/state.go:204 +0x41 fp=0xc4208dfe48 sp=0xc4208dfdc0 pc=0x9c13e1
docker[49495]: k8s.io/kops/vendor/github.com/weaveworks/mesh.(*gossipSender).deliver(0xc420672a00, 0xc420a7e060, 0xc4208dfef0, 0x0, 0x0)
docker[49495]:         /go/src/k8s.io/kops/vendor/github.com/weaveworks/mesh/gossip.go:144 +0x147 fp=0xc4208dfeb8 sp=0xc4208dfe48 pc=0x93b687
docker[49495]: k8s.io/kops/vendor/github.com/weaveworks/mesh.(*gossipSender).run(0xc420672a00, 0xc420a7e060, 0xc420077500, 0xc420077620)
docker[49495]:         /go/src/k8s.io/kops/vendor/github.com/weaveworks/mesh/gossip.go:105 +0x1f6 fp=0xc4208dffc0 sp=0xc4208dfeb8 pc=0x93b4f6
docker[49495]: runtime.goexit()
docker[49495]:         /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc4208dffc8 sp=0xc4208dffc0 pc=0x45a9f1
docker[49495]: created by k8s.io/kops/vendor/github.com/weaveworks/mesh.newGossipSender
docker[49495]:         /go/src/k8s.io/kops/vendor/github.com/weaveworks/mesh/gossip.go:94 +0x172