kube-peers IP reclaim feature can over-claim

[bboreham]

I created a cluster, scaled to 10 nodes, down to 5, back to 10, and two nodes managed to reclaim the same space at the same time.

From one node:

INFO: 2017/11/29 19:21:37.136241 Our name is ee:96:3b:d6:6d:60(ip-172-20-99-84.eu-central-1.compute.internal)
...
INFO: 2017/11/29 19:21:38.064558 [kube-peers] Added myself to peer list &{[{62:67:b2:a1:15:50 ip-172-20-127-112.eu-central-1.compute.internal} {4a:b2:c6:50:52:f5 ip-172-20-35-59.eu-central-1.compute.internal} {1a:14:77:48:b4:0d ip-172-20-58-132.eu-central-1.compute.internal} {22:9a:06:34:32:03 ip-172-20-73-46.eu-central-1.compute.internal} {ba:69:1b:1f:7b:04 ip-172-20-55-111.eu-central-1.compute.internal} {92:d9:a7:94:80:46 ip-172-20-122-130.eu-central-1.compute.internal} {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal} {e6:80:6f:a0:11:6f ip-172-20-75-150.eu-central-1.compute.internal} {ee:96:3b:d6:6d:60 ip-172-20-99-84.eu-central-1.compute.internal}]}
DEBU: 2017/11/29 19:21:38.064654 [kube-peers] Nodes that have disappeared: map[ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:38.064664 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:38.064700 [kube-peers] Noting I plan to remove  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:38.068689 [kube-peers] Nodes that have disappeared: map[ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:38.068707 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:38.068712 [kube-peers] Existing annotation ee:96:3b:d6:6d:60
DEBU: 2017/11/29 19:21:38.068720 weave DELETE to http://127.0.0.1:6784/peer/72:a8:c3:3f:68:0d with map[]
INFO: 2017/11/29 19:21:38.100040 [kube-peers] rmpeer of 72:a8:c3:3f:68:0d : 65536 IPs taken over from 72:a8:c3:3f:68:0d

INFO: 2017/11/29 19:21:38.107448 [kube-peers] Removing peer  72:a8:c3:3f:68:0d . Expecting to remove linked annotation next.
INFO: 2017/11/29 19:21:38.112162 [kube-peers] Removing annotation  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:38.115509 [kube-peers] Finished removal of  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:38.115561 [kube-peers] Nodes that have disappeared: map[]

From the other:

INFO: 2017/11/29 19:21:32.045892 Our name is e6:80:6f:a0:11:6f(ip-172-20-75-150.eu-central-1.compute.internal)
...
INFO: 2017/11/29 19:21:32.959286 [kube-peers] Added myself to peer list &{[{62:67:b2:a1:15:50 ip-172-20-127-112.eu-central-1.compute.internal} {4a:b2:c6:50:52:f5 ip-172-20-35-59.eu-central-1.compute.internal} {1a:14:77:48:b4:0d ip-172-20-58-132.eu-central-1.compute.internal} {22:9a:06:34:32:03 ip-172-20-73-46.eu-central-1.compute.internal} {16:14:90:3c:d9:e5 ip-172-20-78-194.eu-central-1.compute.internal} {62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal} {ba:69:1b:1f:7b:04 ip-172-20-55-111.eu-central-1.compute.internal} {3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal} {92:d9:a7:94:80:46 ip-172-20-122-130.eu-central-1.compute.internal} {36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal} {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal} {e6:80:6f:a0:11:6f ip-172-20-75-150.eu-central-1.compute.internal}]}
DEBU: 2017/11/29 19:21:32.959511 [kube-peers] Nodes that have disappeared: map[ip-172-20-106-72.eu-central-1.compute.internal:{3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal} ip-172-20-64-36.eu-central-1.compute.internal:{36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal} ip-172-20-78-194.eu-central-1.compute.internal:{16:14:90:3c:d9:e5 ip-172-20-78-194.eu-central-1.compute.internal} ip-172-20-99-7.eu-central-1.compute.internal:{62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal} ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:32.959550 [kube-peers] Preparing to remove disappeared peer {62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:32.959591 [kube-peers] Noting I plan to remove  62:2e:00:f5:cd:cc
DEBU: 2017/11/29 19:21:32.962975 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:32.963010 [kube-peers] Noting I plan to remove  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:32.966621 [kube-peers] Preparing to remove disappeared peer {16:14:90:3c:d9:e5 ip-172-20-78-194.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:32.966637 [kube-peers] Noting I plan to remove  16:14:90:3c:d9:e5
DEBU: 2017/11/29 19:21:32.970819 [kube-peers] Preparing to remove disappeared peer {3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:32.970834 [kube-peers] Noting I plan to remove  3e:fa:e9:fd:b2:90
DEBU: 2017/11/29 19:21:32.982891 [kube-peers] Preparing to remove disappeared peer {36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:32.982907 [kube-peers] Noting I plan to remove  36:da:1c:f4:7a:6d
DEBU: 2017/11/29 19:21:32.987067 [kube-peers] Nodes that have disappeared: map[ip-172-20-99-7.eu-central-1.compute.internal:{62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal} ip-172-20-106-72.eu-central-1.compute.internal:{3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal} ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal} ip-172-20-78-194.eu-central-1.compute.internal:{16:14:90:3c:d9:e5 ip-172-20-78-194.eu-central-1.compute.internal} ip-172-20-64-36.eu-central-1.compute.internal:{36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:32.987086 [kube-peers] Preparing to remove disappeared peer {16:14:90:3c:d9:e5 ip-172-20-78-194.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:32.987091 [kube-peers] Existing annotation e6:80:6f:a0:11:6f
DEBU: 2017/11/29 19:21:32.987144 weave DELETE to http://127.0.0.1:6784/peer/16:14:90:3c:d9:e5 with map[]
INFO: 2017/11/29 19:21:33.018783 [kube-peers] rmpeer of 16:14:90:3c:d9:e5 : 65536 IPs taken over from 16:14:90:3c:d9:e5

INFO: 2017/11/29 19:21:33.029794 [kube-peers] Removing peer  16:14:90:3c:d9:e5 . Expecting to remove linked annotation next.
INFO: 2017/11/29 19:21:33.137763 [kube-peers] Removing annotation  16:14:90:3c:d9:e5
DEBU: 2017/11/29 19:21:33.338461 [kube-peers] Finished removal of  16:14:90:3c:d9:e5
DEBU: 2017/11/29 19:21:33.338480 [kube-peers] Preparing to remove disappeared peer {36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:33.338485 [kube-peers] Noting I plan to remove  36:da:1c:f4:7a:6d
DEBU: 2017/11/29 19:21:33.537883 [kube-peers] Preparing to remove disappeared peer {62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:33.537901 [kube-peers] Noting I plan to remove  62:2e:00:f5:cd:cc
DEBU: 2017/11/29 19:21:33.741999 [kube-peers] Preparing to remove disappeared peer {3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:33.742070 [kube-peers] Noting I plan to remove  3e:fa:e9:fd:b2:90
DEBU: 2017/11/29 19:21:33.937120 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:33.937153 [kube-peers] Noting I plan to remove  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:34.136812 [kube-peers] Nodes that have disappeared: map[ip-172-20-106-72.eu-central-1.compute.internal:{3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal} ip-172-20-99-7.eu-central-1.compute.internal:{62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal} ip-172-20-64-36.eu-central-1.compute.internal:{36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal} ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:34.136834 [kube-peers] Preparing to remove disappeared peer {62:2e:00:f5:cd:cc ip-172-20-99-7.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:34.136840 [kube-peers] Existing annotation e6:80:6f:a0:11:6f
DEBU: 2017/11/29 19:21:34.136853 weave DELETE to http://127.0.0.1:6784/peer/62:2e:00:f5:cd:cc with map[]
INFO: 2017/11/29 19:21:34.150225 [kube-peers] rmpeer of 62:2e:00:f5:cd:cc : 131072 IPs taken over from 62:2e:00:f5:cd:cc

INFO: 2017/11/29 19:21:34.563965 [kube-peers] Removing peer  62:2e:00:f5:cd:cc . Expecting to remove linked annotation next.
INFO: 2017/11/29 19:21:34.737818 [kube-peers] Removing annotation  62:2e:00:f5:cd:cc
DEBU: 2017/11/29 19:21:34.936497 [kube-peers] Finished removal of  62:2e:00:f5:cd:cc
DEBU: 2017/11/29 19:21:34.936519 [kube-peers] Preparing to remove disappeared peer {36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:34.936526 [kube-peers] Noting I plan to remove  36:da:1c:f4:7a:6d
DEBU: 2017/11/29 19:21:35.136014 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:35.136036 [kube-peers] Noting I plan to remove  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:35.354098 [kube-peers] Preparing to remove disappeared peer {3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:35.354117 [kube-peers] Noting I plan to remove  3e:fa:e9:fd:b2:90
DEBU: 2017/11/29 19:21:35.564999 [kube-peers] Nodes that have disappeared: map[ip-172-20-64-36.eu-central-1.compute.internal:{36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal} ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal} ip-172-20-106-72.eu-central-1.compute.internal:{3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:35.565024 [kube-peers] Preparing to remove disappeared peer {3e:fa:e9:fd:b2:90 ip-172-20-106-72.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:35.565030 [kube-peers] Existing annotation e6:80:6f:a0:11:6f
DEBU: 2017/11/29 19:21:35.565037 weave DELETE to http://127.0.0.1:6784/peer/3e:fa:e9:fd:b2:90 with map[]
INFO: 2017/11/29 19:21:35.810983 [kube-peers] rmpeer of 3e:fa:e9:fd:b2:90 : 65536 IPs taken over from 3e:fa:e9:fd:b2:90

INFO: 2017/11/29 19:21:35.936530 [kube-peers] Removing peer  3e:fa:e9:fd:b2:90 . Expecting to remove linked annotation next.
INFO: 2017/11/29 19:21:36.138748 [kube-peers] Removing annotation  3e:fa:e9:fd:b2:90
DEBU: 2017/11/29 19:21:36.339719 [kube-peers] Finished removal of  3e:fa:e9:fd:b2:90
DEBU: 2017/11/29 19:21:36.339741 [kube-peers] Preparing to remove disappeared peer {36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:36.339746 [kube-peers] Noting I plan to remove  36:da:1c:f4:7a:6d
DEBU: 2017/11/29 19:21:36.536330 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:36.536349 [kube-peers] Noting I plan to remove  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:36.740268 [kube-peers] Nodes that have disappeared: map[ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal} ip-172-20-64-36.eu-central-1.compute.internal:{36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:36.740291 [kube-peers] Preparing to remove disappeared peer {36:da:1c:f4:7a:6d ip-172-20-64-36.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:36.740297 [kube-peers] Existing annotation e6:80:6f:a0:11:6f
DEBU: 2017/11/29 19:21:36.740305 weave DELETE to http://127.0.0.1:6784/peer/36:da:1c:f4:7a:6d with map[]
INFO: 2017/11/29 19:21:37.024804 [kube-peers] rmpeer of 36:da:1c:f4:7a:6d : 65536 IPs taken over from 36:da:1c:f4:7a:6d

INFO: 2017/11/29 19:21:37.135799 [kube-peers] Removing peer  36:da:1c:f4:7a:6d . Expecting to remove linked annotation next.
INFO: 2017/11/29 19:21:37.338286 [kube-peers] Removing annotation  36:da:1c:f4:7a:6d
DEBU: 2017/11/29 19:21:37.537810 [kube-peers] Finished removal of  36:da:1c:f4:7a:6d
DEBU: 2017/11/29 19:21:37.537862 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:37.537876 [kube-peers] Noting I plan to remove  72:a8:c3:3f:68:0d
DEBU: 2017/11/29 19:21:37.736414 [kube-peers] Nodes that have disappeared: map[ip-172-20-34-56.eu-central-1.compute.internal:{72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}]
DEBU: 2017/11/29 19:21:37.736434 [kube-peers] Preparing to remove disappeared peer {72:a8:c3:3f:68:0d ip-172-20-34-56.eu-central-1.compute.internal}
DEBU: 2017/11/29 19:21:37.736440 [kube-peers] Existing annotation e6:80:6f:a0:11:6f
DEBU: 2017/11/29 19:21:37.736447 weave DELETE to http://127.0.0.1:6784/peer/72:a8:c3:3f:68:0d with map[]
INFO: 2017/11/29 19:21:37.976496 [kube-peers] rmpeer of 72:a8:c3:3f:68:0d : 65536 IPs taken over from 72:a8:c3:3f:68:0d

2017/11/29 19:21:38 Optimistic locking conflict: trying again: Operation cannot be fulfilled on configmaps "weave-net": the object has been modified; please apply your changes to the latest version and try again

[bboreham]

Fixed by #3192