Restarting kubelet forgets CNI state and reuses pod IPs

[groner]

I have a cluster built with kube-aws-v0.8.3 and hyperkube-v1.4.5_coreos.0.

While trying to shake out an EBS volume mount problem on one of the minions, I restarted kubelet using systemctl restart kubelet on that host. After the restart, it seems active pod IPs are getting reassigned.

core@ip-10-0-48-65 ~ $ journalctl | grep Determined.pod.ip 
Nov 07 22:31:20 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: I1107 22:31:20.320259    1706 docker_manager.go:2162] Determined pod ip after infra change: "kubernetes-dashboard-v1.4.1-wbo4r_kube-system(e6f3c346-a539-11e6-b0a2-06cd71806322)": "10.2.37.2"
Nov 07 22:31:20 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: I1107 22:31:20.549372    1706 docker_manager.go:2162] Determined pod ip after infra change: "kube-dns-v20-83ltq_kube-system(e6f4151f-a539-11e6-b0a2-06cd71806322)": "10.2.37.3"
Nov 07 22:31:20 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: I1107 22:31:20.608372    1706 docker_manager.go:2162] Determined pod ip after infra change: "heapster-v1.2.0-3646253287-1nj4n_kube-system(e6f4a391-a539-11e6-b0a2-06cd71806322)": "10.2.37.4"
Nov 07 22:31:36 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: I1107 22:31:36.128090    1706 docker_manager.go:2162] Determined pod ip after infra change: "heapster-v1.2.0-4088228293-q7nnq_kube-system(f0f4afa3-a539-11e6-b0a2-06cd71806322)": "10.2.37.5"
Nov 08 16:25:09 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: I1108 16:25:09.883226    1706 docker_manager.go:2162] Determined pod ip after infra change: "sterling-etcd-txvpf_default(cca2979b-a5cf-11e6-b0a2-06cd71806322)": "10.2.37.6"
Nov 08 16:28:54 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: I1108 16:28:54.127206    1706 docker_manager.go:2162] Determined pod ip after infra change: "sterling-etcd-2180225719-92xfx_default(6639c26d-a5d0-11e6-b0a2-06cd71806322)": "10.2.37.7"
[kubelet restart happens here]
Nov 08 20:43:38 ip-10-0-48-65.ec2.internal kubelet-wrapper[10530]: I1108 20:43:38.716862   10530 docker_manager.go:2162] Determined pod ip after infra change: "sterling-etcd-2180225719-g6a5e_default(3c3850ff-a5ee-11e6-b0a2-06cd71806322)": "10.2.37.2"
Nov 08 21:03:54 ip-10-0-48-65.ec2.internal kubelet-wrapper[10530]: I1108 21:03:54.104308   10530 docker_manager.go:2162] Determined pod ip after infra change: "haproxy-init-3189418104-vd5cn_default(dafe6a15-a5f6-11e6-b0a2-06cd71806322)": "10.2.37.3"
Nov 08 21:05:21 ip-10-0-48-65.ec2.internal kubelet-wrapper[10530]: I1108 21:05:21.789693   10530 docker_manager.go:2162] Determined pod ip after infra change: "bash_default(0f4309fa-a5f7-11e6-b0a2-06cd71806322)": "10.2.37.4"
Nov 08 21:06:29 ip-10-0-48-65.ec2.internal kubelet-wrapper[10530]: I1108 21:06:29.234689   10530 docker_manager.go:2162] Determined pod ip after infra change: "haproxy-init_default(3778d187-a5f7-11e6-b0a2-06cd71806322)": "10.2.37.5"
Nov 08 21:11:11 ip-10-0-48-65.ec2.internal kubelet-wrapper[10530]: I1108 21:11:11.348714   10530 docker_manager.go:2162] Determined pod ip after infra change: "haproxy-init_default(dfa25588-a5f7-11e6-b0a2-06cd71806322)": "10.2.37.6"
Nov 08 21:58:29 ip-10-0-48-65.ec2.internal kubelet-wrapper[10530]: I1108 21:58:29.094155   10530 docker_manager.go:2162] Determined pod ip after infra change: "sterling-etcd-2180225719-sj3a3_default(7a7ea798-a5fe-11e6-b0a2-06cd71806322)": "10.2.37.7"

This message from the time o the restart seems to explain the problem:

Nov 07 22:31:20 ip-10-0-48-65.ec2.internal kubelet-wrapper[1706]: 2016/11/07 22:31:20 Error retriving last reserved ip: Failed to retrieve last reserved ip: open /var/lib/cni/networks/podnet/last_reserved_ip: no such file or directory

[groner]

Is adding --volume cni,kind=host,source=/var/lib/cni --mount volume=cni,target=/var/lib/cni to RKT_OPTS the right solution?

[aaronlevy]

Hi @groner, the kube-aws project has moved to its own top-level repo (https://github.com/coreos/kube-aws) - can you re-open this issue there?

side-note -- I'm not sure that podnet has been tested at all (we use flannel/calico in this repo).