Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Pods get stuck in ContainerCreating #2822

Closed
rlnrln opened this issue Feb 28, 2017 · 7 comments
Closed

Pods get stuck in ContainerCreating #2822

rlnrln opened this issue Feb 28, 2017 · 7 comments
Labels
resolution/duplicate
Milestone
n/a

Comments

@rlnrln
Copy link

rlnrln commented Feb 28, 2017

Kubernetes 1.5.2, created with Kops 1.5.0 alpha 4.

Complete logs from recently rebooted node.

kubelet logs

Feb 28 14:21:37 ip-10-128-171-159 systemd[1]: Starting Kubernetes Kubelet Server...
Feb 28 14:21:37 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:21:37 ip-10-128-171-159 kubelet[735]: Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
Feb 28 14:21:37 ip-10-128-171-159 kubelet[735]: Flag --babysit-daemons has been deprecated, Will be removed in a future version.
Feb 28 14:21:37 ip-10-128-171-159 kubelet[735]: Flag --config has been deprecated, Use --pod-manifest-path instead. Will be removed in a future version.
Feb 28 14:21:37 ip-10-128-171-159 kubelet[735]: I0228 14:21:37.924451     735 feature_gate.go:181] feature gates: map[]
Feb 28 14:21:37 ip-10-128-171-159 kubelet[735]: I0228 14:21:37.927125     735 aws.go:746] Building AWS cloudprovider
Feb 28 14:21:37 ip-10-128-171-159 kubelet[735]: I0228 14:21:37.927150     735 aws.go:700] Zone not specified in configuration file; querying AWS metadata service
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.159248     735 aws.go:832] AWS cloud filtering on tags: map[KubernetesCluster:euw1.example.com]
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.159315     735 server.go:369] Successfully initialized cloud provider: "aws" from the config file: ""
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.684024     735 docker.go:356] Connecting to docker on unix:///var/run/docker.sock
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.684049     735 docker.go:376] Start docker client with request timeout=2m0s
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.698264     735 iptables.go:176] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.699794     735 iptables.go:176] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.699844     735 server.go:511] cloud provider determined current node name to be ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.699954     735 manager.go:143] cAdvisor running in container: "/"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.709363     735 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.716941     735 fs.go:117] Filesystem partitions: map[/dev/xvda1:{mountpoint:/var/lib/docker/devicemapper major:202 minor:1 fsType:ext4 blockSize:0}]
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.719796     735 manager.go:198] Machine: {NumCores:2 CpuFrequency:2400088 MemoryCapacity:8374382592 MachineID:e07fdb2c1a294e30a1ddb5369eec94db SystemUUID:EC217A27-A415-88F9-23A6-6882B9628251 BootID:3d9ba94c-d746-4685-ad6f-3846be3bd07e Filesystems:[{Device:/dev/xvda1 Capacity:1006420578304 Type:vfs Inodes:262144000 HasInodes:true}] DiskMap:map[254:0:{Name:dm-0 Major:254 Minor:0 Size:107374182400 Scheduler:none} 254:1:{Name:dm-1 Major:254 Minor:1 Size:10737418240 Scheduler:none} 202:0:{Name:xvda Major:202 Minor:0 Size:1073741824000 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:02:56:18:ae:82:db Speed:10000 Mtu:9001}] Topology:[{Id:0 Memory:8374382592 Cores:[{Id:0 Threads:[0 1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:31457280 Type:Unified Level:3}]}] CloudProvider:AWS InstanceType:m4.large InstanceID:i-0edd16e05d5730212}
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.720252     735 manager.go:204] Version: {KernelVersion:4.4.41-k8s ContainerOsVersion:Debian GNU/Linux 8 (jessie) DockerVersion:1.12.3 CadvisorVersion: CadvisorRevision:}
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.722548     735 server.go:511] cloud provider determined current node name to be ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.722762     735 server.go:700] Using root directory: /var/lib/kubelet
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.724188     735 kubelet.go:307] cloud provider determined current node name to be ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.724272     735 kubelet.go:242] Adding manifest file: /etc/kubernetes/manifests
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.724313     735 file.go:47] Watching path "/etc/kubernetes/manifests"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.724327     735 kubelet.go:252] Watching apiserver
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.726498     735 iptables.go:176] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.726526     735 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.726546     735 kubelet.go:477] Hairpin mode set to "hairpin-veth"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.729720     735 plugins.go:181] Loaded network plugin "cni"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.737999     735 docker_manager.go:257] Setting dockerRoot to /var/lib/docker
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.738014     735 docker_manager.go:260] Setting cgroupDriver to cgroupfs
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.741999     735 plugins.go:344] Loaded volume plugin "kubernetes.io/aws-ebs"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.742326     735 plugins.go:344] Loaded volume plugin "kubernetes.io/empty-dir"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.742631     735 plugins.go:344] Loaded volume plugin "kubernetes.io/gce-pd"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.742931     735 plugins.go:344] Loaded volume plugin "kubernetes.io/git-repo"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743229     735 plugins.go:344] Loaded volume plugin "kubernetes.io/host-path"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743527     735 plugins.go:344] Loaded volume plugin "kubernetes.io/nfs"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743850     735 plugins.go:344] Loaded volume plugin "kubernetes.io/secret"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743872     735 plugins.go:344] Loaded volume plugin "kubernetes.io/iscsi"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743891     735 plugins.go:344] Loaded volume plugin "kubernetes.io/glusterfs"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743904     735 plugins.go:344] Loaded volume plugin "kubernetes.io/rbd"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743918     735 plugins.go:344] Loaded volume plugin "kubernetes.io/cinder"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743931     735 plugins.go:344] Loaded volume plugin "kubernetes.io/quobyte"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743945     735 plugins.go:344] Loaded volume plugin "kubernetes.io/cephfs"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743967     735 plugins.go:344] Loaded volume plugin "kubernetes.io/downward-api"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743980     735 plugins.go:344] Loaded volume plugin "kubernetes.io/fc"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.743993     735 plugins.go:344] Loaded volume plugin "kubernetes.io/flocker"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.744006     735 plugins.go:344] Loaded volume plugin "kubernetes.io/azure-file"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.744020     735 plugins.go:344] Loaded volume plugin "kubernetes.io/configmap"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.744033     735 plugins.go:344] Loaded volume plugin "kubernetes.io/vsphere-volume"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.744048     735 plugins.go:344] Loaded volume plugin "kubernetes.io/azure-disk"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.744061     735 plugins.go:344] Loaded volume plugin "kubernetes.io/photon-pd"
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.747630     735 server.go:770] Started kubelet v1.5.2
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.749467     735 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.749488     735 kubelet_node_status.go:246] Adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.large
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.749498     735 kubelet_node_status.go:257] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=eu-west-1b
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.749506     735 kubelet_node_status.go:261] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=eu-west-1
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: E0228 14:21:38.749852     735 kubelet.go:1145] Image garbage collection failed: unable to find data for container /
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.750262     735 server.go:140] Starting to listen read-only on 0.0.0.0:10255
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.753808     735 server.go:123] Starting to listen on 0.0.0.0:10250
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.756344     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: E0228 14:21:38.763239     735 kubelet.go:1634] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: E0228 14:21:38.763581     735 kubelet.go:1642] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.763911     735 kubelet_node_status.go:358] Recording NodeHasSufficientDisk event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.764231     735 kubelet_node_status.go:358] Recording NodeHasSufficientMemory event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.764550     735 kubelet_node_status.go:358] Recording NodeHasNoDiskPressure event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.765355     735 container_manager_linux.go:335] Updating kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.765767     735 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.765810     735 status_manager.go:129] Starting to sync pod status with apiserver
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.765853     735 kubelet.go:1714] Starting kubelet main sync loop.
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.765877     735 kubelet.go:1725] skipping pod synchronization - [container runtime is down]
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.766160     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 446
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.766170     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 446
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.766176     735 container_manager_linux.go:434] Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.766233     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 735
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.766241     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 735
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.766269     735 volume_manager.go:240] The desired_state_of_world populator starts
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.766275     735 volume_manager.go:242] Starting Kubelet Volume Manager
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: E0228 14:21:38.780392     735 factory.go:291] devicemapper filesystem stats will not be reported: unable to find thin_ls binary
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.780410     735 factory.go:295] Registering Docker factory
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: W0228 14:21:38.780422     735 manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.780429     735 factory.go:54] Registering systemd factory
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.780637     735 factory.go:86] Registering Raw factory
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.781303     735 manager.go:1106] Started watching for new ooms in manager
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.784969     735 oomparser.go:185] oomparser using systemd
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.785502     735 manager.go:288] Starting recovery of all containers
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.833347     735 manager.go:293] Recovery completed
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: E0228 14:21:38.840346     735 eviction_manager.go:204] eviction manager: unexpected err: failed GetNode: node 'ip-10-128-171-159.eu-west-1.compute.internal' not found
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.866612     735 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.866638     735 kubelet_node_status.go:246] Adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.large
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.866649     735 kubelet_node_status.go:257] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=eu-west-1b
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.866657     735 kubelet_node_status.go:261] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=eu-west-1
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.868150     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.871762     735 kubelet_node_status.go:358] Recording NodeHasSufficientDisk event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.871796     735 kubelet_node_status.go:358] Recording NodeHasSufficientMemory event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.871811     735 kubelet_node_status.go:358] Recording NodeHasNoDiskPressure event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.871860     735 kubelet_node_status.go:74] Attempting to register node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.883377     735 kubelet_node_status.go:113] Node ip-10-128-171-159.eu-west-1.compute.internal was previously registered
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.883395     735 kubelet_node_status.go:77] Successfully registered node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.884691     735 kubelet_network.go:226] Setting Pod CIDR:  -> 100.96.10.0/24
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.886610     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.887503     735 kubelet_node_status.go:358] Recording NodeHasSufficientDisk event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:38 ip-10-128-171-159 kubelet[735]: I0228 14:21:38.887544     735 kubelet_node_status.go:358] Recording NodeReady event message for node ip-10-128-171-159.eu-west-1.compute.internal
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.630695     735 kubelet.go:1781] SyncLoop (ADD, "file"): "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)"
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.630990     735 kubelet.go:1781] SyncLoop (ADD, "api"): "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f), kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f), m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f), prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f), weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9), roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f), kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(a3afb19f-faab-11e6-9a9d-027b34db1ad5), m1-stage-2548681241-fhwq1_default(41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f), roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f), node-exporter-60n35_default(a0fe2ad1-faab-11e6-a97a-0668c40b08b9), kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f), kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)"
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631421     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-fhwq1_default(41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"2ec9f211db30676478f8622f7ea0c67f30c06da3b4383a629756873795c82736"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631473     735 kubelet.go:1816] SyncLoop (PLEG): "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55316f7f-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"40e1b3a1335d3819148293849e5d19737ac1caa8d2d8366ce8ef40119732875c"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631500     735 kubelet.go:1816] SyncLoop (PLEG): "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55316f7f-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"0e8acd798c03802adf60d7362d9e3b4445e6b14586973d794294355cd048f248"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: W0228 14:21:44.631525     735 pod_container_deletor.go:77] Container "0e8acd798c03802adf60d7362d9e3b4445e6b14586973d794294355cd048f248" not found in pod's containers
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631541     735 kubelet.go:1816] SyncLoop (PLEG): "roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d25d70c7-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"58aba2b31ace7a30c6051bf891ab2136e413df498687b25c4d93ca4920655751"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631571     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55f01a53-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"693ddc182256a31d1c064d400cdc0a29fbbc24d1f1c2347c48f17a25bdfdb65e"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631598     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55f01a53-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"72f7dfd1ca529119805e7859b8f6dc0d33d95dee776c9e1886e5071793f78662"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: W0228 14:21:44.631626     735 pod_container_deletor.go:77] Container "72f7dfd1ca529119805e7859b8f6dc0d33d95dee776c9e1886e5071793f78662" not found in pod's containers
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631642     735 kubelet.go:1816] SyncLoop (PLEG): "node-exporter-60n35_default(a0fe2ad1-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe2ad1-faab-11e6-a97a-0668c40b08b9", Type:"ContainerDied", Data:"1fb6b836d4defd775a14aa9dbee8925734208ac6b900be78ab319dddd2da3175"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631668     735 kubelet.go:1816] SyncLoop (PLEG): "node-exporter-60n35_default(a0fe2ad1-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe2ad1-faab-11e6-a97a-0668c40b08b9", Type:"ContainerDied", Data:"29a26206b542bf6e59f8b65b68618a4c1fcc6780989c053cc778ab0145c78629"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631693     735 kubelet.go:1816] SyncLoop (PLEG): "roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"def2ee23-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"787dc16e98e1d64879c2d06a57d5b348c34626a08d4f290c63b2cbdb04dc786a"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631719     735 kubelet.go:1816] SyncLoop (PLEG): "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"cd900d3ac87403022ac7585baac3891090e4b6b2c6a35655468509d2fd5231db"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631750     735 kubelet.go:1816] SyncLoop (PLEG): "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)", event: &pleg.PodLifecycleEvent{ID:"9e952864c1e39b812975e6f2f20d8bd3", Type:"ContainerDied", Data:"36a7d3eefa07df0356db1928f4d61c325025a875bae6844b6672cb9667cfe50a"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631780     735 kubelet.go:1816] SyncLoop (PLEG): "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)", event: &pleg.PodLifecycleEvent{ID:"9e952864c1e39b812975e6f2f20d8bd3", Type:"ContainerDied", Data:"317fcf1fe4ffda494d551394ee9e887ad311dbbb84e3693ba6d4f795d4a90f42"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631807     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerDied", Data:"fa60478cada169201e08540ab4fa8155e5359d2ad57d9a4ccaec51d0c339287d"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631839     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerDied", Data:"c801ca2d2c812a42525e8ee649cc995b3714d2d09bb2ed3d83b542121eb93f2a"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631866     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerDied", Data:"396df8b977abbe3acf81491bc3b94332441be16da72d1a1c99ca85f717d61d01"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: W0228 14:21:44.631890     735 pod_container_deletor.go:77] Container "396df8b977abbe3acf81491bc3b94332441be16da72d1a1c99ca85f717d61d01" not found in pod's containers
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.631905     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerDied", Data:"6c20216752f63240d09d367e1ade270e283d42b345ca5f1ef17098d9c4783b08"}
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: E0228 14:21:44.647215     735 kubelet.go:1508] Failed creating a mirror pod for "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)": pods "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal" already exists
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751085     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f" (UID: "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751129     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-conf" (spec.Name: "cni-conf") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751152     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751179     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/55316f7f-fdb4-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "55316f7f-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55316f7f-fdb4-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751393     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/d25d70c7-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751420     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-ssl-certs-host" (spec.Name: "ssl-certs-host") pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751456     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/empty-dir/a0fe5e51-faab-11e6-a97a-0668c40b08b9-weavedb" (spec.Name: "weavedb") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751479     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-bin" (spec.Name: "cni-bin") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751501     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-bin2" (spec.Name: "cni-bin2") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751528     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751553     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/def2ee23-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "def2ee23-fdc0-11e6-bc48-02ecf0186f8f" (UID: "def2ee23-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751685     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-varlog" (spec.Name: "varlog") pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751710     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751735     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/deaee4aa-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f" (UID: "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751764     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-kubeconfig" (spec.Name: "kubeconfig") pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751788     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751824     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/55f01a53-fdb4-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "55f01a53-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55f01a53-fdb4-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.751919     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/d21a03b2-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f")
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852144     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-varlog" (spec.Name: "varlog") to pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852214     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") to pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852686     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-varlog" (spec.Name: "varlog") pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852817     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/deaee4aa-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f" (UID: "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852866     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-kubeconfig" (spec.Name: "kubeconfig") to pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852920     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") to pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.852977     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/55f01a53-fdb4-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") to pod "55f01a53-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55f01a53-fdb4-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853023     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/d21a03b2-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") to pod "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853143     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f" (UID: "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853183     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-conf" (spec.Name: "cni-conf") to pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853246     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") to pod "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853360     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/def2ee23-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "def2ee23-fdc0-11e6-bc48-02ecf0186f8f" (UID: "def2ee23-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853420     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/55316f7f-fdb4-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "55316f7f-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55316f7f-fdb4-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853473     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/d25d70c7-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853526     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-ssl-certs-host" (spec.Name: "ssl-certs-host") to pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853637     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/empty-dir/a0fe5e51-faab-11e6-a97a-0668c40b08b9-weavedb" (spec.Name: "weavedb") to pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853681     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-bin" (spec.Name: "cni-bin") to pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853718     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-bin2" (spec.Name: "cni-bin2") to pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.853792     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.854307     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-kubeconfig" (spec.Name: "kubeconfig") pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.863525     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-conf" (spec.Name: "cni-conf") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.869819     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/empty-dir/a0fe5e51-faab-11e6-a97a-0668c40b08b9-weavedb" (spec.Name: "weavedb") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.870635     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-bin" (spec.Name: "cni-bin") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.871063     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/9e952864c1e39b812975e6f2f20d8bd3-ssl-certs-host" (spec.Name: "ssl-certs-host") pod "9e952864c1e39b812975e6f2f20d8bd3" (UID: "9e952864c1e39b812975e6f2f20d8bd3").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.871126     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/a0fe5e51-faab-11e6-a97a-0668c40b08b9-cni-bin2" (spec.Name: "cni-bin2") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:44 ip-10-128-171-159 kubelet[735]: I0228 14:21:44.947458     735 docker_manager.go:1975] Need to restart pod infra container for "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)" because it is not found
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.166859     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/deaee4aa-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f" (UID: "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.236996     735 docker_manager.go:1975] Need to restart pod infra container for "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.366642     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.536994     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.566654     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.767607     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.836378     735 docker_manager.go:1975] Need to restart pod infra container for "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)" because it is not found
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.837901     735 docker_manager.go:1975] Need to restart pod infra container for "node-exporter-60n35_default(a0fe2ad1-faab-11e6-a97a-0668c40b08b9)" because it is not found
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: I0228 14:21:45.966734     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/55f01a53-fdb4-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "55f01a53-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55f01a53-fdb4-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: E0228 14:21:45.996894     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: E0228 14:21:45.996918     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:45 ip-10-128-171-159 kubelet[735]: E0228 14:21:45.996956     735 docker_manager.go:2201] Failed to setup network for pod "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.134994     735 docker_manager.go:1975] Need to restart pod infra container for "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.166993     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d21a03b2-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.187505     735 docker_manager.go:1564] Killing container "00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740 default/mailupdated-mover-prod-2178277528-sprgb" with 30 second grace period
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.366752     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/55316f7f-fdb4-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "55316f7f-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55316f7f-fdb4-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.435557     735 docker_manager.go:1975] Need to restart pod infra container for "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.436189     735 docker_manager.go:1975] Need to restart pod infra container for "kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.485392     735 docker_manager.go:1605] Container "00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740 default/mailupdated-mover-prod-2178277528-sprgb" exited after 297.85438ms
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: E0228 14:21:46.485724     735 pod_workers.go:184] Error syncing pod deaee4aa-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "mailupdated-mover-prod-2178277528-sprgb_default" with SetupNetworkError: "Failed to setup network for pod \"mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.571717     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f" (UID: "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.766795     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: E0228 14:21:46.936446     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: E0228 14:21:46.936471     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: E0228 14:21:46.936506     735 docker_manager.go:2201] Failed to setup network for pod "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:46 ip-10-128-171-159 kubelet[735]: I0228 14:21:46.966754     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d25d70c7-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.036154     735 docker_manager.go:1975] Need to restart pod infra container for "roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.037048     735 docker_manager.go:1975] Need to restart pod infra container for "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.047338     735 docker_manager.go:1564] Killing container "df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a default/m1-stage-2548681241-b1r9w" with 30 second grace period
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.166956     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/def2ee23-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "def2ee23-fdc0-11e6-bc48-02ecf0186f8f" (UID: "def2ee23-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.260715     735 kubelet.go:1816] SyncLoop (PLEG): "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)", event: &pleg.PodLifecycleEvent{ID:"9e952864c1e39b812975e6f2f20d8bd3", Type:"ContainerStarted", Data:"6c943418f56f1b3f5243635368f3460d2549ae934022360359e25770bb08e666"}
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.260783     735 kubelet.go:1816] SyncLoop (PLEG): "kube-proxy-ip-10-128-171-159.eu-west-1.compute.internal_kube-system(9e952864c1e39b812975e6f2f20d8bd3)", event: &pleg.PodLifecycleEvent{ID:"9e952864c1e39b812975e6f2f20d8bd3", Type:"ContainerStarted", Data:"d1e4ae36277c213dd5c273ca0e68345f5e12f1a7ec4c232cf034c697148ab618"}
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.334902     735 docker_manager.go:1975] Need to restart pod infra container for "roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.344555     735 provider.go:119] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.344752     735 docker.go:239] Pulling image prom/node-exporter:latest without credentials
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: I0228 14:21:47.475178     735 docker_manager.go:1605] Container "df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a default/m1-stage-2548681241-b1r9w" exited after 427.583944ms
Feb 28 14:21:47 ip-10-128-171-159 kubelet[735]: E0228 14:21:47.475258     735 pod_workers.go:184] Error syncing pod de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "m1-stage-2548681241-b1r9w_default" with SetupNetworkError: "Failed to setup network for pod \"m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.044250     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.044272     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.044308     735 docker_manager.go:2201] Failed to setup network for pod "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.191707     735 docker_manager.go:1564] Killing container "6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f kube-system/kube-dns-autoscaler-2715466192-k3k4l" with 30 second grace period
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: W0228 14:21:48.311309     735 container.go:352] Failed to create summary reader for "/docker/6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f": none of the resources are being tracked.
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.311570     735 helpers.go:101] Unable to get network stats from pid 1350: couldn't read network stats: failure opening /proc/1350/net/dev: open /proc/1350/net/dev: no such file or directory
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.314382     735 kubelet.go:1816] SyncLoop (PLEG): "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"deaee4aa-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"00dba414cb54b03d74ca3bf216568d0284b51be39c242e0d8a237b7359788740"}
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.446199     735 kube_docker_client.go:331] Stop pulling image "prom/node-exporter:latest": "Status: Image is up to date for prom/node-exporter:latest"
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.526806     735 docker_manager.go:1605] Container "6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f kube-system/kube-dns-autoscaler-2715466192-k3k4l" exited after 335.069308ms
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.526882     735 pod_workers.go:184] Error syncing pod 55f01a53-fdb4-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "kube-dns-autoscaler-2715466192-k3k4l_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.567838     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/deaee4aa-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f" (UID: "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.614916     735 docker_manager.go:1975] Need to restart pod infra container for "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.651366     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.651384     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.651436     735 docker_manager.go:2201] Failed to setup network for pod "kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.767447     735 status_manager.go:441] Removing Pod "m1-stage-2548681241-fhwq1_default(41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f)" from etcd
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.827936     735 docker_manager.go:1564] Killing container "2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c kube-system/kube-dns-782804071-qkf9l" with 30 second grace period
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.841492     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.841515     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.841551     735 docker_manager.go:2201] Failed to setup network for pod "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.856802     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.857294     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: E0228 14:21:48.857689     735 docker_manager.go:2201] Failed to setup network for pod "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.859117     735 docker_manager.go:1564] Killing container "5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c default/prometheus-1433796061-dthh8" with 30 second grace period
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.977182     735 docker_manager.go:1564] Killing container "df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f kube-system/kubernetes-dashboard-3203831700-b2f05" with 30 second grace period
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: W0228 14:21:48.977872     735 container.go:352] Failed to create summary reader for "/docker/5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c": none of the resources are being tracked.
Feb 28 14:21:48 ip-10-128-171-159 kubelet[735]: I0228 14:21:48.978104     735 helpers.go:101] Unable to get network stats from pid 1513: couldn't read network stats: failure opening /proc/1513/net/dev: open /proc/1513/net/dev: no such file or directory
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.001738     735 kubelet.go:1797] SyncLoop (DELETE, "api"): "m1-stage-2548681241-fhwq1_default(41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f)"
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.041892     735 kubelet.go:1791] SyncLoop (REMOVE, "api"): "m1-stage-2548681241-fhwq1_default(41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f)"
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.041939     735 kubelet.go:1989] Failed to delete pod "m1-stage-2548681241-fhwq1_default(41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f)", err: pod not found
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.109219     735 docker_manager.go:1605] Container "2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c kube-system/kube-dns-782804071-qkf9l" exited after 281.244161ms
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: E0228 14:21:49.109293     735 pod_workers.go:184] Error syncing pod d21a03b2-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "kube-dns-782804071-qkf9l_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.399663     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"df358e9a4ec065205c3cdb65ccc3bc1c70eb8ab96ef96d769792effa0885887a"}
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.521530     735 docker_manager.go:1605] Container "5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c default/prometheus-1433796061-dthh8" exited after 662.041607ms
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: E0228 14:21:49.521604     735 pod_workers.go:184] Error syncing pod 55316f7f-fdb4-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "prometheus-1433796061-dthh8_default" with SetupNetworkError: "Failed to setup network for pod \"prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.570286     735 docker_manager.go:1605] Container "df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f kube-system/kubernetes-dashboard-3203831700-b2f05" exited after 593.077229ms
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: E0228 14:21:49.570350     735 pod_workers.go:184] Error syncing pod d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "kubernetes-dashboard-3203831700-b2f05_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: W0228 14:21:49.665477     735 container.go:352] Failed to create summary reader for "/docker/df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f": none of the resources are being tracked.
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.671665     735 reconciler.go:189] UnmountVolume operation started for volume "kubernetes.io/secret/41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") from pod "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f" (UID: "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: E0228 14:21:49.676897     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: E0228 14:21:49.676917     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: E0228 14:21:49.676957     735 docker_manager.go:2201] Failed to setup network for pod "roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.685421     735 operation_executor.go:992] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (OuterVolumeSpecName: "default-token-750mk") pod "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f" (UID: "41a2c9ac-fdc0-11e6-bc48-02ecf0186f8f"). InnerVolumeSpecName "default-token-750mk". PluginName "kubernetes.io/secret", VolumeGidValue ""
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.700147     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.758653     735 docker_manager.go:1564] Killing container "3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c default/roapp-datamover-prod-4070366716-pw26l" with 30 second grace period
Feb 28 14:21:49 ip-10-128-171-159 kubelet[735]: I0228 14:21:49.766848     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.167488     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.204916     735 docker_manager.go:1605] Container "3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c default/roapp-datamover-prod-4070366716-pw26l" exited after 446.234016ms
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: E0228 14:21:50.204988     735 pod_workers.go:184] Error syncing pod d25d70c7-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "roapp-datamover-prod-4070366716-pw26l_default" with SetupNetworkError: "Failed to setup network for pod \"roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.488693     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d21a03b2-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c"}
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: E0228 14:21:50.538102     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: E0228 14:21:50.538122     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: E0228 14:21:50.538153     735 docker_manager.go:2201] Failed to setup network for pod "roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.637860     735 docker_manager.go:1564] Killing container "28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e default/roapp-datamover-prod-4070366716-bkm5r" with 30 second grace period
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: W0228 14:21:50.638642     735 container.go:352] Failed to create summary reader for "/docker/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c": none of the resources are being tracked.
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: E0228 14:21:50.643743     735 summary.go:70] Partial failure issuing GetContainerInfoV2: partial failures: ["/docker/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c": RecentStats: unable to find data for container /docker/3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c]
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.767238     735 status_manager.go:402] Status for pod "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f" is up-to-date; skipping
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.789893     735 docker_manager.go:1975] Need to restart pod infra container for "kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:50 ip-10-128-171-159 kubelet[735]: I0228 14:21:50.966777     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d21a03b2-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d21a03b2-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:51 ip-10-128-171-159 kubelet[735]: I0228 14:21:51.124240     735 docker_manager.go:1605] Container "28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e default/roapp-datamover-prod-4070366716-bkm5r" exited after 486.33061ms
Feb 28 14:21:51 ip-10-128-171-159 kubelet[735]: E0228 14:21:51.124323     735 pod_workers.go:184] Error syncing pod def2ee23-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "roapp-datamover-prod-4070366716-bkm5r_default" with SetupNetworkError: "Failed to setup network for pod \"roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:51 ip-10-128-171-159 kubelet[735]: W0228 14:21:51.253903     735 container.go:352] Failed to create summary reader for "/docker/28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e": none of the resources are being tracked.
Feb 28 14:21:51 ip-10-128-171-159 kubelet[735]: I0228 14:21:51.322497     735 kubelet.go:1816] SyncLoop (PLEG): "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55316f7f-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c"}
Feb 28 14:21:51 ip-10-128-171-159 kubelet[735]: I0228 14:21:51.566807     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/55316f7f-fdb4-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "55316f7f-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55316f7f-fdb4-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:51 ip-10-128-171-159 kubelet[735]: I0228 14:21:51.622654     735 docker_manager.go:1975] Need to restart pod infra container for "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: E0228 14:21:52.291201     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: E0228 14:21:52.291222     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: E0228 14:21:52.291254     735 docker_manager.go:2201] Failed to setup network for pod "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: I0228 14:21:52.481617     735 docker_manager.go:1564] Killing container "0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e default/mailupdated-mover-prod-2178277528-sprgb" with 30 second grace period
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: I0228 14:21:52.743034     735 docker_manager.go:1605] Container "0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e default/mailupdated-mover-prod-2178277528-sprgb" exited after 261.385413ms
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: E0228 14:21:52.743169     735 pod_workers.go:184] Error syncing pod deaee4aa-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "mailupdated-mover-prod-2178277528-sprgb_default" with SetupNetworkError: "Failed to setup network for pod \"mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: I0228 14:21:52.858352     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55f01a53-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"6dd4f3881fb5ea60f83bf60c9fd1b43c9f80e41c95b93057b4dc0d1b4181005f"}
Feb 28 14:21:52 ip-10-128-171-159 kubelet[735]: I0228 14:21:52.891398     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/55f01a53-fdb4-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "55f01a53-fdb4-11e6-bc48-02ecf0186f8f" (UID: "55f01a53-fdb4-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: I0228 14:21:53.002278     735 kubelet.go:1849] SyncLoop (container unhealthy): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)"
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: E0228 14:21:53.111327     735 cni.go:255] Error adding network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: E0228 14:21:53.111350     735 cni.go:209] Error while adding to cni network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: E0228 14:21:53.111447     735 docker_manager.go:2201] Failed to setup network for pod "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: I0228 14:21:53.158774     735 docker_manager.go:1975] Need to restart pod infra container for "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: I0228 14:21:53.317739     735 docker_manager.go:1564] Killing container "20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679 default/m1-stage-2548681241-b1r9w" with 30 second grace period
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: I0228 14:21:53.797200     735 docker_manager.go:1605] Container "20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679 default/m1-stage-2548681241-b1r9w" exited after 479.433959ms
Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: E0228 14:21:53.797270     735 pod_workers.go:184] Error syncing pod de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f, skipping: failed to "SetupNetwork" for "m1-stage-2548681241-b1r9w_default" with SetupNetworkError: "Failed to setup network for pod \"m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)\" using network plugins \"cni\": unable to allocate IP address: Post http://127.0.0.1:6784/ip/20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod"
Feb 28 14:21:54 ip-10-128-171-159 kubelet[735]: W0228 14:21:54.273489     735 container.go:352] Failed to create summary reader for "/docker/20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679": none of the resources are being tracked.
Feb 28 14:21:54 ip-10-128-171-159 kubelet[735]: I0228 14:21:54.703468     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:54 ip-10-128-171-159 kubelet[735]: I0228 14:21:54.846537     735 kubelet.go:1816] SyncLoop (PLEG): "node-exporter-60n35_default(a0fe2ad1-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe2ad1-faab-11e6-a97a-0668c40b08b9", Type:"ContainerStarted", Data:"bb7207310417327924d7ed1bcacd555c4bf4cb743ab1731a24826ad1a83d265f"}
Feb 28 14:21:54 ip-10-128-171-159 kubelet[735]: I0228 14:21:54.978932     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerStarted", Data:"d9e3131854742b01990ceea6a2b26337b043638b29092e54656af4938ed3ec15"}
Feb 28 14:21:54 ip-10-128-171-159 kubelet[735]: I0228 14:21:54.997301     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:55 ip-10-128-171-159 kubelet[735]: I0228 14:21:55.027273     735 kubelet.go:1816] SyncLoop (PLEG): "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f"}
Feb 28 14:21:55 ip-10-128-171-159 kubelet[735]: I0228 14:21:55.095511     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:55 ip-10-128-171-159 kubelet[735]: I0228 14:21:55.327429     735 docker_manager.go:1975] Need to restart pod infra container for "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: E0228 14:21:56.169109     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'kubernetes-dashboard-3203831700-b2f05' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: I0228 14:21:56.172240     735 kubelet.go:1816] SyncLoop (PLEG): "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"df2a17902ebdab4cc0def54b90489c714d0fe5e2ca2c5c3be7ac6343de64bd4f"}
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: I0228 14:21:56.172302     735 kubelet.go:1816] SyncLoop (PLEG): "kubernetes-dashboard-3203831700-b2f05_kube-system(d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d10f4f4e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"f838e4aec2cf8302d5f47d4b5f75bc6530f93409922bf227c7fbd78329b45b41"}
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: I0228 14:21:56.695941     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679"}
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: I0228 14:21:56.698814     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f" (UID: "de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:56 ip-10-128-171-159 kubelet[735]: I0228 14:21:56.996543     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.225709     735 kubelet.go:1816] SyncLoop (PLEG): "node-exporter-60n35_default(a0fe2ad1-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe2ad1-faab-11e6-a97a-0668c40b08b9", Type:"ContainerStarted", Data:"2f0856a865616468beed5302c3bf0657de3d6f8171d3e74bd13045f2eb509b11"}
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.299920     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.326506     735 kubelet.go:1816] SyncLoop (PLEG): "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"deaee4aa-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"2eed3a74c3ec65593ea96a16d912743c4da49daf87966f3727c322f174a96f32"}
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: W0228 14:21:57.326550     735 pod_container_deletor.go:77] Container "2eed3a74c3ec65593ea96a16d912743c4da49daf87966f3727c322f174a96f32" not found in pod's containers
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.326573     735 kubelet.go:1816] SyncLoop (PLEG): "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"deaee4aa-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"0c5389901f2cac1acf619752708f5a33a8eadb94183aee36fbd4fcea1e22660e"}
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.400395     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/deaee4aa-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f" (UID: "deaee4aa-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.627150     735 docker_manager.go:1975] Need to restart pod infra container for "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:57 ip-10-128-171-159 kubelet[735]: I0228 14:21:57.922539     735 kubelet.go:1816] SyncLoop (PLEG): "roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"def2ee23-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"28d43bd0bc62c2043819662c6bed43737d8ea20bf3b1ab3c804e73760603474e"}
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: I0228 14:21:58.002024     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/def2ee23-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "def2ee23-fdc0-11e6-bc48-02ecf0186f8f" (UID: "def2ee23-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: I0228 14:21:58.222668     735 docker_manager.go:1975] Need to restart pod infra container for "roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: I0228 14:21:58.769230     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerStarted", Data:"2c2fc3703a9bd0478c1e7ad86323f36c5ebeb28263704a0e99e2cbdbc64e317a"}
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: I0228 14:21:58.769280     735 kubelet.go:1816] SyncLoop (PLEG): "weave-net-jxlmt_kube-system(a0fe5e51-faab-11e6-a97a-0668c40b08b9)", event: &pleg.PodLifecycleEvent{ID:"a0fe5e51-faab-11e6-a97a-0668c40b08b9", Type:"ContainerStarted", Data:"8d24bdb4d48f2a60dcdb4e14a044067fdbde1d39b364e62dbd1c4352d217bd33"}
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: I0228 14:21:58.803878     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: E0228 14:21:58.895540     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'kube-dns-782804071-qkf9l' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:21:58 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:21:59 ip-10-128-171-159 kubelet[735]: I0228 14:21:59.019339     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d21a03b2-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"2af2d20287a8f348494f1f9317e27f396de4b509f2d9d16665913c6038da412c"}
Feb 28 14:21:59 ip-10-128-171-159 kubelet[735]: I0228 14:21:59.019407     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-782804071-qkf9l_kube-system(d21a03b2-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d21a03b2-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"226a9a296e3df946461e16ef6edb222499156dffefb7154224b5df6bf7fba857"}
Feb 28 14:21:59 ip-10-128-171-159 kubelet[735]: E0228 14:21:59.584110     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'prometheus-1433796061-dthh8' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:21:59 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:21:59 ip-10-128-171-159 kubelet[735]: I0228 14:21:59.587377     735 kubelet.go:1816] SyncLoop (PLEG): "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55316f7f-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"5c015992ca437f203346a90788dfaa4139967908ddb5af72c4d7856029bc061c"}
Feb 28 14:21:59 ip-10-128-171-159 kubelet[735]: I0228 14:21:59.587434     735 kubelet.go:1816] SyncLoop (PLEG): "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55316f7f-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"a98b105caac5e019f17b55178ca2f62e9e36b758ee56763512337f99a5cc5e06"}
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: I0228 14:22:00.072892     735 kubelet.go:1816] SyncLoop (PLEG): "roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d25d70c7-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerDied", Data:"3445772bfa7382334f319d7e936cb830f443ad932227fe6f5cd8c9ea4bb6b83c"}
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: E0228 14:22:00.085264     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'kube-dns-autoscaler-2715466192-k3k4l' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: I0228 14:22:00.107175     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d25d70c7-fdc0-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f" (UID: "d25d70c7-fdc0-11e6-bc48-02ecf0186f8f").
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: I0228 14:22:00.373334     735 docker_manager.go:1975] Need to restart pod infra container for "roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: I0228 14:22:00.545663     735 kubelet.go:1816] SyncLoop (PLEG): "kube-dns-autoscaler-2715466192-k3k4l_kube-system(55f01a53-fdb4-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"55f01a53-fdb4-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"8b8c81718306462ae82b4fd4e65c68159b9fd47d76970f1e296531b7734a042e"}
Feb 28 14:22:00 ip-10-128-171-159 kubelet[735]: I0228 14:22:00.588378     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.560082     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'kube-dns-782804071-qkf9l' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.578515     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'prometheus-1433796061-dthh8' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.590491     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'roapp-datamover-prod-4070366716-pw26l' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: I0228 14:22:01.592388     735 kubelet.go:1816] SyncLoop (PLEG): "roapp-datamover-prod-4070366716-pw26l_default(d25d70c7-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d25d70c7-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"a78aecece7591047e466e3e454459022d651a9ec6827ab010bf9def6bdc51f2c"}
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.602493     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'roapp-datamover-prod-4070366716-bkm5r' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: I0228 14:22:01.604470     735 kubelet.go:1816] SyncLoop (PLEG): "roapp-datamover-prod-4070366716-bkm5r_default(def2ee23-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"def2ee23-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"6c02ecf9e637482455e8530f1b316525b365b7fa7b4c6f62a3738ac383fb3139"}
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.614283     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'kubernetes-dashboard-3203831700-b2f05' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.625661     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'mailupdated-mover-prod-2178277528-sprgb' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: I0228 14:22:01.627461     735 kubelet.go:1816] SyncLoop (PLEG): "mailupdated-mover-prod-2178277528-sprgb_default(deaee4aa-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"deaee4aa-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"ebc1d7c158b2cc5f719063470c239e79c52feeb721645c75438795a18a276719"}
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: E0228 14:22:01.638269     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'm1-stage-2548681241-b1r9w' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: I0228 14:22:01.639918     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"c5b81eb261e2c77f74c48998fafd5e0a2141a636aa377a413d6f93d83c3d3c6a"}
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: I0228 14:22:01.711214     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:22:01 ip-10-128-171-159 kubelet[735]: I0228 14:22:01.711795     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:22:10 ip-10-128-171-159 kubelet[735]: I0228 14:22:10.616848     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:22:20 ip-10-128-171-159 kubelet[735]: I0228 14:22:20.636048     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:22:30 ip-10-128-171-159 kubelet[735]: I0228 14:22:30.658093     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:22:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:22:40 ip-10-128-171-159 kubelet[735]: I0228 14:22:40.675698     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:22:45 ip-10-128-171-159 kubelet[735]: I0228 14:22:45.168031     735 kubelet.go:1797] SyncLoop (DELETE, "api"): "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)"
Feb 28 14:22:50 ip-10-128-171-159 kubelet[735]: I0228 14:22:50.694129     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:23:00 ip-10-128-171-159 kubelet[735]: I0228 14:23:00.710220     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:23:03 ip-10-128-171-159 kubelet[735]: I0228 14:23:03.225605     735 kubelet.go:1781] SyncLoop (ADD, "api"): "m1-stage-2548681241-72677_default(6a174332-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:23:03 ip-10-128-171-159 kubelet[735]: I0228 14:23:03.341290     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/6a174332-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "6a174332-fdc1-11e6-bc48-02ecf0186f8f" (UID: "6a174332-fdc1-11e6-bc48-02ecf0186f8f")
Feb 28 14:23:03 ip-10-128-171-159 kubelet[735]: I0228 14:23:03.441596     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/6a174332-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "6a174332-fdc1-11e6-bc48-02ecf0186f8f" (UID: "6a174332-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:23:03 ip-10-128-171-159 kubelet[735]: I0228 14:23:03.446371     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/6a174332-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "6a174332-fdc1-11e6-bc48-02ecf0186f8f" (UID: "6a174332-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:23:03 ip-10-128-171-159 kubelet[735]: I0228 14:23:03.526228     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-2548681241-72677_default(6a174332-fdc1-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:23:04 ip-10-128-171-159 kubelet[735]: E0228 14:23:04.003009     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'm1-stage-2548681241-72677' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:23:04 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:23:04 ip-10-128-171-159 kubelet[735]: I0228 14:23:04.003168     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-72677_default(6a174332-fdc1-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"6a174332-fdc1-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"5b382575f4041e33262bbee01a8762648339356360c6ec5896dbcc78d672ab60"}
Feb 28 14:23:10 ip-10-128-171-159 kubelet[735]: I0228 14:23:10.728993     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:23:19 ip-10-128-171-159 kubelet[735]: I0228 14:23:19.674893     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:23:20 ip-10-128-171-159 kubelet[735]: I0228 14:23:20.749596     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:23:29 ip-10-128-171-159 kubelet[735]: I0228 14:23:29.694675     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:23:30 ip-10-128-171-159 kubelet[735]: I0228 14:23:30.769491     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:23:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:23:40 ip-10-128-171-159 kubelet[735]: I0228 14:23:40.792871     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:23:50 ip-10-128-171-159 kubelet[735]: I0228 14:23:50.824028     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:24:00 ip-10-128-171-159 kubelet[735]: I0228 14:24:00.847670     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:24:10 ip-10-128-171-159 kubelet[735]: I0228 14:24:10.869716     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:24:20 ip-10-128-171-159 kubelet[735]: I0228 14:24:20.891645     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:24:30 ip-10-128-171-159 kubelet[735]: I0228 14:24:30.911255     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:24:36 ip-10-128-171-159 kubelet[735]: I0228 14:24:36.738918     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:24:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:24:40 ip-10-128-171-159 kubelet[735]: I0228 14:24:40.942128     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:24:43 ip-10-128-171-159 kubelet[735]: I0228 14:24:43.646812     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:24:50 ip-10-128-171-159 kubelet[735]: I0228 14:24:50.989815     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:01 ip-10-128-171-159 kubelet[735]: I0228 14:25:01.017320     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:11 ip-10-128-171-159 kubelet[735]: I0228 14:25:11.038416     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:21 ip-10-128-171-159 kubelet[735]: I0228 14:25:21.063303     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:31 ip-10-128-171-159 kubelet[735]: I0228 14:25:31.083330     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:25:39 ip-10-128-171-159 kubelet[735]: I0228 14:25:39.664979     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:25:41 ip-10-128-171-159 kubelet[735]: I0228 14:25:41.108385     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:44 ip-10-128-171-159 kubelet[735]: I0228 14:25:44.674897     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:25:50 ip-10-128-171-159 kubelet[735]: I0228 14:25:50.724301     735 kubelet.go:1797] SyncLoop (DELETE, "api"): "prometheus-1433796061-dthh8_default(55316f7f-fdb4-11e6-bc48-02ecf0186f8f)"
Feb 28 14:25:51 ip-10-128-171-159 kubelet[735]: I0228 14:25:51.124735     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:25:59 ip-10-128-171-159 kubelet[735]: I0228 14:25:59.792812     735 kubelet.go:1781] SyncLoop (ADD, "api"): "prometheus-1433796061-59q8b_default(d348b757-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:25:59 ip-10-128-171-159 kubelet[735]: I0228 14:25:59.904363     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/d348b757-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "d348b757-fdc1-11e6-bc48-02ecf0186f8f" (UID: "d348b757-fdc1-11e6-bc48-02ecf0186f8f")
Feb 28 14:26:00 ip-10-128-171-159 kubelet[735]: I0228 14:26:00.004655     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/d348b757-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "d348b757-fdc1-11e6-bc48-02ecf0186f8f" (UID: "d348b757-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:00 ip-10-128-171-159 kubelet[735]: I0228 14:26:00.009829     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d348b757-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "d348b757-fdc1-11e6-bc48-02ecf0186f8f" (UID: "d348b757-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:00 ip-10-128-171-159 kubelet[735]: I0228 14:26:00.093529     735 docker_manager.go:1975] Need to restart pod infra container for "prometheus-1433796061-59q8b_default(d348b757-fdc1-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:26:00 ip-10-128-171-159 kubelet[735]: E0228 14:26:00.671240     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'prometheus-1433796061-59q8b' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:26:00 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:26:00 ip-10-128-171-159 kubelet[735]: I0228 14:26:00.671411     735 kubelet.go:1816] SyncLoop (PLEG): "prometheus-1433796061-59q8b_default(d348b757-fdc1-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"d348b757-fdc1-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"b1b62e91c6d2603b71669fc17bce115bc1762ffac6e7c0409814637cf498660b"}
Feb 28 14:26:01 ip-10-128-171-159 kubelet[735]: I0228 14:26:01.164314     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:26:11 ip-10-128-171-159 kubelet[735]: I0228 14:26:11.183693     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:26:21 ip-10-128-171-159 kubelet[735]: I0228 14:26:21.211494     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:26:31 ip-10-128-171-159 kubelet[735]: I0228 14:26:31.229234     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:26:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:26:39 ip-10-128-171-159 kubelet[735]: I0228 14:26:39.614642     735 kubelet.go:1155] Image garbage collection succeeded
Feb 28 14:26:39 ip-10-128-171-159 kubelet[735]: W0228 14:26:39.631145     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 446
Feb 28 14:26:39 ip-10-128-171-159 kubelet[735]: W0228 14:26:39.631157     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 446
Feb 28 14:26:39 ip-10-128-171-159 kubelet[735]: I0228 14:26:39.631164     735 container_manager_linux.go:434] Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:26:39 ip-10-128-171-159 kubelet[735]: W0228 14:26:39.631224     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 735
Feb 28 14:26:39 ip-10-128-171-159 kubelet[735]: W0228 14:26:39.631232     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 735
Feb 28 14:26:41 ip-10-128-171-159 kubelet[735]: I0228 14:26:41.280763     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:26:46 ip-10-128-171-159 kubelet[735]: I0228 14:26:46.710402     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:26:51 ip-10-128-171-159 kubelet[735]: I0228 14:26:51.299314     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.334093     735 kubelet.go:1781] SyncLoop (ADD, "api"): "m1-stage-3524448618-p1l8j_default(f2a717fb-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.402081     735 kubelet.go:1797] SyncLoop (DELETE, "api"): "m1-stage-2548681241-72677_default(6a174332-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.420315     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/f2a717fb-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2a717fb-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2a717fb-fdc1-11e6-bc48-02ecf0186f8f")
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.520643     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/f2a717fb-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "f2a717fb-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2a717fb-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.527613     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f2a717fb-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2a717fb-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2a717fb-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.634748     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-3524448618-p1l8j_default(f2a717fb-fdc1-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.639520     735 kubelet.go:1781] SyncLoop (ADD, "api"): "m1-stage-3524448618-kxdl0_default(f2d0d876-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.671357     735 kubelet.go:1781] SyncLoop (ADD, "api"): "m1-stage-2548681241-k3rjr_default(f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.722301     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/f2d0d876-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2d0d876-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2d0d876-fdc1-11e6-bc48-02ecf0186f8f")
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.822637     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/f2d0d876-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "f2d0d876-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2d0d876-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.822672     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f")
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.843465     735 kubelet.go:1797] SyncLoop (DELETE, "api"): "m1-stage-3524448618-kxdl0_default(f2d0d876-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.863711     735 kubelet.go:1781] SyncLoop (ADD, "api"): "m1-stage-2548681241-fbcx6_default(f2f1d774-fdc1-11e6-bc48-02ecf0186f8f)"
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.872327     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f2d0d876-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2d0d876-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2d0d876-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.922942     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.929795     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.941434     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-3524448618-kxdl0_default(f2d0d876-fdc1-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:26:52 ip-10-128-171-159 kubelet[735]: I0228 14:26:52.972626     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-2548681241-k3rjr_default(f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:26:53 ip-10-128-171-159 kubelet[735]: I0228 14:26:53.023294     735 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/f2f1d774-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2f1d774-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2f1d774-fdc1-11e6-bc48-02ecf0186f8f")
Feb 28 14:26:53 ip-10-128-171-159 kubelet[735]: I0228 14:26:53.123624     735 reconciler.go:306] MountVolume operation started for volume "kubernetes.io/secret/f2f1d774-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") to pod "f2f1d774-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2f1d774-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:53 ip-10-128-171-159 kubelet[735]: I0228 14:26:53.128129     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f2f1d774-fdc1-11e6-bc48-02ecf0186f8f-default-token-750mk" (spec.Name: "default-token-750mk") pod "f2f1d774-fdc1-11e6-bc48-02ecf0186f8f" (UID: "f2f1d774-fdc1-11e6-bc48-02ecf0186f8f").
Feb 28 14:26:53 ip-10-128-171-159 kubelet[735]: I0228 14:26:53.164446     735 docker_manager.go:1975] Need to restart pod infra container for "m1-stage-2548681241-fbcx6_default(f2f1d774-fdc1-11e6-bc48-02ecf0186f8f)" because it is not found
Feb 28 14:26:53 ip-10-128-171-159 kubelet[735]: I0228 14:26:53.728143     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: E0228 14:26:54.174609     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'm1-stage-3524448618-p1l8j' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: I0228 14:26:54.176541     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-3524448618-p1l8j_default(f2a717fb-fdc1-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"f2a717fb-fdc1-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"61aa50535ce2ca634ab1565a3955d578ebc1d47dd41e497c721a9f8e8d3b4e5a"}
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: E0228 14:26:54.186546     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'm1-stage-2548681241-k3rjr' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: I0228 14:26:54.186788     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-k3rjr_default(f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"f2d4fe46-fdc1-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"4038239174c30363f95fff8002a01b07b64a4a793764e328b9b8e446ab1cc815"}
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: E0228 14:26:54.197553     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'm1-stage-3524448618-kxdl0' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:26:54 ip-10-128-171-159 kubelet[735]: I0228 14:26:54.197708     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-3524448618-kxdl0_default(f2d0d876-fdc1-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"f2d0d876-fdc1-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"e3de558e1d279850ecec12664dadf815dfd7770e8853561390f10306d110ab67"}
Feb 28 14:26:55 ip-10-128-171-159 kubelet[735]: E0228 14:26:55.212313     735 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'm1-stage-2548681241-fbcx6' - Unexpected command output Device "eth0" does not exist.
Feb 28 14:26:55 ip-10-128-171-159 kubelet[735]: with error: exit status 1
Feb 28 14:26:55 ip-10-128-171-159 kubelet[735]: I0228 14:26:55.212453     735 kubelet.go:1816] SyncLoop (PLEG): "m1-stage-2548681241-fbcx6_default(f2f1d774-fdc1-11e6-bc48-02ecf0186f8f)", event: &pleg.PodLifecycleEvent{ID:"f2f1d774-fdc1-11e6-bc48-02ecf0186f8f", Type:"ContainerStarted", Data:"e7c9eaf72e103a05e18bfd2104e42b32c0665201776a7f4b5cc4beea2ba2603d"}
Feb 28 14:27:01 ip-10-128-171-159 kubelet[735]: I0228 14:27:01.315845     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:27:11 ip-10-128-171-159 kubelet[735]: I0228 14:27:11.332380     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:27:21 ip-10-128-171-159 kubelet[735]: I0228 14:27:21.354782     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:27:31 ip-10-128-171-159 kubelet[735]: I0228 14:27:31.373863     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:27:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:27:41 ip-10-128-171-159 kubelet[735]: I0228 14:27:41.392770     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:27:51 ip-10-128-171-159 kubelet[735]: I0228 14:27:51.409584     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:28:00 ip-10-128-171-159 kubelet[735]: I0228 14:28:00.673804     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:28:01 ip-10-128-171-159 kubelet[735]: I0228 14:28:01.428190     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:28:11 ip-10-128-171-159 kubelet[735]: I0228 14:28:11.453984     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:28:12 ip-10-128-171-159 kubelet[735]: I0228 14:28:12.699025     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:28:21 ip-10-128-171-159 kubelet[735]: I0228 14:28:21.469041     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:28:31 ip-10-128-171-159 kubelet[735]: I0228 14:28:31.488327     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:28:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:28:41 ip-10-128-171-159 kubelet[735]: I0228 14:28:41.507013     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:28:51 ip-10-128-171-159 kubelet[735]: I0228 14:28:51.525888     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:29:01 ip-10-128-171-159 kubelet[735]: I0228 14:29:01.551464     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:29:11 ip-10-128-171-159 kubelet[735]: I0228 14:29:11.570146     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:29:21 ip-10-128-171-159 kubelet[735]: I0228 14:29:21.589335     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:29:29 ip-10-128-171-159 kubelet[735]: I0228 14:29:29.673999     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:29:31 ip-10-128-171-159 kubelet[735]: I0228 14:29:31.608010     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:29:32 ip-10-128-171-159 kubelet[735]: I0228 14:29:32.680352     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:29:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:29:41 ip-10-128-171-159 kubelet[735]: I0228 14:29:41.629385     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:29:51 ip-10-128-171-159 kubelet[735]: I0228 14:29:51.659960     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:01 ip-10-128-171-159 kubelet[735]: I0228 14:30:01.696811     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:11 ip-10-128-171-159 kubelet[735]: I0228 14:30:11.726291     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:21 ip-10-128-171-159 kubelet[735]: I0228 14:30:21.745760     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:31 ip-10-128-171-159 kubelet[735]: I0228 14:30:31.764505     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:30:41 ip-10-128-171-159 kubelet[735]: I0228 14:30:41.640513     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:30:41 ip-10-128-171-159 kubelet[735]: I0228 14:30:41.782067     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:51 ip-10-128-171-159 kubelet[735]: I0228 14:30:51.802967     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:30:55 ip-10-128-171-159 kubelet[735]: I0228 14:30:55.669537     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:31:01 ip-10-128-171-159 kubelet[735]: I0228 14:31:01.821565     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:31:11 ip-10-128-171-159 kubelet[735]: I0228 14:31:11.841431     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:31:21 ip-10-128-171-159 kubelet[735]: I0228 14:31:21.864485     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:31:31 ip-10-128-171-159 kubelet[735]: I0228 14:31:31.889584     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:31:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:31:39 ip-10-128-171-159 kubelet[735]: W0228 14:31:39.632665     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 446
Feb 28 14:31:39 ip-10-128-171-159 kubelet[735]: W0228 14:31:39.632685     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 446
Feb 28 14:31:39 ip-10-128-171-159 kubelet[735]: I0228 14:31:39.632693     735 container_manager_linux.go:434] Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:31:39 ip-10-128-171-159 kubelet[735]: W0228 14:31:39.632763     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 735
Feb 28 14:31:39 ip-10-128-171-159 kubelet[735]: W0228 14:31:39.632772     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 735
Feb 28 14:31:41 ip-10-128-171-159 kubelet[735]: I0228 14:31:41.920298     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:31:51 ip-10-128-171-159 kubelet[735]: I0228 14:31:51.943984     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:31:54 ip-10-128-171-159 kubelet[735]: I0228 14:31:54.695564     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:32:01 ip-10-128-171-159 kubelet[735]: I0228 14:32:01.965521     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:32:09 ip-10-128-171-159 kubelet[735]: I0228 14:32:09.728803     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:32:12 ip-10-128-171-159 kubelet[735]: I0228 14:32:12.000121     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:32:22 ip-10-128-171-159 kubelet[735]: I0228 14:32:22.018305     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:32:32 ip-10-128-171-159 kubelet[735]: I0228 14:32:32.037083     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:32:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:32:42 ip-10-128-171-159 kubelet[735]: I0228 14:32:42.063730     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:32:52 ip-10-128-171-159 kubelet[735]: I0228 14:32:52.082187     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:32:55 ip-10-128-171-159 kubelet[735]: I0228 14:32:55.640700     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:33:02 ip-10-128-171-159 kubelet[735]: I0228 14:33:02.111209     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:33:12 ip-10-128-171-159 kubelet[735]: I0228 14:33:12.131711     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:33:12 ip-10-128-171-159 kubelet[735]: I0228 14:33:12.673754     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:33:22 ip-10-128-171-159 kubelet[735]: I0228 14:33:22.151505     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:33:32 ip-10-128-171-159 kubelet[735]: I0228 14:33:32.170444     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:33:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:33:42 ip-10-128-171-159 kubelet[735]: I0228 14:33:42.188974     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:33:52 ip-10-128-171-159 kubelet[735]: I0228 14:33:52.207300     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:34:02 ip-10-128-171-159 kubelet[735]: I0228 14:34:02.223306     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:34:12 ip-10-128-171-159 kubelet[735]: I0228 14:34:12.241838     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:34:22 ip-10-128-171-159 kubelet[735]: I0228 14:34:22.281960     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:34:25 ip-10-128-171-159 kubelet[735]: I0228 14:34:25.730667     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:34:32 ip-10-128-171-159 kubelet[735]: I0228 14:34:32.310340     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:34:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:34:41 ip-10-128-171-159 kubelet[735]: I0228 14:34:41.666236     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:34:42 ip-10-128-171-159 kubelet[735]: I0228 14:34:42.328923     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:34:52 ip-10-128-171-159 kubelet[735]: I0228 14:34:52.352058     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:02 ip-10-128-171-159 kubelet[735]: I0228 14:35:02.369903     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:12 ip-10-128-171-159 kubelet[735]: I0228 14:35:12.389434     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:22 ip-10-128-171-159 kubelet[735]: I0228 14:35:22.477177     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:32 ip-10-128-171-159 kubelet[735]: I0228 14:35:32.505616     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:35:42 ip-10-128-171-159 kubelet[735]: I0228 14:35:42.527672     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:52 ip-10-128-171-159 kubelet[735]: I0228 14:35:52.547296     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:35:52 ip-10-128-171-159 kubelet[735]: I0228 14:35:52.717057     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:36:01 ip-10-128-171-159 kubelet[735]: I0228 14:36:01.635583     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:36:02 ip-10-128-171-159 kubelet[735]: I0228 14:36:02.564043     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:36:12 ip-10-128-171-159 kubelet[735]: I0228 14:36:12.584064     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:36:22 ip-10-128-171-159 kubelet[735]: I0228 14:36:22.605934     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:36:32 ip-10-128-171-159 kubelet[735]: I0228 14:36:32.634526     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:36:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:36:39 ip-10-128-171-159 kubelet[735]: W0228 14:36:39.634112     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 446
Feb 28 14:36:39 ip-10-128-171-159 kubelet[735]: W0228 14:36:39.634151     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 446
Feb 28 14:36:39 ip-10-128-171-159 kubelet[735]: I0228 14:36:39.634159     735 container_manager_linux.go:434] Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:36:39 ip-10-128-171-159 kubelet[735]: W0228 14:36:39.634221     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 735
Feb 28 14:36:39 ip-10-128-171-159 kubelet[735]: W0228 14:36:39.634229     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 735
Feb 28 14:36:42 ip-10-128-171-159 kubelet[735]: I0228 14:36:42.651798     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:36:52 ip-10-128-171-159 kubelet[735]: I0228 14:36:52.678955     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:36:56 ip-10-128-171-159 kubelet[735]: I0228 14:36:56.652120     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:37:02 ip-10-128-171-159 kubelet[735]: I0228 14:37:02.705451     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:37:12 ip-10-128-171-159 kubelet[735]: I0228 14:37:12.734015     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:37:14 ip-10-128-171-159 kubelet[735]: I0228 14:37:14.692532     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:37:22 ip-10-128-171-159 kubelet[735]: I0228 14:37:22.753056     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:37:32 ip-10-128-171-159 kubelet[735]: I0228 14:37:32.771908     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:37:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:37:42 ip-10-128-171-159 kubelet[735]: I0228 14:37:42.790734     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:37:52 ip-10-128-171-159 kubelet[735]: I0228 14:37:52.811490     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:38:02 ip-10-128-171-159 kubelet[735]: I0228 14:38:02.836624     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:38:09 ip-10-128-171-159 kubelet[735]: I0228 14:38:09.718898     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:38:12 ip-10-128-171-159 kubelet[735]: I0228 14:38:12.864078     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:38:22 ip-10-128-171-159 kubelet[735]: I0228 14:38:22.884808     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:38:32 ip-10-128-171-159 kubelet[735]: I0228 14:38:32.670489     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:38:32 ip-10-128-171-159 kubelet[735]: I0228 14:38:32.901786     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:38:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:38:42 ip-10-128-171-159 kubelet[735]: I0228 14:38:42.936051     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:38:52 ip-10-128-171-159 kubelet[735]: I0228 14:38:52.973844     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:39:02 ip-10-128-171-159 kubelet[735]: I0228 14:39:02.996612     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:39:13 ip-10-128-171-159 kubelet[735]: I0228 14:39:13.021089     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:39:21 ip-10-128-171-159 kubelet[735]: I0228 14:39:21.677692     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:39:23 ip-10-128-171-159 kubelet[735]: I0228 14:39:23.038950     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:39:33 ip-10-128-171-159 kubelet[735]: I0228 14:39:33.057190     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:39:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:39:43 ip-10-128-171-159 kubelet[735]: I0228 14:39:43.080425     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:39:46 ip-10-128-171-159 kubelet[735]: I0228 14:39:46.634630     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:39:53 ip-10-128-171-159 kubelet[735]: I0228 14:39:53.106177     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:03 ip-10-128-171-159 kubelet[735]: I0228 14:40:03.123466     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:13 ip-10-128-171-159 kubelet[735]: I0228 14:40:13.142608     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:23 ip-10-128-171-159 kubelet[735]: I0228 14:40:23.160452     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:33 ip-10-128-171-159 kubelet[735]: I0228 14:40:33.175248     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:40:43 ip-10-128-171-159 kubelet[735]: I0228 14:40:43.194076     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:45 ip-10-128-171-159 kubelet[735]: I0228 14:40:45.673273     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:40:53 ip-10-128-171-159 kubelet[735]: I0228 14:40:53.220444     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:40:58 ip-10-128-171-159 kubelet[735]: I0228 14:40:58.702938     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe5e51-faab-11e6-a97a-0668c40b08b9-default-token-w5s1z" (spec.Name: "default-token-w5s1z") pod "a0fe5e51-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe5e51-faab-11e6-a97a-0668c40b08b9").
Feb 28 14:41:03 ip-10-128-171-159 kubelet[735]: I0228 14:41:03.242981     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:41:13 ip-10-128-171-159 kubelet[735]: I0228 14:41:13.262686     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:41:23 ip-10-128-171-159 kubelet[735]: I0228 14:41:23.282564     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:41:33 ip-10-128-171-159 kubelet[735]: I0228 14:41:33.302828     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:41:38 ip-10-128-171-159 systemd[1]: Started Kubernetes Kubelet Server.
Feb 28 14:41:39 ip-10-128-171-159 kubelet[735]: W0228 14:41:39.634552     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 446
Feb 28 14:41:39 ip-10-128-171-159 kubelet[735]: W0228 14:41:39.635055     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 446
Feb 28 14:41:39 ip-10-128-171-159 kubelet[735]: I0228 14:41:39.635391     735 container_manager_linux.go:434] Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:41:39 ip-10-128-171-159 kubelet[735]: W0228 14:41:39.635788     735 container_manager_linux.go:728] CPUAccounting not enabled for pid: 735
Feb 28 14:41:39 ip-10-128-171-159 kubelet[735]: W0228 14:41:39.636108     735 container_manager_linux.go:731] MemoryAccounting not enabled for pid: 735
Feb 28 14:41:43 ip-10-128-171-159 kubelet[735]: I0228 14:41:43.321450     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:41:53 ip-10-128-171-159 kubelet[735]: I0228 14:41:53.338703     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:42:03 ip-10-128-171-159 kubelet[735]: I0228 14:42:03.357170     735 aws.go:897] Could not determine public IP from AWS metadata.
Feb 28 14:42:07 ip-10-128-171-159 kubelet[735]: I0228 14:42:07.658133     735 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/a0fe2ad1-faab-11e6-a97a-0668c40b08b9-default-token-750mk" (spec.Name: "default-token-750mk") pod "a0fe2ad1-faab-11e6-a97a-0668c40b08b9" (UID: "a0fe2ad1-faab-11e6-a97a-0668c40b08b9").

kube-weave logs

INFO: 2017/02/28 14:21:53.110489 Command line options: map[status-addr:0.0.0.0:6782 http-addr:127.0.0.1:6784 ipalloc-range:10.32.0.0/12 name:4e:79:89:47:38:6f nickname:ip-10-128-171-159 no-dns:true port:6783 docker-api: datapath:datapath ipalloc-init:consensus=4]
INFO: 2017/02/28 14:21:53.118157 Communication between peers is unencrypted.
INFO: 2017/02/28 14:21:53.132280 Our name is 4e:79:89:47:38:6f(ip-10-128-171-159)
INFO: 2017/02/28 14:21:53.132388 Launch detected - using supplied peer list: [10.128.146.44 10.128.169.34 10.128.171.159 10.128.182.241]
INFO: 2017/02/28 14:21:53.200006 [allocator 4e:79:89:47:38:6f] Initialising with persisted data
INFO: 2017/02/28 14:21:53.200089 Sniffing traffic on datapath (via ODP)
INFO: 2017/02/28 14:21:53.202119 ->[10.128.171.159:6783] attempting connection
INFO: 2017/02/28 14:21:53.202352 ->[10.128.182.241:6783] attempting connection
INFO: 2017/02/28 14:21:53.202489 ->[10.128.146.44:6783] attempting connection
INFO: 2017/02/28 14:21:53.202592 ->[10.128.169.34:6783] attempting connection
INFO: 2017/02/28 14:21:53.212381 ->[10.128.171.159:46157] connection accepted
INFO: 2017/02/28 14:21:53.217975 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2017/02/28 14:21:53.221206 Listening for metrics requests on 0.0.0.0:6782
INFO: 2017/02/28 14:21:53.236326 ->[10.128.146.44:6783|fa:ce:b9:a9:74:d3(ip-10-128-146-44)]: connection ready; using protocol version 2
INFO: 2017/02/28 14:21:53.271311 overlay_switch ->[fa:ce:b9:a9:74:d3(ip-10-128-146-44)] using fastdp
INFO: 2017/02/28 14:21:53.272996 ->[10.128.171.159:6783|4e:79:89:47:38:6f(ip-10-128-171-159)]: connection shutting down due to error: cannot connect to ourself
INFO: 2017/02/28 14:21:53.279215 ->[10.128.171.159:46157|4e:79:89:47:38:6f(ip-10-128-171-159)]: connection shutting down due to error: cannot connect to ourself
INFO: 2017/02/28 14:21:53.279353 ->[10.128.169.34:6783|9a:bf:c6:d2:da:9a(ip-10-128-169-34)]: connection ready; using protocol version 2
INFO: 2017/02/28 14:21:53.283156 ->[10.128.182.241:6783|76:aa:74:27:76:ce(ip-10-128-182-241)]: connection ready; using protocol version 2
INFO: 2017/02/28 14:21:53.283272 overlay_switch ->[9a:bf:c6:d2:da:9a(ip-10-128-169-34)] using fastdp
INFO: 2017/02/28 14:21:53.283300 ->[10.128.146.44:6783|fa:ce:b9:a9:74:d3(ip-10-128-146-44)]: connection added (new peer)
INFO: 2017/02/28 14:21:53.283459 overlay_switch ->[76:aa:74:27:76:ce(ip-10-128-182-241)] using fastdp
INFO: 2017/02/28 14:21:53.283815 ->[10.128.169.34:6783|9a:bf:c6:d2:da:9a(ip-10-128-169-34)]: connection added (new peer)
INFO: 2017/02/28 14:21:53.283975 ->[10.128.182.241:6783|76:aa:74:27:76:ce(ip-10-128-182-241)]: connection added (new peer)
INFO: 2017/02/28 14:21:53.294066 ->[10.128.182.241:6783|76:aa:74:27:76:ce(ip-10-128-182-241)]: connection fully established
INFO: 2017/02/28 14:21:53.308735 ->[10.128.146.44:6783|fa:ce:b9:a9:74:d3(ip-10-128-146-44)]: connection fully established
INFO: 2017/02/28 14:21:53.309124 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2017/02/28 14:21:53.309544 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2017/02/28 14:21:53.310062 ->[10.128.169.34:6783|9a:bf:c6:d2:da:9a(ip-10-128-169-34)]: connection fully established
INFO: 2017/02/28 14:21:53.322485 sleeve ->[10.128.182.241:6783|76:aa:74:27:76:ce(ip-10-128-182-241)]: Effective MTU verified at 8939
INFO: 2017/02/28 14:21:53.322662 sleeve ->[10.128.146.44:6783|fa:ce:b9:a9:74:d3(ip-10-128-146-44)]: Effective MTU verified at 8939
INFO: 2017/02/28 14:21:53.328324 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2017/02/28 14:21:53.341330 sleeve ->[10.128.169.34:6783|9a:bf:c6:d2:da:9a(ip-10-128-169-34)]: Effective MTU verified at 8939
INFO: 2017/02/28 14:21:53.596461 Discovered local MAC 62:01:46:98:d0:cb
INFO: 2017/02/28 14:21:53.663624 Weave version 1.9.1 is available; please update at https://github.com/weaveworks/weave/releases/download/v1.9.1/weave
INFO: 2017/02/28 14:21:53.760375 Discovered local MAC 4e:79:89:47:38:6f
10.34.2.6
INFO: 2017/02/28 14:21:57.100386 Discovered local MAC 5e:67:2d:f7:af:3d
INFO: 2017/02/28 14:36:53.135372 Expired MAC 4e:79:89:47:38:6f at 4e:79:89:47:38:6f(ip-10-128-171-159)
INFO: 2017/02/28 14:36:53.135405 Expired MAC 5e:67:2d:f7:af:3d at 4e:79:89:47:38:6f(ip-10-128-171-159)
INFO: 2017/02/28 14:36:53.135417 Expired MAC 62:01:46:98:d0:cb at 4e:79:89:47:38:6f(ip-10-128-171-159)

Scheduling deployment of prometheus:

rln@rln-hoppolek:~/.ssh$ kubectl describe pods prometheus-1433796061-59q8b
Name:           prometheus-1433796061-59q8b
Namespace:      default
Node:           ip-10-128-171-159.eu-west-1.compute.internal/10.128.171.159
Start Time:     Tue, 28 Feb 2017 14:25:59 +0000
Labels:         app=prometheus
                pod-template-hash=1433796061
Status:         Pending
IP:
Controllers:    ReplicaSet/prometheus-1433796061
Containers:
  prometheus:
    Container ID:
    Image:              docker-registry.example.com/testing/prometheus:latest
    Image ID:
    Port:               9090/TCP
    Args:
      -storage.local.retention=6h
      -storage.local.memory-chunks=500000
      -config.file=/etc/prometheus/prometheus.yml
    Requests:
      cpu:              100m
    State:              Waiting
      Reason:           ContainerCreating
    Ready:              False
    Restart Count:      0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-750mk (ro)
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  default-token-750mk:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-750mk
QoS Class:      Burstable
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason          Message
  ---------     --------        -----   ----                    -------------   --------        ------          -------
  26m           26m             1       {default-scheduler }                    Normal          Scheduled       Successfully assigned prometheus-1433796061-59q8b to ip-10-128-171-159.eu-west-1.compute.internal
@bboreham
Copy link
Contributor

The last "connection refused" is:

Feb 28 14:21:53 ip-10-128-171-159 kubelet[735]: E0228 14:21:53.111447     735 docker_manager.go:2201] Failed to setup network for pod "m1-stage-2548681241-b1r9w_default(de7bdd6e-fdc0-11e6-bc48-02ecf0186f8f)" using network plugins "cni": unable to allocate IP address: Post http://127.0.0.1:6784/ip/20e61a9927959f0cfe54e64fba57ab528eae69748bbcaa30afe58bd38bed2679: dial tcp 127.0.0.1:6784: getsockopt: connection refused; Skipping pod

and the weave-kube container starts right after that:

INFO: 2017/02/28 14:21:53.110489 Command line options: map[status-addr:0.0.0.0:6782 http-addr:127.0.0.1:6784 ipalloc-range:10.32.0.0/12 name:4e:79:89:47:38:6f nickname:ip-10-128-171-159 no-dns:true port:6783 docker-api: datapath:datapath ipalloc-init:consensus=4]

@bboreham
Copy link
Contributor

To clarify, the issue is that the pod sits in "ContainerCreating" state, and I cannot see anything in those logs to explain why. I'm going to edit the issue description as "node can't connect to 127.0.0.1:6784" is just a transient state before it all gets going

@bboreham bboreham changed the title Pods get stuck in ContainerCreating; node can't connect to 127.0.0.1:6784 Pods get stuck in ContainerCreating Feb 28, 2017
@bboreham
Copy link
Contributor

After much investigating via Slack, the problem was traced here:

# curl 'http://127.0.0.1:6784/status'
        Version: 1.8.2 (version 1.9.1 available - please upgrade!)

        Service: router
       Protocol: weave 1..2
           Name: 66:2b:6a:ca:34:88(ip-10-128-152-185)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 4
    Connections: 4 (3 established, 1 failed)
          Peers: 4 (with 12 established connections)
 TrustedSubnets: none

        Service: ipam
         Status: waiting for IP range grant from peers
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

The "waiting for IP range grant from peers" status indicates that Weave Net's IPAM believes that all the IP address space is owned by other nodes in the cluster, but actually none of those nodes are able to be contacted at the moment.

So the underlying cause is #2797

@rlnrln
Copy link
Author

rlnrln commented Mar 1, 2017 •
edited
Loading

Thank you, Bryan, for all the assistance in tracking down this issue yesterday.

Here's my workaround. Big red warnings:

  • All unreachable hosts were first identified as gone forever.
  • Do not run this on more than one node.
  • This may screw up your kubernetes cluster if something goes wrong.
  • There is a failsafe 'echo' added to the command in case you didn't read the above warnings.
% for i in $(curl -s 'http://127.0.0.1:6784/status/ipam' | grep 'unreachable\!$' | sort -k2 -n -r | awk -F'(' '{print $2}' | sed 's/).*//'); do echo curl -X DELETE 127.0.0.1:6784/peer/$i; done
65536 IPs taken over from ip-10-128-184-15
32768 IPs taken over from ip-10-128-159-154
32768 IPs taken over from ip-10-128-170-84
...

@bboreham
Copy link
Contributor

bboreham commented Mar 1, 2017

Thanks, @rlnrln. Going to close this as a duplicate.

@bboreham bboreham closed this as completed Mar 1, 2017
@bboreham bboreham added this to the n/a milestone Mar 21, 2017
@AlexRRR AlexRRR mentioned this issue Dec 19, 2017
Closed
@mhasrup
Copy link

mhasrup commented Jan 10, 2022

Please help me , i am trying to use weave networking solution my k8 cluster. but i am getting below errors.
See below PODS are not running. (This page i referring for K8 setup but stuck at POD networking step)
https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/README.md

vagrant@master-1:$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-69cbb76ff8-nql5k 0/1 ContainerCreating 0 17h
coredns-69cbb76ff8-p44xg 0/1 ContainerCreating 0 17h
weave-net-2vdr7 1/2 CrashLoopBackOff 22 17h
weave-net-pqrx8 1/2 CrashLoopBackOff 23 17h
vagrant@master-1:$

============================
2. check my k8 version
vagrant@master-1:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

===============
3. see my kubelet logs in worker-1 logs

kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-01-09 11:20:30 UTC; 17h ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 17379 (kubelet)
Tasks: 13 (limit: 2360)
CGroup: /system.slice/kubelet.service
└─17379 /usr/local/bin/kubelet --config=/var/lib/kubelet/kubelet-config.yaml --image-pull-progress-deadline=2m --kubeconfig=/var/lib

Jan 10 04:35:53 worker-1 kubelet[17379]: E0110 04:35:53.787040 17379 pod_workers.go:190] Error syncing pod ed5aa617-713c-11ec-9aa3-0229e0c3be
Jan 10 04:36:05 worker-1 kubelet[17379]: E0110 04:36:05.613166 17379 pod_workers.go:190] Error syncing pod 9acb983f-713c-11ec-9aa3-0229e0c3be
Jan 10 04:36:08 worker-1 kubelet[17379]: I0110 04:36:08.687387 17379 kuberuntime_manager.go:415] No ready sandbox for pod "coredns-69cbb76ff8
Jan 10 04:36:08 worker-1 kubelet[17379]: W0110 04:36:08.691641 17379 cni.go:302] CNI failed to retrieve network namespace path: cannot find n
Jan 10 04:36:08 worker-1 kubelet[17379]: weave-cni: unable to release IP address: Delete "http://127.0.0.1:6784/ip/b1858edb33e7a3f91fdc10f72a42
Jan 10 04:36:08 worker-1 kubelet[17379]: E0110 04:36:08.722868 17379 cni.go:345] Error deleting kube-system_coredns-69cbb76ff8-nql5k/b1858edb
Jan 10 04:36:08 worker-1 kubelet[17379]: E0110 04:36:08.723621 17379 remote_runtime.go:119] StopPodSandbox "b1858edb33e7a3f91fdc10f72a42b18db
Jan 10 04:36:08 worker-1 kubelet[17379]: E0110 04:36:08.723837 17379 kuberuntime_manager.go:815] Failed to stop sandbox {"docker" "b1858edb33
Jan 10 04:36:08 worker-1 kubelet[17379]: E0110 04:36:08.724008 17379 kuberuntime_manager.go:610] killPodWithSyncResult failed: failed to "Kil
Jan 10 04:36:08 worker-1 kubelet[17379]: E0110 04:36:08.724147 17379 pod_workers.go:190] Error syncing pod ed5aa617-713c-11ec-9aa3-0229e0c3be

==============================

  1. check my POD logs

vagrant@master-1:~$ kubectl describe pod weave-net-2vdr7 -n kube-system
Name: weave-net-2vdr7
Namespace: kube-system
Priority: 2000001000
PriorityClassName: system-node-critical
Node: worker-1/192.168.5.21
Start Time: Sun, 09 Jan 2022 11:09:30 +0000
Labels: controller-revision-hash=6fd65954f8
name=weave-net
pod-template-generation=3
Annotations:
Status: Running
IP: 192.168.5.21
Controlled By: DaemonSet/weave-net
Init Containers:
weave-init:
Container ID: docker://7b3e10e3a89a064955e3533dd2c082e8e6e9a98e4d2c2afc3775f7f2dd6f740b
Image: ghcr.io/weaveworks/launcher/weave-kube:2.8.1
Image ID: docker-pullable://ghcr.io/weaveworks/launcher/weave-kube@sha256:d797338e7beb17222e10757b71400d8471bdbd9be13b5da38ce2ebf597fb4e63
Port:
Host Port:
Command:
/home/weave/init.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 09 Jan 2022 11:09:32 +0000
Finished: Sun, 09 Jan 2022 11:09:33 +0000
Ready: True
Restart Count: 0
Environment:
Mounts:
/host/etc from cni-conf (rw)
/host/home from cni-bin2 (rw)
/host/opt from cni-bin (rw)
/lib/modules from lib-modules (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-68jgk (ro)
Containers:
weave:
Container ID: docker://7cfc10bcf41be5529e4ea3695f74549f3ec707766e03f8cec0b70d24e2f91ad9
Image: ghcr.io/weaveworks/launcher/weave-kube:2.8.1
Image ID: docker-pullable://ghcr.io/weaveworks/launcher/weave-kube@sha256:d797338e7beb17222e10757b71400d8471bdbd9be13b5da38ce2ebf597fb4e63
Port:
Host Port:
Command:
/home/weave/launch.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 10 Jan 2022 04:32:48 +0000
Finished: Mon, 10 Jan 2022 04:33:18 +0000
Ready: False
Restart Count: 20
Requests:
cpu: 50m
memory: 100Mi
Readiness: http-get http://127.0.0.1:6784/status delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
HOSTNAME: (v1:spec.nodeName)
INIT_CONTAINER: true
Mounts:
/host/etc/machine-id from machine-id (ro)
/host/var/lib/dbus from dbus (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-68jgk (ro)
/weavedb from weavedb (rw)
weave-npc:
Container ID: docker://d18dfe0bb8fc41259f1ccae8e344a1671f5b1b60a5ac807aadb85fb8acbf8568
Image: ghcr.io/weaveworks/launcher/weave-npc:2.8.1
Image ID: docker-pullable://ghcr.io/weaveworks/launcher/weave-npc@sha256:38d3e30a97a2260558f8deb0fc4c079442f7347f27c86660dbfc8ca91674f14c
Port:
Host Port:
State: Running
Started: Sun, 09 Jan 2022 11:09:36 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 50m
memory: 100Mi
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-68jgk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
weavedb:
Type: HostPath (bare host directory volume)
Path: /var/lib/weave
HostPathType:
cni-bin:
Type: HostPath (bare host directory volume)
Path: /opt
HostPathType:
cni-bin2:
Type: HostPath (bare host directory volume)
Path: /home
HostPathType:
cni-conf:
Type: HostPath (bare host directory volume)
Path: /etc
HostPathType:
dbus:
Type: HostPath (bare host directory volume)
Path: /var/lib/dbus
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
machine-id:
Type: HostPath (bare host directory volume)
Path: /etc/machine-id
HostPathType: FileOrCreate
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
weave-net-token-68jgk:
Type: Secret (a volume populated by a Secret)
SecretName: weave-net-token-68jgk
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: :NoSchedule
:NoExecute
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message

Normal Scheduled 17h default-scheduler Successfully assigned kube-system/weave-net-2vdr7 to worker-1
Normal Pulled 17h kubelet, worker-1 Container image "ghcr.io/weaveworks/launcher/weave-kube:2.8.1" already present on machine
Normal Created 17h kubelet, worker-1 Created container
Normal Started 17h kubelet, worker-1 Started container
Normal Pulled 17h kubelet, worker-1 Container image "ghcr.io/weaveworks/launcher/weave-npc:2.8.1" already present on machine
Normal Started 17h kubelet, worker-1 Started container
Normal Created 17h kubelet, worker-1 Created container
Normal Pulled 17h (x3 over 17h) kubelet, worker-1 Container image "ghcr.io/weaveworks/launcher/weave-kube:2.8.1" already present on machine
Normal Started 17h (x3 over 17h) kubelet, worker-1 Started container
Normal Created 17h (x3 over 17h) kubelet, worker-1 Created container
Warning Unhealthy 17h (x9 over 17h) kubelet, worker-1 Readiness probe failed: Get http://127.0.0.1:6784/status: dial tcp 127.0.0.1:6784: connect: connection refused
Warning BackOff 17h (x30 over 17h) kubelet, worker-1 Back-off restarting failed container
Normal Created 17h (x3 over 17h) kubelet, worker-1 Created container
Normal Started 17h (x3 over 17h) kubelet, worker-1 Started container
Normal Pulled 16h (x9 over 17h) kubelet, worker-1 Container image "ghcr.io/weaveworks/launcher/weave-kube:2.8.1" already present on machine
Warning BackOff 16h (x158 over 17h) kubelet, worker-1 Back-off restarting failed container
Warning Unhealthy 112s (x43 over 17h) kubelet, worker-1 Readiness probe failed: Get http://127.0.0.1:6784/status: dial tcp 127.0.0.1:6784: connect: connection refused

I dont understand what else to check . Anybody please help me. i want to set up k8 cluster i my laptop first so i will practise and will clear CKA exam.

@bboreham
Copy link
Contributor

Some tips at https://www.weave.works/docs/net/latest/kubernetes/kube-addon/#troubleshooting

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
resolution/duplicate
Projects
None yet
Milestone
n/a
Development

No branches or pull requests
3 participants
@bboreham @rlnrln @mhasrup