Cluster based on Azure CNI, POD in error state when start the node, indicate error "docker_sandbox.go:375] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook".

[tiffanyguo721]

Cluster based on Azure CNI, we found POD in error state when start the node, usually stop and start the node, POD statu can back to normal, is it an issue with Azure CNI or Kubernets?

Version:
Azure CNI: v1.0.17
Kubernetes:1.12.7
AKS engine:v0.33.4

node is started at Jul 09 03:49 (UTC), when started many pods are in error state, download the kubernete log and CNI log as below, when stop and start the node, POD status back to normal, this issue can repro whenever the node is started, we found there're similar issues with different CNI other than Azure CNI, but also indicate " failed to read pod IP from plugin/docker", please help investigate, thanks

kubenetes error log.txt

Azure CNI log.txt

[tamilmani1989]

@tiffanyguo721 It looks like you attached azure-vnet-ipam.log. Can you also attach azure-vnet.log?

[tiffanyguo721]

@tamilmani1989 Please kindly check the azure-vnet.log, node is started at Jul 09 03:49 (UTC),

azure-vnet.log.txt

Thanks

[tamilmani1989]

@tiffanyguo721 remove if azure-vnet.json.lock or azure-vnet-ipam.json.lock present in /var/run directory and then retry deploying pods..also i would recommend you to move to latest cni

[tiffanyguo721]

@tamilmani1989 i didn't find any azure-vnet.json.lock or azure-vnet-ipam.json.lock under /var/run directory, what might the issue according to the logs? this issue happened before IP address exausted

[tiffanyguo721]

@tamilmani1989 I did some research , according to your comment, suspect it is "timed out locking store" issue, this issue repro today, I found the "azure-vnet.json.lock" file under /var/run directory, pod back to running after deleting this file.
But I found a fix mentioned regarding this issue: https://github.com/Azure/azure-container-networking/releases/tag/v1.0.17
It mean issue been fixed in v1.0.17. but my environment is based on v1.0.17, but issue still exist, do we have a permanent fix?

Meanwhile, I will move to latest cni, and then monitor if still exist

[tamilmani1989]

That fix is for windows. In Linux, we haven't seen this.

can you send us entire logs. /var/run-azure-vnet*.log (like log.1,log.2..etc) all logs. This happens only if CNI crashes/panic in the middle. Wanted to see what happened there.

[tiffanyguo721]

@tamilmani1989

HI ,
Azure vnet logs attached, please help investigate, thanks

Azure-vnet logs.zip

[tamilmani1989]

@tiffanyguo721 This is fixed. The fix is merged to master. We will release v1.0.25 by today.