CNI is stopped working after upgrade to v1.16

[kvaps]

What happened:

After upgrade kubernetes 1.15 --> v1.16, containers are stuck on Pending:

  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               <unknown>            default-scheduler  Successfully assigned ingress-nginx/nginx-ingress-controller-95b5597dc-c4kqk to pve2
  Warning  FailedCreatePodSandBox  2m57s                kubelet, pve2      Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "2b10637b5413345671e640e554394a601fe0e7b80db70fe0d4a0db7c4435bb26" network for pod "nginx-ingress-controller-95b5597dc-c4kqk": networkPlugin cni failed to set up pod "nginx-ingress-controller-95b5597dc-c4kqk_ingress-nginx" network: error getting ClusterInformation: connection is unauthorized: Unauthorized, failed to clean up sandbox container "2b10637b5413345671e640e554394a601fe0e7b80db70fe0d4a0db7c4435bb26" network for pod "nginx-ingress-controller-95b5597dc-c4kqk": networkPlugin cni failed to teardown pod "nginx-ingress-controller-95b5597dc-c4kqk_ingress-nginx" network: error getting ClusterInformation: connection is unauthorized: Unauthorized]
  Normal   SandboxChanged          9s (x11 over 2m56s)  kubelet, pve2      Pod sandbox changed, it will be killed and re-created.

What you expected to happen:

Pods are up and running

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

After downgrade kubelet v0.16.0-00 to 1.15.4-00 pods started working

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: barmetal
• OS (e.g: cat /etc/os-release):

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• Kernel (e.g. uname -a):

Linux pve1 4.15.18-12-pve #1 SMP PVE 4.15.18-36 (Fri, 05 Apr 2019 18:47:13 +0200) x86_64 GNU/Linux

• Install tools: kubeadm

kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Network plugin and version (if this is a network-related bug): cni bridge (bridget):

kubernetes-cni/kubernetes-xenial,now 0.7.5-00 amd64 [installed,automatic]

• Others:

[kvaps]

/sig node

[zouyee]

similar issue: #83048

[github-aix-pw]

agreed. as far as I can tell the bridget.sh script does not define cniVersion.

[kvaps]

Thank you guys, problem solved by updating plugin and adding csiVersion to csi config