watchercache.go spamming log: Failed to create watcher ListRoot="/calico/ipam/v2/assignment/"

[holmesb]

Hi,

We've just upgraded from v3.16.5 to v3.18.1. The calico-kube-controllers pod log is filled (many events per second) with messages from watchercache.go.

2021-04-23 06:58:23.990 [INFO][1] watchercache.go 243: Failed to create watcher ListRoot="/calico/ipam/v2/assignment/" error=connection is unauthorized: unknown (get IPAMBlocks.crd.projectcalico.org) performFullResync=true
2021-04-23 06:58:23.990 [INFO][1] watchercache.go 174: Full resync is required ListRoot="/calico/ipam/v2/assignment/"

Its CPU consumption has also increased significantly.

I've checked that a ClusterRole & ClusterRoleBinding: calico-node both exist. They are the same as on another cluster running v3.16.5 that doesn't exhibit this problem. Also, there are no notable differences in the calico-kube-controllers deployments on the two clusters, other than the image version. Configmap: calico-config is the same.

Steps to Reproduce (for bugs)

Upgrade from Calico v3.16.5 to v3.18.1. Also occurs when doing a clean install of k8s and Calico v3.18.1.

Your Environment

• Calico version 3.18.1
• Orchestrator version: k8s 1.19.7
• Operating System and version: Ubuntu 20.04

Many thanks

[caseydavenport]

@holmesb did you update the ClusterRole as part of that upgrade?

When upgrading Calico image versions, you also need to update the associated Kubernetes resources, as they are pretty tightly coupled.

v3.18.0 introduced a new requirement for kube-controllers to watch IPAM blocks. If you compare the v3.18 ClusterRole with the v3.16 cluster role, you'll notice this change:

   # IPAM resources are manipulated when nodes are deleted.
    - apiGroups: ["crd.projectcalico.org"]
      resources:
        - ippools
      verbs:
        - list
    - apiGroups: ["crd.projectcalico.org"]
      resources:
        - blockaffinities
        - ipamblocks
        - ipamhandles
      verbs:
        - get
        - list
        - create
        - update
        - delete
+       - watch

In general, don't expect the old manifests to work with newer code (and vice-versa).

I'd recommend looking at using the Calico operator to install Calico, which will make sure the running configuration matches the running code without needing human intervention.