Kubelet log flooded by SyncLoop on nginx-ingress-controller

[Leen15]

Hi, I just upgraded rancher from 2.4.8 to 2.5.2.
The rancher installation is on docker, and the same node was the controlplane of a cluster of 2 nodes (1 controlplane, 1 worker), version 1.18.10 deployed by Rancher UI.
After the upgrade, the load of the node has became out of control, so I promoted the worker to controlplane, and tried to convert the first node to worker.
After multiple times, I'm still not able to have that node as a worker (I receive multiple errors about ephemeral storage, I don't know why).
So I removed it and I kept the cluster with only 1 node.
Now, in this node I have kubelet log flooded by these logs:

I1112 15:46:41.012218    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.029633    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.046936    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.067353    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.087268    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.106162    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.131468    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.147064    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.168857    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.180464    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.263065    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.263308    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.263412    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.266875    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.276597    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.286090    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.305156    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.315786    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.325132    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.334698    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"
I1112 15:46:41.348565    2795 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-qcmfj_ingress-nginx(4e192151-4cd5-452b-9da1-8487645b8329)"

At the same time, both kubelet and api-server use LOTS of resources of the node.

Any idea why this thing is happening and how to resolve?

The cluster is working, and nginx ingress too.

Thanks

[Leen15]

The issue seems to be related to #30045.
As suggested by @superseb on slack, I disabled the project network isolation and the log is gone. Also node load is now normal.

[sowmyav27]

Verified an upgrade from 2.4.8 to 2.5.2

• On 2.4.8 rancher
• Deploy a custom cluster - Project Network Isolation enabled.
• Enable Monitoring in this cluster
• Upgrade to 2.5.2 rancher.
• Kubelet logs on worker node:

I1113 17:21:33.933579   10578 kubelet.go:1917] SyncLoop (UPDATE, "api"): "exporter-node-cluster-monitoring-vwpb7_cattle-prometheus(a67eaa77-5cdd-4683-8e16-17fdda1f43e6)"
I1113 17:21:33.950426   10578 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-28k4x_ingress-nginx(ebde362b-fa65-45c3-9f65-1aae0d1a589c)"
I1113 17:21:33.994919   10578 kubelet.go:1917] SyncLoop (UPDATE, "api"): "exporter-node-cluster-monitoring-vwpb7_cattle-prometheus(a67eaa77-5cdd-4683-8e16-17fdda1f43e6)"
I1113 17:21:34.011517   10578 kubelet.go:1917] SyncLoop (UPDATE, "api"): "nginx-ingress-controller-28k4x_ingress-nginx(ebde362b-fa65-45c3-9f65-1aae0d1a589c)"

• Monitoring metrics- considerable increase in CPU utilization after an upgrade. Upgrade was done around 9.20 am