Backport: cluster-monitoring is being constantly re-deployed

[alena1108]

Backport #19945

[jiaqiluo]

The bug fix is validated on Rancher: v2.2.3-rc8

Test 1:

• add a cluster with 3 etcd nodes, 2 control plane nodes and any number of worker nodes
• enable the cluster monitoring
• enable the project monitoring

Result:

• notice that the app cluster-monitoring and project-monitoring are deployed only one time.

Test 2:

• run Rancher: v2.2.2
• add two identical clusters and each has 3 etcd nodes, 2 control plane nodes and any number of worker nodes
• on cluster1 enable the cluster monitoring
• upgrade Rancher to v2.2.3-rc8
• check if the cluster-monitoring app gets re-deployed multiple times
• on cluster1 enable the project monitoring
• on cluster2 enable the cluster monitoring and the project monitoring

Result:

• the app cluster-monitoring and project-monitoring are deployed only one time.

[jiaqiluo]

The bug is not fixed, see #19945 (comment)

[alena1108]

Can be tested with v2.2.2-patch1-rc2

[jiaqiluo]

The bug fix is validated on Rancher: v2.2.2-patch1-rc2

Test 1 - single install, fresh install:

• add a cluster with 3 etcd nodes, 2 control plane nodes, and 2 worker nodes
• enable the cluster monitoring
• deploy some workloads and apps
• run automation tests on the cluster

Result:

• notice that the app cluster-monitoring and project-monitoring are deployed only one time.

Test 2 - upgrade path, HA install:

• run Rancher: v2.2.2 on 3-node HA mode
• add one custom RKE cluster and one imported RKE cluster, each has 3 etcd nodes, 2 control plane nodes, and 2 worker nodes
• enable the cluster monitoring
• run pre-upgrade tests
• upgrade Rancher to v2.2.3-rc8
• run post-upgrade tests
• disable cluster monitoring to confirm that old config maps are deleted
• enable cluster monitoring again
• deploy some workloads and apps
• run automation tests on the cluster

Result:

• the app cluster-monitoring and project-monitoring are deployed only one time.

[krisdevopsbot]

Upgraded cluster to 2.2.3 disabled/enabled monitoring (cattle-prometheus namespace was deleted.) Enabled cluster monitoring again and seeing cluster-monitor already at cluster-monitoring.v18. Is there anything in the upgrade path I may have missed?

[jiaqiluo]

Hi @krisdevopsbot
What is the Rancher version of your setup before upgrading? Do you mean that you disable the monitoring before upgrading, and enable it after upgrading to v2.2.3?

Did you delete the namespace cattle-prometheus manually? Disabling monitoring does not delete the namespace by itself. And it should not be a problem if you delete it manually.

[roycec]

Hello,

I upgraded from 2.2.2 to 2.2.3 yesterday. I then saw, the configmaps still getting created. Today I disabled the monitoring feature (on cluster level), waited until the components were removed and activated monitoring again. But with the same result, that a new configmap is created every few minutes.

[krisdevopsbot]

Yes, I disabled the feature and checked to make sure cattle-prometheus was deleted prior up upgrade

[krisdevopsbot]

I built a new 3 node cluster on new VMs with RKE (HA airgap install) and still seeing this on 2.2.3

[krisdevopsbot]

#20186 workaround fixed it