ClusterGroup status is not updated after removing cluster from group

[janeczku]

Steps to reproduce:

1. Register two clusters
2. Create a Cluster Group using a common label attached to both clusters
3. Create a GitRepo resource targeting the cluster group
4. After everything is in-sync, remove the common label from one of the cluster such that the group selector only matches a single cluster

Expected:

status.clusterCount field of the ClusterGroup resource is updated to 1.

Actual:

status.clusterCount field of the ClusterGroup resource remains 2 for another 5-10 minutes. Only then it changes to 1.

Other information:

When adding a cluster to a group instead of removing, the status is reflected immediately.

[nickgerace]

Leaving in "Review" until a Fleet RC containing this change is available in Rancher.

[davidnuzik]

I successfully reproduced

Reproduction Environment:

Rancher version: v2.6.0
Rancher cluster type: single-node docker install
Docker version: 20.10

Downstream cluster type: (not relevant)
Downstream K8s version: (not relevant)

Reproduction steps:

1. Register two clusters
2. Create a Cluster Group using a common label attached to both clusters
3. Create a GitRepo resource targeting the cluster group
4. After everything is in-sync, remove the common label from one of the cluster such that the group selector only matches a single cluster

Note, the status.clusterCount field of the ClusterGroup resource is expected to update to 1, but it remains 2 for about 5-10 minutes (even if you refresh the gitRepos page /dashboard/c/_/fleet/fleet.cattle.io.gitrepo). You can, however, click on the gitrepo to see the resources and it doesn't show the cluster listed there anymore only the main gitrepos page still shows 2 clusters listed. After that time the main gitrepo page does change to 1. It took me about 12 minutes until this number updated. Status should be updated immediately. (I successfully reproduced the issue)

Additional Info:
If you are curious why you are seeing these reproduction steps before the issue passes to QA it's because it's a new procedure - we try to reproduce issues early before they move to QA swimlanes.

[nickgerace]

Available on Rancher v2.6-head in Fleet v0.3.8-rc1

[davidnuzik]

My checks PASSED
I filed a new issue, unrelated to the fix, which is mostly likely a frontend issue by my guess. The Cluster Group object updates in the UI and only shows 1 cluster, but on the Git Repos page it will consistently still show 2 clusters ready which is not true. https://github.com/rancher/dashboard/issues/4564

Reproduction Steps:

Not required.

---

Validation Environment:

Rancher version: v2.6-head cc2b8be 11/12/2021
Rancher cluster type: single-node docker install
Docker version: 20.10

Downstream cluster type: n/a (is not relevant)
Downstream K8s version: n/a (is not relevant)

Validation steps:

1. Create two clusters, each with a cluster label of fleet=true wait for them to become Active.
2. Create a cluster group that selects for the label.
3. Create a GitRepo resource targeting (deploying to) the cluster group. Wait for both clusters to become ready in the UI.
4. After everything is in sync and ready, remove the cluster label from one of the clusters (the cluster group selector should only match a single cluster now).
5. Confirmed that once I remove the label from one of the clusters, the cluster group is reconciled -- meaning it's updated and now only lists 1 cluster in the group.
   Note: I did notice, though, that on the GitRepos page, it still mentions 2 clusters ready, which is not true. I filed a separate bug here for this: https://github.com/rancher/dashboard/issues/4564