[Hosted Rancher] Fleet infinitely creates cluster management secrets when no system namespace exists

[nickgerace]

This is a child issue for #34746

SURE-3301

[SheilaghM]

Since #34746 is a release-blocker, so is this one.

[nickgerace]

QA Validation

This is a child issue for a performance bug.
We just need to ensure that there is not a regression in performance or functionality.

1. Create a downstream cluster
2. Install a GitRepo CR example: https://github.com/rancher/fleet-examples (multi-cluster/helm is a good one)
3. Ensure that the following logs (or similar) do not flood in the leader Rancher pod:

2021/08/31 21:26:18 [INFO] Creating secret [cattle-webhook-tls] into namespace [fleet-local]
2021/08/31 21:26:18 [INFO] Creating secret [cattle-webhook-tls] into namespace [cattle-fleet-clusters-system]
2021/08/31 21:26:18 [INFO] Creating secret [cattle-webhook-ca] into namespace [fleet-default]
2021/08/31 21:26:18 [ERROR] error syncing 'cattle-prometheus/cluster-alerting.v2': handler helm-app: failed to delete cattle-prometheus/cluster-alerting catalog.cattle.io/v1, Kind=App for helm-app cattle-prometheus/cluster-alerting.v2: apps.catalog.cattle.io "cluster-alerting" not found, requeuing
2021/08/31 21:26:18 [INFO] Creating secret [helm-operation-bjjrz] into namespace [cluster-fleet-default-c-7jqbv-5f62cb7a1a39]
2021/08/31 21:26:18 [INFO] Creating secret [sh.helm.release.v1.rancher-operator.v3] into namespace [fleet-default]
2021/08/31 21:26:18 [INFO] Creating secret [sh.helm.release.v1.fleet-crd.v4] into namespace [cluster-fleet-local-local-1a3d67d0a899]
2021/08/31 21:26:18 [INFO] Creating secret [sh.helm.release.v1.rancher-webhook.v1] into namespace [cluster-fleet-default-c-7jqbv-5f62cb7a1a39]

[nickgerace]

Please note, when validating, you may hit: #34746

Degraded performance is expected due to the parent issue. However, the aforementioned logs related to infinite cluster management secrets being created should no longer appear.

[izaac]

Reproduced on Hosted Rancher v2.5.9 and upgraded Tenant Rancher v2.5.9 -> v2.6.0

Steps:

• Create k3s HA cluster and install Rancher in it. (Super) - v2.5.9
• Create a second k3s HA cluster and import it in the Super Rancher v2.5.9
• Install Rancher in the imported k3s HA cluster (Tenant)
• Login in the Tenant Rancher UI
• Create a Downstream RKE1 cluster
• Install a GitRepo CR example: https://github.com/rancher/fleet-examples (multi-cluster/helm is a good one) - Target all Clusters
• Upgrade the Tenant Rancher to v2.6.0
• Check the Tenant Rancher server logs

2021/09/21 23:49:16 [ERROR] error syncing 'rancher-operator-system': handler namespace-logging-configsysncer: can't find system project, requeuing
2021/09/21 23:49:16 [INFO] Creating secret [sh.helm.release.v1.rancher.v2] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [cert-manager-webhook-ca] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [sh.helm.release.v1.rancher-operator.v1] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [sh.helm.release.v1.rancher-webhook.v1] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [sh.helm.release.v1.rancher-operator-crd.v1] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [mc-m-d2fd4] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [serving-cert] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [cattle-webhook-ca] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [ip-172-31-13-145.node-password.k3s] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [sh.helm.release.v1.rancher-operator-crd.v1] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [ip-172-31-14-0.node-password.k3s] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [mc-m-8x8wk] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [helm-operation-bvtn9] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [k3s-serving] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [helm-operation-nf6ht] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [helm-operation-ztbnd] into namespace [fleet-local]
2021/09/21 23:49:16 [INFO] Creating secret [helm-operation-9zmfh] into namespace [fleet-local]

I also tried this same scenario without upgrading using only v2.6.0 for both Super and Tenant Rancher setups and I wasn't able to reproduce.
I'll work on the fix validation next.

[izaac]

Not able to reproduce in Hosted Rancher v2.5.9 with upgraded Tenant Rancher v2.5.9 -> v2.6-head (09/21/2021) b24724f
Fleet agent on downstream cluster of Tenant Rancher: rancher/fleet-agent:v0.3.7-rc1

Followed these steps.