[BUG][CAPR] rancher-provisioning-capi-patch-sa job failing due to lack of exclusion from PSA enforcement

[thaneunsoo]

Rancher Server Setup

• Rancher version: v2.7-head 787c056
• Installation option (Docker install/Helm Chart): Helm
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):

Information about the Cluster

• Kubernetes version: v1.26.7+rke2r1
• Cluster Type (Local/Downstream):
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):
    AWS node driver

User Information

• What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
  • If custom, define the set of permissions: Admin

Describe the bug

Downstream cluster is unable to provision and cluster is stuck at waiting for viable init node. I don't see the machines getting created on the AWS portal and the latest log message is Creating server [fleet-default/auto-aws-kkswl-pool0-01d54a4c-4kr6x] of kind (Amazonec2Machine) for machine auto-aws-kkswl-pool0-74f9b78f74xcl6q9-fjnch in infrastructure provider

To Reproduce

1. Provision RKE2 AWS node driver cluster

Result
Cluster gets stuck while provisioning with the status waiting for viable init node

Expected Result

Cluster is able to provision successfully

Screenshots

[slickwarren]

This is not resolved for us.
We looked into our automation and nothing looks out of the ordinary. Here's what we know:

• this only fails for rke1 local clusters (rke2 on the same 1.26.7 k8s version deploys just fine)
• if you manually set the rancher-provisioning-capi-patch-sa 's namespace to enforce:psa:privileged and the cattle-fleet-local-system to enforce:psa:privileged , both rancher-provisioning-capi-patch-sa job and the fleet-agent job are successful
• without doing this, the cluster never resolves and is basically unusable

[slickwarren]

tested versions: (all using rke v1.4.8)
(Tim): v2.7.7-rc4
Caleb: v2.7-head and v2.8-head

[aiyengar2]

I would suspect that the issue here is that changes that were made to introduce the new CAPI chart may not have been coordinated with changes to FeatureAppNS list, which should track all feature chart / system namespaces to exclude PSA enforcements from them.

cattle-fleet-local-system and cattle-provisioning-capi-system (the CAPI Provisioning Namespace that is set as the chart's release namespace) do not appear to be in this list, so the fix should simply be to add those namespaces to the list.

[slickwarren]

using these namspaces in the pod security configuration, I was able to resolve the issue for fleet. It appears that cattle-provisioning-capi-system is missing from this list though.

[snasovich]

/forwardport v2.8.0

[thaneunsoo]

Test Environment:

Rancher version: v2.7-head 1d25044
Rancher cluster type: HA
Docker version: 20.10

Downstream cluster type: RKE2 node driver cluster

---

Testing:

Tested this issue with the following steps:

1. Provision RKE2 AWS node driver cluster

Result
The job no longer fails but the cluster still isn't able to come up.

@Oats87 Should I close this ticket now that the job isn't faililng and open a different issue? The cluster went to delete after about 5 minutes and is now just stuck in deleting

[thaneunsoo]

My bad @Oats87 this was an issue with our jenkins job which was fixed here #42743

Closing this issue as fixed.