add a webhook check for the agent-tls-mode setting

[samjustus]

make sure that it can't be enabled unless the clusters have all reported they can accept the strict setting.

[maxsokolovsky]

QA Testing

Root cause

The webhook doesn't validate changes to the new agent-tls-mode setting's value and default fields.

What was fixed, or what changes have occurred

If agent-tls-mode has default or value updated from system-store to strict, then all non-local clusters must
have a status condition AgentTlsStrictCheck set to True, unless the new setting has an overriding
annotation cattle.io/force=true.

Areas or cases that should be tested

What areas could experience regressions?

Steps

1. Get a Rancher instance meant to be used with this version of the webhook. Configure it to use system-store as the
   value of the agent-tls-mode setting. To do this, start Rancher with an env var CATTLE_AGENT_TLS_MODE set
   to system-store.
2. Provision two downstream clusters.
3. Note the v3.Clusters' conditions in status. AgentTlsStrictCheck should be True.
4. Set the condition to False or anything else for one of them. Don't do this on the local cluster, as the webhook
   ignores its condition for this check.
5. Try changing the value of the agent-tls-mode setting to strict.
6. Observe an error from the webhook.
7. Try this again, but also set the overriding annotation cattle.io/force=true on the setting.
8. Ensure the webhook issues no errors, and the setting's been updated.

[brudnak]

✅ PASSED

Validation Environment

Component	Version / Type
Rancher version starting	v2.9-3c4ccdc5bc9fde3510089153b5ad58fdbe604880-head
Rancher version upgraded	not applicable
Rancher commit link	3c4ccdc
Installation option	Helm (high availability)
RKE binary version used	v1.6.0-rc8
If Helm Chart k8s cluster	v1.30.2
Cert Details	external tls aws acm
Docker version	20.10.7, build f0df350
Helm version	v2.16.8-rancher2
Downstream cluster type	Linode k3s
Downstream K8s version	v1.30.2+k3s2
Authentication providers enabled	local
Logged in user role	admin
Browser type	Google Chrome
Browser version	126.0.6478.127 (Official Build) (x86_64)
Dashboard	master 102f1e6
Webhook version	rancher/rancher-webhook:v0.5.0-rc13

🏗️ Rancher Installation Details... CLICK TO EXPAND! ⬅️

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.crds.yaml

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.15.0

helm install rancher rancher-alpha/rancher --devel \
  --namespace cattle-system \
  --set hostname=$URL_VAR \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=$EMAIL_VAR\
  --set letsEncrypt.ingress.class=nginx \
  --set bootstrapPassword=$PW_VAR \
  --set rancherImage=rancher/rancher \
  --set rancherImageTag=v2.9-head \
  --version 2.9.0-alpha7 \
  --set global.cattle.psp.enabled=false \
  --set agentTLSMode=system-store \
  --set privateCA=true

https://letsencrypt.org/certificates/ >>> ISRG Root X1 >>> pem

kubectl -n cattle-system create secret generic tls-ca \
  --from-file=cacerts.pem=./cacerts.pem

---

🧪 Test Cases

#	Priority	Description & Link	PASS/FAIL
1	P1	set agent-tls-mode setting during helm install to system-store	✅ PASS
2	P1	set agent-tls-mode setting during helm install to strict	✅ PASS
3	P0	Changing agent-tls-mode setting with kubectl	✅ PASS
4	P0	Changing agent-tls-mode setting with the UI	✅ PASS
5	P1	Adding force annotation using kubectl, change setting with kubectl	✅ PASS
6	P1	Adding force annotation using kubectl, change setting with the UI	✅ PASS
7	P1	Upgrade	✅ PASS
8	P1	Set tls-mode Strict with multiple downstream AgentTlsStrictCheck=True	✅ PASS

🚨 8 test cases... CLICK TO EXPAND! (For table links to work) ⬅️

1 / set agent-tls-mode setting during helm install to system-store Status: ✅ PASS

🔺 back to top

Test 1 details... Click to expand

Test Steps for Validation

1. Install Rancher via helm

2. Include this set:

   • --set agentTLSMode=system-store

3. Once Rancher is setup check that status of the setting via kubectl

   kubectl get setting agent-tls-mode -o yaml

4. You should see the setting as system-store

✅ Expected Outcome

For system-store to be set

✅ Actual Outcome

Value system-store was set

---

2 / set agent-tls-mode setting during helm install to strict Status: ⏸️ NOT TESTED YET

🔺 back to top

Test 2 details... Click to expand

Test Steps for Validation

1. Install Rancher via helm

2. Include this set:

   • --set agentTLSMode=strict

3. Once Rancher is setup check that status of the setting via kubectl

   kubectl get setting agent-tls-mode -o yaml

4. You should see the setting as strict

✅ Expected Outcome

For strict to be set

✅ Actual Outcome

Strict was correctly set

---

3 / Changing agent-tls-mode setting with kubectl Status: ✅ PASS

🔺 back to top

Test 3 details... Click to expand

Test Steps for Validation

1. Create a rancher with --set agentTLSMode=system-store

2. Create a downstream cluster

3. Ensure this downstream cluster has AgentTlsStrictCheck set to false

   • This can be checked by going to cluster management > edit the downstream cluster as yaml > check under status at the bottom for AgentTlsStrictCheck

     status: 'False'
     type: AgentTlsStrictCheck

4. Now try changing the agent-tls-mode setting to strict using kubectl

   kubectl edit setting agent-tls-mode

5. Change the value from system-store to strict

   • This should be blocked by webhook since at least 1 downstream cluster has AgentTlsStrictCheck set to 'False'

✅ Expected Outcome

For webhook to block this change to the agent-tls-mode setting

✅ Actual Outcome

Getting the expected error:

error: settings.management.cattle.io "agent-tls-mode" could not be patched: admission webhook "rancher.cattle.io.settings.management.cattle.io" denied the request: value.default: Forbidden: AgentTlsStrictCheck condition of cluster c-m-c7bshql9 isn't 'True'

---

4 / Changing agent-tls-mode setting with the UI Status: ✅ PASS

🔺 back to top

Test 4 details... Click to expand

Test Steps for Validation

1. Create a rancher with --set agentTLSMode=system-store
2. Create a downstream cluster
3. Ensure this downstream cluster has AgentTlsStrictCheck set to false

   • This can be checked by going to cluster management > edit the downstream cluster as yaml > check under status at the bottom for AgentTlsStrictCheck

   • If it's not already set as False, you can change it to False and save the yaml

     status: 'False'
     type: AgentTlsStrictCheck

4. Now try changing the agent-tls-mode setting to strict using the Rancher UI
5. This can be accessed from starting at dashboard/home > click Global Settings (bottom left globe icon) > Settings > click the ellipsis (three dots) at the setting agent-tls-mode > Edit Setting > change the value to Strict > Save
6. This should be blocked by webhook as long as at least one downstream cluster has AgentTlsStrictCheck set to false

✅ Expected Outcome

The UI action should be blocked by webhook

✅ Actual Outcome

Getting the expected error of:

admission webhook "rancher.cattle.io.settings.management.cattle.io" denied the request: value.default: Forbidden: AgentTlsStrictCheck condition of cluster c-m-c7bshql9 isn't 'True'

---

5 / Adding force annotation using kubectl, change setting with kubectl Status: ✅ PASS

🔺 back to top

Test 5 details... Click to expand

Test Steps for Validation

1. Create a rancher with --set agentTLSMode=system-store

2. Create a downstream cluster

3. Ensure this downstream cluster has AgentTlsStrictCheck set to false

   • This can be checked by going to cluster management > edit the downstream cluster as yaml > check under status at the bottom for AgentTlsStrictCheck

   • If it's not already set as False, you can change it to False and save the yaml

     status: 'False'
     type: AgentTlsStrictCheck

4. Now try changing the agent-tls-mode setting to strict using the Rancher UI

5. This can be accessed from starting at dashboard/home > click Global Settings (bottom left globe icon) > Settings > click the ellipsis (three dots) at the setting agent-tls-mode > Edit Setting > change the value to Strict > Save

6. This should be blocked by webhook as long as at least one downstream cluster has AgentTlsStrictCheck set to false

7. Now edit the agent-tls-mode setting using kubectl > kubectl edit setting agent-tls-mode

8. Add this annotation under metadata:

   metadata:
     annotations:
       cattle.io/force: "true"

9. After this annotation is added try changing the agent-tls-setting to strict using kubectl

10. Now even if there is a downstream cluster that has AgentTlsStrictCheck set to false the webhook will allow it now

✅ Expected Outcome

The webhook should allow the setting to be changed now, even with a downstream cluster that has AgentTlsStrictCheck set to false

✅ Actual Outcome

Successful edit with annotation added

---

6 / dding force annotation using kubectl, change setting with the UI Status: ✅ PASS

🔺 back to top

Test 6 details... Click to expand

Test Steps for Validation

1. Create a rancher with --set agentTLSMode=system-store

2. Create a downstream cluster

3. Ensure this downstream cluster has AgentTlsStrictCheck set to false

   • This can be checked by going to cluster management > edit the downstream cluster as yaml > check under status at the bottom for AgentTlsStrictCheck

   • If it's not already set as False, you can change it to False and save the yaml

     status: 'False'
     type: AgentTlsStrictCheck

4. Now try changing the agent-tls-mode setting to strict using the Rancher UI

5. This can be accessed from starting at dashboard/home > click Global Settings (bottom left globe icon) > Settings > click the ellipsis (three dots) at the setting agent-tls-mode > Edit Setting > change the value to Strict > Save

6. This should be blocked by webhook as long as at least one downstream cluster has AgentTlsStrictCheck set to false

7. Now edit the agent-tls-mode setting using kubectl > kubectl edit setting agent-tls-mode

8. Add this annotation under metadata:

   metadata:
     annotations:
       cattle.io/force: "true"

9. After this annotation is added try changing the agent-tls-setting to strict using the Rancher UI

10. Now even if there is a downstream cluster that has AgentTlsStrictCheck set to false the webhook will allow it now

✅ Expected Outcome

The webhook should allow the setting to be changed now, even with a downstream cluster that has AgentTlsStrictCheck set to false

✅ Actual Outcome

Successful edit with annotation added

---

7 / Upgrade Status: ✅ PASS

🔺 back to top

Test 7 details... Click to expand

Test Steps for Validation

1. Start with Rancher 2.8.5
2. Upgrade Rancher to 2.9-head
3. Add this env var to the rancher deployment > CATTLE_AGENT_TLS_MODE > value: system-store
4. Create a downstream cluster
5. Ensure this downstream cluster has AgentTlsStrictCheck set to false

   • This can be checked by going to cluster management > edit the downstream cluster as yaml > check under status at the bottom for AgentTlsStrictCheck

     status: 'False'
     type: AgentTlsStrictCheck

6. Now try changing the agent-tls-mode setting to strict using the Rancher UI
7. This can be accessed from starting at dashboard/home > click Global Settings (bottom left globe icon) > Settings > click the ellipsis (three dots) at the setting agent-tls-mode > Edit Setting > change the value to Strict > Save
8. This should be blocked by webhook as long as at least one downstream cluster has AgentTlsStrictCheck set to false

✅ Expected Outcome

The UI action should be blocked by webhook

✅ Actual Outcome

Action is blocked by webhook

---

8 / Set tls-mode Strict with multiple downstream AgentTlsStrictCheck=True Status: ✅ PASS

🔺 back to top

Test 8 details... Click to expand

Test Steps for Validation

1. Create a rancher with --set agentTLSMode=system-store

2. Create two downstream clusters

3. Ensure both downstream clusters have AgentTlsStrictCheck set to 'True'

   • This can be checked by going to cluster management > edit the downstream cluster as yaml > check under status at the bottom for AgentTlsStrictCheck

4. Now try changing the agent-tls-mode setting to strict using kubectl

   kubectl edit setting agent-tls-mode

5. Change the value from system-store to strict

   • This should be allowed by webhook since both downstream cluster have AgentTlsStrictCheck set to 'True'

✅ Expected Outcome

For the change to be successful since both downstream clusters have AgentTlsStrictCheck set to 'True'

✅ Actual Outcome

The change was successful since both downstream clusters have AgentTlsStrictCheck set to 'True'

---