[RFE] TLS 1.3 Support for Rancher Manager Ingress

[samjustus]

Currently we do not support TLS 1.3 and a customer is asking that we do. Some applications that are part of the cluster could have TLS 1.3 enabled but doing so can end up breaking things.

The business case is that we will eventually have to do this once TLS v1.3 adoption will at some point eclipse TLS v1.2

Versions to be dropped:

TLS 1.0 and TLS 1.1

Support to TLS 1.0 and TLS 1.1 must be dropped. These versions are considered insecure and were deprecated in 2021, see RFC 8996.
The deprecation must happen and be supported in all places and services used by Rancher:
Source code - https://github.com/rancher/rancher/blob/release/v2.7/pkg/tls/base.go#L17-L21.
Single Docker install / Helm install / Ingress - https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/tls-settings.
The configuration must be applied to all TLS services started by Rancher.

Versions to be maintained: TLS 1.2

In the same places listed above.

Versions to be added: TLS 1.3

In the same places listed above.

Notes

Regarding the ciphers to be supported by both TLS 1.2 and 1.3, we should use the compatible list provided by Cloudflare. Cloudfare’s list aligns with Go's ciphers for TLS 1.3. Those ciphers offer a good compromise between client compatibility and security. Old or legacy ciphers must not be supported.
Ingress' default TLS configuration might have to be tailored according to the listed ciphers.
When a PR is ready for review, the Security team will run a TLS scan to confirm that all Rancher's TLS related services/ports do properly support TLS 1.2 and 1.3 and disallow TLS 1.0 and 1.1.
Would be good to get feedback from Product regarding if we should completely remove TLS 1.0 and 1.1 or deprecate them first, giving that this might impact customers (in case we have customers actually using those versions in their environments).
The changes must be release noted.

[samjustus]

SURE-3166

[maxsokolovsky]

QA Testing

Root cause

Rancher doesn't allow setting tls-min-version to 1.3. It also doesn't allow to set any 1.3 ciphers in tls-ciphers.

What was fixed, or what changes have occurred

Rancher's validation of the two settings now allows min version to be 1.3 and enforces 1.3 ciphers if the min version is 1.3.

Areas or cases that should be tested

What areas could experience regressions?

Steps

Run Rancher without changing TLS-related settings (as before)

1. Note that Rancher starts up without errors. Make sure the UI works fine.
2. Inspect the value of the tls-min-version, ensure the value is the default 1.2.

Run Rancher by specifying tls-min-version as 1.3.

1. Note that Rancher fails to start up because the default ciphers are for lower versions of TLS.

Run Rancher by specifying tls-min-version as 1.3 and tis-ciphers as "TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384"

1. Note that Rancher starts up without errors. Make sure the UI works fine.

Run Rancher by specifying one wrong cipher

1. Here is a list of 1.3-specific ciphers. Above that are ciphers for previous versions of TLS.
2. Repeat the previous valid case for 1.3 settings, but add one non-1.3 cipher. All ciphers are comma-separated.
3. Ensure Rancher fails to start up.

[brudnak]

💪 Test Cases

#	Priority	Description & Link	PASS/FAIL
1	P0	Start Rancher with only CATTLE_TLS_MIN_VERSION=1.3 Should Fail	✅ PASS
2	P0	Run Rancher Without Specifying CATTLE_TLS_MIN_VERSION	✅ PASS
3	P0	Run with CATTLE_TLS_MIN_VERSION & CATTLE_TLS_CIPHERS	✅ PASS
4	P0	Run with CATTLE_TLS_MIN_VERSION & Wrong CATTLE_TLS_CIPHERS	✅ PASS
5	P0	Upgrade	✅ PASS
6	P0	Unsupported TLS Versions (TLS 1.0 and TLS 1.1)	✅ PASS
7	P0	Install Rancher without Explicit TLS Version and with TLS 1.3 Ciphers:	✅ PASS
8	P0	Upgrade with CATTLE_TLS_MIN_VERSION=1.3 (Expecting Failure): Attempt to upgrade Rancher with CATTLE_TLS_MIN_VERSION set to 1.3, and ensure that it fails.	✅ PASS
9	P0	Upgrade Without Specifying CATTLE_TLS_MIN_VERSION: Perform an upgrade without CATTLE_TLS_MIN_VERSION and verify that the default value of tls-min-version is correctly set to 1.2.	✅ PASS
10	P0	Upgrade with CATTLE_TLS_MIN_VERSION and Incorrect CATTLE_TLS_CIPHERS: Attempt an upgrade with CATTLE_TLS_MIN_VERSION set to 1.3 and wrong or unsupported CATTLE_TLS_CIPHERS. Verify that the upgrade fails.	✅ PASS

🚨 10 test cases... CLICK TO EXPAND! (For table links to work) ⬅️

1 Start Rancher with only CATTLE_TLS_MIN_VERSION=1.3 Should Fail / Status: ✅ PASS

🔺 back to top

Test 1 details... Click to expand

Test Steps for Validation

1. Attempt to start Rancher on v2.8-head

2. Specify CATTLE_TLS_MIN_VERSION as 1.3

   • -e CATTLE_TLS_MIN_VERSION=1.3

3. SSH onto the instance where Rancher is running

4. Run:

   • docker ps
   • docker logs -f $CONTAINER_ID_GOES_HERE

5. In the logs notice the following:

   2023/10/05 20:58:01 [FATAL] failed to setup TLS listener: unsupported cipher
   

6. As expected, Rancher fails to start

7. Can use if needed: https://github.com/brudnak/linode-docker-cattle

  provisioner "remote-exec" {
    inline = [
      "sudo apt update",
      "sudo curl https://releases.rancher.com/install-docker/20.10.sh | sh",
      "docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged -e CATTLE_BOOTSTRAP_PASSWORD=${var.rancher_bootstrap_password} -e CATTLE_TLS_MIN_VERSION=1.3 rancher/rancher:${var.rancher_instances[count.index].rancher_version} --acme-domain ${random_pet.random_pet[count.index].id}.${var.aws_route53_fqdn}",
    ]
  }

✅ Expected Outcome

For there to be an error message in the logs and Rancher not to start.

✅ Actual Outcome

Error logs to be present and Rancher didn't come up active

---

2 Run Rancher Without Specifying CATTLE_TLS_MIN_VERSION / Status: ✅ PASS

🔺 back to top

Test 2 details... Click to expand

Test Steps for Validation

1. Install Rancher on v2.8-head without setting CATTLE_TLS_MIN_VERSION
2. Ensure functionality
   1. Enable auth provider (GitHub OAuth)
   2. Add two standard users (user1, and user2)
   3. Login as user1
      • Create a downstream RKE1 Linode cluster as user1
      • Create a project
      • Create a namespace
      • Create a deployment
      • Ensure the deployment comes up active
      • Add user2 as a project owner
3. Login as user2 and ensure access to the project
4. Check Rancher logs for anything related to tls version
   1. No errors observed

✅ Expected Outcome

For Rancher to startup and function without issue

✅ Actual Outcome

No issues observed

---

3 Run with CATTLE_TLS_MIN_VERSION & CATTLE_TLS_CIPHERS / Status: ✅ PASS

🔺 back to top

Test 3 details... Click to expand

Test Steps for Validation

1. Install Rancher on 2.8-head and set CATTLE_TLS_MIN_VERSION and CATTLE_TLS_CIPHERS
2. Set CATTLE_TLS_MIN_VERSION=1.3 & CATTLE_TLS_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384
3. Can use: https://github.com/brudnak/linode-docker-cattle

  provisioner "remote-exec" {
    inline = [
      "sudo apt update",
      "sudo curl https://releases.rancher.com/install-docker/20.10.sh | sh",
      "docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged -e CATTLE_BOOTSTRAP_PASSWORD=${var.rancher_bootstrap_password} -e CATTLE_TLS_MIN_VERSION=1.3 -e CATTLE_TLS_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384 rancher/rancher:${var.rancher_instances[count.index].rancher_version} --acme-domain ${random_pet.random_pet[count.index].id}.${var.aws_route53_fqdn}",
    ]
  }

4. Check Rancher logs for any errors
5. Login to Rancher and verify functionality
6. Create two standard users user1, user2
7. Login as user1
   1. Create a downstream Linode K3s cluster
   2. Create a project in the cluster
   3. Create a namespace in the project
   4. Create a deployment in the namespace
   5. Assign user2 as a project-owner

✅ Expected Outcome

No error logs regarding tls and Rancher working as expected.

✅ Actual Outcome

No errors observed in logs regarding tls, Rancher was working as expected.

---

4 Run with CATTLE_TLS_MIN_VERSION & Wrong CATTLE_TLS_CIPHERS / Status: ✅ PASS

🔺 back to top

Test 4 details... Click to expand

Test Steps for Validation

1. Attempt to start Rancher on v2.8-head

2. Specify CATTLE_TLS_MIN_VERSION as 1.3

   • -e CATTLE_TLS_MIN_VERSION=1.3
   • CATTLE_TLS_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

3. SSH onto the instance where Rancher is running

4. Run:

   • docker ps
   • docker logs -f $CONTAINER_ID_GOES_HERE

5. In the logs notice the following:

   2023/10/06 00:26:47 [FATAL] failed to setup TLS listener: unsupported cipher TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, must be one or more of: TLS_AES_128_GCM_SHA256
   TLS_AES_256_GCM_SHA384
   

6. As expected, Rancher fails to start

7. Can use if needed: https://github.com/brudnak/linode-docker-cattle

  provisioner "remote-exec" {
    inline = [
      "sudo apt update",
      "sudo curl https://releases.rancher.com/install-docker/20.10.sh | sh",
      "docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged -e CATTLE_BOOTSTRAP_PASSWORD=${var.rancher_bootstrap_password} -e CATTLE_TLS_MIN_VERSION=1.3 -e CATTLE_TLS_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 rancher/rancher:${var.rancher_instances[count.index].rancher_version} --acme-domain ${random_pet.random_pet[count.index].id}.${var.aws_route53_fqdn}",
    ]
  }

✅ Expected Outcome

Rancher should fail to start and error with a message about needing the correct cipher.

✅ Actual Outcome

Rancher failed to start and was erroring about needing a set list of ciphers.

---

5 Upgrade / Status: ✅ PASS

🔺 back to top

Test 5 details... Click to expand

Test Steps for Validation

1. Start with Rancher on v2.7.7
2. Upgrade Rancher to v2.8-head
3. When upgrading pass the following:
   • CATTLE_TLS_MIN_VERSION=1.3
   • CATTLE_TLS_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384
4. After upgrade check that Rancher is accessible and there are no warnings about TLS ciphers

✅ Expected Outcome

Rancher to be usable after upgrading from v2.7.7 to v2.8-head and providing

• CATTLE_TLS_MIN_VERSION=1.3
• CATTLE_TLS_CIPHERS=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384

✅ Actual Outcome

Rancher comes up active after upgrade and is usable. No warnings seen for TLS cipher.

---

6 Unsupported TLS Versions (TLS 1.0 and TLS 1.1) / Status: ✅ PASS

🔺 back to top

Test 6 details... Click to expand

Test Steps for Validation

1. Pass -e CATTLE_TLS_MIN_VERSION=1.0
   1. Ensure failure
2. Pass -e CATTLE_TLS_MIN_VERSION=1.1
   1. Ensure failure

✅ Expected Outcome

Expected these values to fail

✅ Actual Outcome

Values failed as expected

The values failed

---

7 Install Rancher without Explicit TLS Version and with TLS 1.3 Ciphers: / Status: ✅ PASS

🔺 back to top

Test 7 details... Click to expand

Test Steps for Validation

1. Only pass TLS 1.3 Ciphers
2. Should fail

✅ Expected Outcome

For this fail

✅ Actual Outcome

Failed as expected

---

8 Upgrade with CATTLE_TLS_MIN_VERSION=1.3 (Expecting Failure): Attempt to upgrade Rancher with CATTLE_TLS_MIN_VERSION set to 1.3, and ensure that it fails. / Status: ✅ PASS

🔺 back to top

Test 8 details... Click to expand

Test Steps for Validation

1. Start with Rancher 2.7.8
2. Upgrade to Rancher 2.8-head
3. Only specify tls version and not ciphers

✅ Expected Outcome

Should fail

✅ Actual Outcome

Failed as expected

2023/10/11 00:25:03 [FATAL] failed to setup TLS listener: unsupported cipher TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, must be one or more of: TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256

---

9 Upgrade Without Specifying CATTLE_TLS_MIN_VERSION: Perform an upgrade without CATTLE_TLS_MIN_VERSION and verify that the default value of tls-min-version is correctly set to 1.2. / Status: ✅ PASS

🔺 back to top

Test 9 details... Click to expand

Test Steps for Validation

1. Start with Rancher v2.7.8
2. Upgrade to Rancher v2.8-head
3. Do not specify TLS or Cipher
4. Upgrade is successful
5. TLS default is as expected, 1.2

✅ Expected Outcome

For upgrade to pass without passing tls or ciphers and for default version to be 1.2

✅ Actual Outcome

Upgrade passed without passing tls or ciphers and the default version 1.2

---

10 Upgrade with CATTLE_TLS_MIN_VERSION and Incorrect CATTLE_TLS_CIPHERS: Attempt an upgrade with CATTLE_TLS_MIN_VERSION set to 1.3 and wrong or unsupported CATTLE_TLS_CIPHERS. Verify that the upgrade fails / Status: ✅ PASS

🔺 back to top

Test 10 details... Click to expand

Test Steps for Validation

1. Start with Rancher v2.7.8
2. Upgrade to Rancher b2.8-head
3. Provide min tls as 1.3
4. Provide incorrect values for the ciphers

✅ Expected Outcome

For this to fail

✅ Actual Outcome

Failed as expected

---