Support Traefik as ingress for RKE2

[caroline-suse-rancher]

Some users would like the option to use Traefik as ingress. We do offer this on k3s, but it is now being asked for on RKE2. This would be an addition to our current supported ingress of nginx.

Related Jira ticket is SURE-8403

[manuelbuil]

It would be good to understand the reasons why somebody might choose one over the other to add it in the docs

[mdrahman-suse]

Validated on RC v1.30.3-rc5+rke2r1

Testing

Scenarios with flag ingress-controller: traefik / ingress-controller: traefik,ingress-nginx

• HA cluster
• New cluster
• Upgrade cluster
• Air-gapped cluster
• CIS enabled cluster
• Rancher deployed on cluster
• Dual-stack cluster
• Cluster with Windows as agent

Issues found:

• Airgap: rke2-traefik Image is missing from airgap tarball #6438
• CIS enabled: CIS enabled cluster does not set default-network policies to traefik when ingress-controller: traefik  #6449

[AaronDewes]

I could not find the ingress-controller setting in the documentation, apart from the ADR. Will this setting be added to the documentation with available options?