Release Note: Issues with RKE1 v1.23.x dual-stack for Windows

[rosskirkpat]

Issue:

• flannel might support windows dual stack in 0.17, however
  • support would only be for l2bridge (which we advise customers not to use)
  • windows does not support overlay dual stack
  • windows does not support overlay ipv6 only

Solution:

• bump flannel to 0.17 for windows
• switch from upstream containernetworking 0.9.1 flannel cni-plugin to the flannel-io flannel-cni plugin for linux and windows
• set the flannel config for KDM for all default rke1 windows versions in 2.6.5 to disable ipv6 if windows_prefered_cluster is set to true

https://kubernetes.io/docs/concepts/services-networking/dual-stack

https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#ipv6-networking

[sirredbeard]

Due to lack of upstream support, any solution for dual stack on RKE1 would be experimental and unsupported.

[rosskirkpat]

Here is a summary of the current State of Windows IPv6 and dual-stack.

Microsoft:

Windows Server appears to support dual-stack for overlay networks per microsoft/Windows-Containers#82 (comment)

Upstream Kubernetes:

dual-stack support for l2bridge/host-gateway only. overlay is specifically called out as unsupported.

https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#ipv6-networking

Kubernetes on Windows does not support single-stack "IPv6-only" networking. However, dual-stack IPv4/IPv6 networking for pods and nodes with single-family services is supported.
  
You can use IPv4/IPv6 dual-stack networking with l2bridge networks. See [configure IPv4/IPv6 dual stack](https://kubernetes.io/docs/concepts/services-networking/dual-stack#configure-ipv4-ipv6-dual-stack) for more details.

Note: Overlay (VXLAN) networks on Windows do not support dual-stack networking.

Flannel:

Flannel supports dual-stack and ipv6. It has not been validated that dual-stack works on Windows hosts but the current expectation is that it should be functional.

Calico:

ipv6/dual-stack for Windows is listed as not supported per Calico

Conclusion:

Without upstream support of dual-stack for overlay networks, we are severely limited in what we can offer for RKE1 Windows. Coupling that with the fact that RKE1 Windows has a remaining shelf life of approximately 4 months (due to Docker EE being removed in Sept. 2022 from Windows Server), we should be pushing all Windows workloads to run on RKE2.

Proposed Solution:

Keep kubernetes >=1.23.x as experimental/tech-preview for RKE1 Windows clusters and specifically note that dual-stack is not supported in our documentation and UI. Refocus all efforts relating to IPv6 and dual-stack on Windows to RKE2 and Calico.

We could even take it a step further and block 1.23 provisioning of Windows clusters on RKE1 due to the state of dual stack upstream. In the Rancher UI and in the RKE1 terraform provider, we would add a check to see if the kubernetes version is >=1.23.x and if so, do not allow the windows_prefered_cluster value to be set to true. This would effectively block upgrading to or provisioning new RKE1 1.23 Windows clusters.

[rosskirkpat]

Release note to be added:

Dual-stack and IPv6-only support for RKE1 clusters using the Flannel CNI will be experimental starting in v1.23.x. windows#165.

[sirredbeard]

Release noted in 2.6.5.

Moving to 2.6.6 to continue monitoring dual-stack situation on Windows.

[phillipsj]

RKE1 for Windows is going to be EOL