add document for windows support (#1515)

README.md

@@ -29,6 +29,7 @@ The Kube-OVN community is waiting for your participation!
 - **TroubleShooting Tools**: Handy tools to diagnose, trace, monitor and dump container network traffic to help troubleshoot complicate network issues.
 - **Prometheus & Grafana Integration**: Exposing network quality metrics like pod/node/service/dns connectivity/latency in Prometheus format.
 - **ARM Support**: Kube-OVN can run on x86_64 and arm64 platforms.
+- **Windows Support**: Kube-OVN can run on Windows worker nodes.
 - **Subnet Isolation**: Can configure a Subnet to deny any traffic from source IP addresses not within the same Subnet. Can whitelist specific IP addresses and IP ranges.
 - **Network Policy**: Implementing networking.k8s.io/NetworkPolicy API by high performance ovn ACL.
 - **DualStack IP Support**: Pod can run in IPv4-Only/IPv6-Only/DualStack mode.
@@ -98,6 +99,7 @@ If you want to install Kubernetes from scratch, you can try [kubespray](https://
 - [Performance Tuning](docs/performance-tuning.md)
 - [Cilium Integration](docs/IntegrateCiliumIntoKubeOVN.md)
 - [F5 CES Integration](https://github.com/f5devcentral/container-egress-service/wiki)
+- [Windows Support](docs/windows-support.md)
 
 ## Contribution
 We are looking forward to your PR!

dist/windows/PrepareNode.ps1

@@ -0,0 +1,128 @@
+<#
+.SYNOPSIS
+Assists with preparing a Windows VM prior to calling kubeadm join
+
+.DESCRIPTION
+This script assists with joining a Windows node to a cluster.
+- Downloads Kubernetes binaries (kubelet, kubeadm) at the version specified
+- Registers kubelet as an nssm service. More info on nssm: https://nssm.cc/
+
+.PARAMETER KubernetesVersion
+Kubernetes version to download and use
+
+.PARAMETER ContainerRuntime
+Container that Kubernetes will use. (Docker or containerD)
+
+.EXAMPLE
+PS> .\PrepareNode.ps1 -KubernetesVersion v1.19.3 -ContainerRuntime containerD
+
+#>
+
+Param(
+    [parameter(Mandatory = $true, HelpMessage="Kubernetes version to use")]
+    [string] $KubernetesVersion,
+    [parameter(HelpMessage="Container runtime that Kubernets will use")]
+    [ValidateSet("containerD", "Docker")]
+    [string] $ContainerRuntime = "Docker"
+)
+$ErrorActionPreference = 'Stop'
+
+function DownloadFile($destination, $source) {
+    Write-Host("Downloading $source to $destination")
+    curl.exe --silent --fail -Lo $destination $source
+
+    if (!$?) {
+        Write-Error "Download $source failed"
+        exit 1
+    }
+}
+
+if ($ContainerRuntime -eq "Docker") {
+    if (-not(Test-Path "//./pipe/docker_engine")) {
+        Write-Error "Docker service was not detected - please install start Docker before calling PrepareNode.ps1 with -ContainerRuntime Docker"
+        exit 1
+    }
+} elseif ($ContainerRuntime -eq "containerD") {
+    if (-not(Test-Path "//./pipe/containerd-containerd")) {
+        Write-Error "ContainerD service was not detected - please install and start containerD before calling PrepareNode.ps1 with -ContainerRuntime containerD"
+        exit 1
+    }
+}
+
+if (!$KubernetesVersion.StartsWith("v")) {
+    $KubernetesVersion = "v" + $KubernetesVersion
+}
+Write-Host "Using Kubernetes version: $KubernetesVersion"
+$global:Powershell = (Get-Command powershell).Source
+$global:PowershellArgs = "-ExecutionPolicy Bypass -NoProfile"
+$global:KubernetesPath = "$env:SystemDrive\k"
+$global:StartKubeletScript = "$global:KubernetesPath\StartKubelet.ps1"
+$global:NssmInstallDirectory = "$env:ProgramFiles\nssm"
+$kubeletBinPath = "$global:KubernetesPath\kubelet.exe"
+
+mkdir -force "$global:KubernetesPath"
+$env:Path += ";$global:KubernetesPath"
+[Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::Machine)
+
+DownloadFile $kubeletBinPath https://dl.k8s.io/$KubernetesVersion/bin/windows/amd64/kubelet.exe
+DownloadFile "$global:KubernetesPath\kubeadm.exe" https://dl.k8s.io/$KubernetesVersion/bin/windows/amd64/kubeadm.exe
+
+if ($ContainerRuntime -eq "Docker") {
+    # Create host network to allow kubelet to schedule hostNetwork pods
+    # NOTE: For containerd the 0-containerd-nat.json network config template added by
+    # Install-containerd.ps1 joins pods to the host network.
+    Write-Host "Creating Docker host network"
+    docker network create -d nat host
+} elseif ($ContainerRuntime -eq "containerD") {
+    DownloadFile "c:\k\hns.psm1" https://github.com/Microsoft/SDN/raw/master/Kubernetes/windows/hns.psm1
+    Import-Module "c:\k\hns.psm1"
+    # TODO(marosset): check if network already exists before creatation
+    New-HnsNetwork -Type NAT -Name nat
+}
+
+mkdir -force C:\var\log\kubelet
+mkdir -force C:\var\lib\kubelet\etc\kubernetes
+mkdir -force C:\etc\kubernetes\pki
+New-Item -path C:\var\lib\kubelet\etc\kubernetes\pki -type SymbolicLink -value C:\etc\kubernetes\pki\
+
+$StartKubeletFileContent = '$FileContent = Get-Content -Path "/var/lib/kubelet/kubeadm-flags.env"
+$global:KubeletArgs = $FileContent.TrimStart(''KUBELET_KUBEADM_ARGS='').Trim(''"'')
+
+$global:containerRuntime = {{CONTAINER_RUNTIME}}
+
+if ($global:containerRuntime -eq "Docker") {
+    $netId = docker network ls -f name=host --format "{{ .ID }}"
+
+    if ($netId.Length -lt 1) {
+    docker network create -d nat host
+    }
+}
+
+$cmd = "C:\k\kubelet.exe $global:KubeletArgs --cert-dir=$env:SYSTEMDRIVE\var\lib\kubelet\pki --config=/var/lib/kubelet/config.yaml --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --hostname-override=$(hostname) --pod-infra-container-image=`"mcr.microsoft.com/oss/kubernetes/pause:3.6`" --enable-debugging-handlers --cgroups-per-qos=false --enforce-node-allocatable=`"`" --network-plugin=cni --resolv-conf=`"`" --log-dir=/var/log/kubelet --logtostderr=false --image-pull-progress-deadline=20m"
+
+Invoke-Expression $cmd'
+$StartKubeletFileContent = $StartKubeletFileContent -replace "{{CONTAINER_RUNTIME}}", "`"$ContainerRuntime`""
+Set-Content -Path $global:StartKubeletScript -Value $StartKubeletFileContent
+
+Write-Host "Installing nssm"
+mkdir -Force $global:NssmInstallDirectory
+DownloadFile nssm.zip https://k8stestinfrabinaries.blob.core.windows.net/nssm-mirror/nssm-2.24.zip
+tar C $global:NssmInstallDirectory -xvf .\nssm.zip --strip-components 2 */win64/*.exe
+Remove-Item -Force .\nssm.zip
+
+$env:path += ";$global:NssmInstallDirectory"
+$newPath = "$global:NssmInstallDirectory;" +
+[Environment]::GetEnvironmentVariable("PATH", [EnvironmentVariableTarget]::Machine)
+
+[Environment]::SetEnvironmentVariable("PATH", $newPath, [EnvironmentVariableTarget]::Machine)
+
+Write-Host "Registering kubelet service"
+nssm install kubelet $global:Powershell $global:PowershellArgs $global:StartKubeletScript
+
+if ($ContainerRuntime -eq "Docker") {
+    nssm set kubelet DependOnService docker
+} elseif ($ContainerRuntime -eq "containerD") {
+    nssm set kubelet DependOnService containerd
+}
+
+New-NetFirewallRule -Name kubelet -DisplayName 'kubelet' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250

docs/windows-support.md

@@ -0,0 +1,106 @@
+# Windows Support
+
+Since version 1.10, Kube-OVN supports running on a cluster with a mixture of Linux and Windows nodes.
+
+## Known Limitations
+
+1. Kube-OVN must be installed without SSL, so you need to set `ENABLE_SSL` to `false` if you are using installation script;
+2. `kube-proxy` is not supported currently, so Kubernetes services cannot be accessed from or through Windows nodes;
+3. Neither single-stack "IPv6-only" networking nor dual-stack IPv4/IPv6 networking is supported;
+4. Pod/Workload QoS is not supported;
+5. Windows nodes don't support dynamic subnet creation/deletion, you must have all subnets created before deploying Kube-OVN on the Windows nodes;
+6. Windows nodes don't support multiple provider networks, the only provider network MUST be name `provider` if you are using underlay networking;
+7. Windows nodes don't support dynamic tunnel interface configuration;
+8. Windows nodes don't support dynamic provider interface configuration.
+
+## Prerequisites
+
+1. Kubernetes server must be at or later than version 1.17;
+2. Only Windows Server 2019 with 64-bit x86_64 architecture is supported;
+3. Obtain a Windows Server 2019 license (or higher) in order to configure the Windows node that hosts Windows containers;
+4. You must have KB4489899 installed to make VXLAN/Overlay networking work;
+5. Install Hyper-V with management tools on the Windows node.
+
+## Installation
+
+### Joining a Windows Worker Node
+
+#### Install Container Runtime
+
+> Currently only Docker Engine is supported.
+
+Install the Containers feature by running the following command in PowerShell:
+
+```ps1
+Install-WindowsFeature -Name containers
+```
+
+Install Docker following the official document - [Install Docker Engine - Enterprise on Windows Servers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=Windows-Server#install-docker).
+
+#### Install kubeadm and kubelet
+
+Run [PrepareNode.ps1](../dist/windows/PrepareNode.ps1) on the Windows node to install `kubeadm` and `kubelet`. Here is an example:
+
+```ps1
+.\PrepareNode.ps1 -KubernetesVersion v1.22.9 -ContainerRuntime Docker
+```
+
+#### Join the Windows Node
+
+Use the command that was given to you when you ran kubeadm init on a control plane host. If you no longer have this command, or the token has expired, you can run kubeadm token create --print-join-command (on a control plane host) to generate a new token and join command.
+
+Then you should be able to view the Windows node in your cluster by running:
+
+```sh
+kubectl get nodes -o wide
+```
+
+### Install Kube-OVN
+
+Download Windows package and extract it.
+
+#### Install Open vSwitch for Hyper-V
+
+Turn on `TESTSIGNING` boot option or `Disable Driver Signature Enforcement` during boot. The following commands can be used:
+
+```ps1
+bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS
+bcdedit /set TESTSIGNING ON
+bcdedit /set nointegritychecks ON
+```
+
+Then you can open the OpenvSwitch.msi to install Open vSwitch.
+
+You can verify the installation by querying the service status:
+
+```ps1
+PS > Get-Service | findstr ovs
+Running  ovsdb-server  Open vSwitch DB Service
+Running  ovs-vswitchd  Open vSwitch Service
+```
+
+#### Install OVN and Kube-OVN
+
+Execute the `install.ps1` to install OVN and Kube-OVN on the Windows node. Here is an example:
+
+```ps1
+.\install.ps1 -KubeConfig C:\k\admin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10.96.0.0/12
+```
+
+For more available parameters, please refer [install.ps1](../dist/windows/install.ps1).
+
+By default, Kube-OVN uses the network adapter hosting the node IP as the tunnel interface. If you want to use a custom adapter, you can add a node annotation, such as `ovn.kubernetes.io/tunnel_interface=Ethernet1`, before installing Kube-OVN.
+
+## Known Issues
+
+1. Hyper-V cannot be installed on the Windows node due to the lack of virtualization capabilities required by Hyper-V. You can install Hyper-V with the following commands:
+
+```ps1
+Install-WindowsFeature containers
+Install-WindowsFeature Hyper-V-Powershell
+dism /online /enable-feature /featurename:Microsoft-Hyper-V /all /NoRestart
+dism /online /disable-feature /featurename:Microsoft-Hyper-V-Online /NoRestart
+Restart-Computer
+```
+
+1. Pod running on a Windows node is stuck at `Terminating` state and cannot be deleted. You can reboot the Windows nodes to solve the problem.