Windows container support?

[biiiipy]

Is it possible to add Windows worker nodes to microk8s cluster? Haven't seen this topology mentioned anywhere.
So basically:
microk8s clusters in WSL2 and plain windows worker nodes with windows containers (without running the microk8s on the windows worker nodes, which would be linux containers).

[balchua]

Joining non microk8s nodes to a microk8s cluster you mean?

[biiiipy]

Yes, non-microk8s nodes (plain windows worker nodes) to microk8s clusters running on WSL2.
The goal is to have microk8s cluster running in WSL2, BUT run windows containers (on Windows Server 2019).

[balchua]

From what i can tell, not possible.

[joedborg]

Hi @biiiipy, to get the Linux workers (i.e. MicroK8s) on Windows, we recommend using the Windows installer found here: https://microk8s.io/docs/install-alternatives#heading--windows.

Getting the Windows workers running would be a different challenge, but (I think), if you could get MicroK8s running with the installer, you could enroll manually installed Windows Kubelets; in theory at least. There is no precedence for this, but I'd be interested if you get it working.

[organom]

I've already deployed a native kubernetes cluster working with both linux and windows nodes, but its a real pain, lots of configuration and a huge mess with the windows nodes having to proxy the traffic via the linux nodes.

I recently discovered microk8s and really enjoyed installing a PoC cluster with it, its stable, fast and easy to use. But having windows containers running in the same k8s cluster is, unfortunately, also a need in the project.

From the microk8s side, we have no access to the kubeadm so we can't do kubeadm token list or kubeadm token create $(kubeadm token generate). microk8s add-node will generate something like microk8s join 192.168.1.154:25000/be0f5f9e286f7e81b1b62e45719048a2
From the native node side, we would generally run something like kubeadm join 192.168.1.146:6443 --token 9a1qzb.ggdku08tuu2bzv8h --discovery-token-ca-cert-hash sha256:682dc1573fc3b6ca71f99f19979c99b07b7190248f2ff72a5b6865ad265489ad to join the node to the cluster.

@joedborg how would you enroll a manual windows kubernetes? I tried to search in the documentation but couldn't find anything :'(
Any idea how we could get the needed values out of microk8s in order to try to join the native node with them ?

Edit 16/10/2020 11:31: I found that we can get the certificate from microk8s with microk8s config in the field certificate-authority-data that is base64. So i just decoded it copied it to a file ca.crt and run openssl x509 -in ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2>/dev/null | sha256sum | cut -d' ' -f1 this theoretically gets me the sha256 value of the cert to append to --discovery-token-ca-cert-hash sha256: . So I'm just missing the token, since the one generated by the add-node isn't in the format of the native kubeadm token.

[ktsakalozos]

Hi @organom,

all certs are in /var/snap/microk8s/current/certs and credentials (eg tokens etc) are in /var/snap/microk8s/current/credentials. In case you need to configure any services you will find the services arguments in /var/snap/microk8s/current/args .

Are you following any guide/doc I could look at? How do you install the windows node?

[biiiipy]

@organom great to see someone else in similar situation :)
Also, did I understand correctly, that we don't need dedicated Linux nodes with microk8s on Windows, cause microk8s runs them in Hyper-V Ubuntu VMs (I only tried to go WSL2+Microk8s route)?
So the missing piece is getting windows native nodes (for native windows containers) to join cluster running in microk8s? If that's the case, then this makes microk8s the best solution for Windows centric environments, by a huge margin, IF we can get windows containers working, of course...

[organom]

@biiiipy yes, that is my understanding of it also. On a native kubernetes, you setup the master, Linux nodes and then the windows nodes. Linux nodes are still mandatory because the windows machines use them to proxy the traffic.
On microK8s, there is no need for master (each installation is master and node), so a single machine would suffice. The windows, macos or linux installation in case of microk8s is always the same, a linux node/master running. Would be great if microk8s would add a flag to also boot a windows native node, but since this is probably not an easy feat, my thought was to keep the windows nodes as native kubernetes installation, add the required proxy updates to microk8s and try to somehow join them to the microk8s cluster

@ktsakalozos I followed the native kubernetes installation guide https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/ and https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/ .

As a first step I would be happy to at least be able to join the native windows node into a microk8s cluster, but the token difference (9a1qzb.ggdku08tuu2bzv8h expected from native instead of be0f5f9e286f7e81b1b62e45719048a2 that microk8s gives) is preventing me from testing it, after that there is still the question of the network CNI flannel specific windows configurations (that in the case of microk8s would need to be calico), but one fight at a time ;)

[zejji]

+1 for this. We have legacy .NET Framework containers which can only be run on a native Windows worker, so it would be very helpful to be able to easily spin up such a worker using microk8s!

[joedborg]

We've just finished the documentation for this:

https://discuss.kubernetes.io/t/add-a-windows-worker-node-to-microk8s/13782

[organom]

We've just finished the documentation for this:

https://discuss.kubernetes.io/t/add-a-windows-worker-node-to-microk8s/13782

@joedborg The document looks awesome, but it misses (I believe) the part of the joining the windows node to the microk8s node. How do you do it?

Thanks for the help

[joedborg]

@organom When you start the kubelet service (towards the end), it connects by reading the kubeconfig file you wrote to the Windows node.

[organom]

@joedborg I see, didn't knew kubelet would accept it like that :) I will give it a try next week.
Thank you again for writing this up and letting us know ;)

[joedborg]

@organom yeah, the kubeconfig contains all you need to reach and authenticate with the control plane.

No problem, I'll close this issue. Please feel free to open a new one if you come across anything specific to the documentation.

[organom]

Just a fast update from my side. Worked like a charm.
Following the tutorial I was able to have the windows node joined to the microk8s cluster and run a simple windows test pod (powershell curl of google.com).

Thank you again for making the tutorial available

[matt-psaltis]

Trying to work out if this can be done entirely from a Windows 10 (pro/ent) host. microk8s on WSL2 (ubuntu)
What are the options for the Windows Worker?

• Hyper-V Server 2019 VM?
• Can this work with the Win 10 host directly? I.E. it becomes the worker?

How are other people configuring this?

[organom]

Trying to work out if this can be done entirely from a Windows 10 (pro/ent) host. microk8s on WSL2 (ubuntu)
What are the options for the Windows Worker?

* Hyper-V Server 2019 VM?

* Can this work with the Win 10 host directly? I.E. it becomes the worker?

How are other people configuring this?

There are 2 important concepts to separate here:

• one thing is running microk8s in windows
• other thing is windows nodes vs linux nodes

MicroK8s will run in windows correctly, having the linux node installed and running by default (https://microk8s.io/docs/install-alternatives).

If the intention is to use Windows nodes (in order to run windows pods inside) (https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/) then you need to use Windows Server 2019. Microk8s doesn't run windows nodes, but does accept windows nodes to be joined to an already existing microK8s cluster.

So, answering your question, you can have microk8s installed in the win10 host directly (running the linux node it automatically starts with), and then connect to a VM (hyper-v, virtualbox, vmware player, etc), install there Windows server 2019 and the node side of https://discuss.kubernetes.io/t/add-a-windows-worker-node-to-microk8s/13782 and join it to the cluster.
Just make sure both machines can "see" each other.

In my particular case, I use a 5 VMs setup inside virtualbox with 3 linux (for testing high availability) and 2 Windows Server 2019, this way i can easily pause the whole cluster and get back to it when needed.

Hope it helps :)

[matt-psaltis]

Thank you very much for such a detailed explanation.

I'm definitely focusing on the Windows nodes side of things and given similarities between Windows 10 and Server 2019, I wondered if enough of the components were available to bend that requirement somewhat.

Thanks again!

[FrankAtHexagon]

Has anyone ever seen this error while rolling out a deployment that uses Windows containers?

Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "kubeletwin/pause": Error response from daemon: pull access denied for kubeletwin/pause, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I searched all over and finally found this section from the Calico docs. I essentially ended up having to do:

docker pull mcr.microsoft.com/k8s/core/pause:1.2.0

and then:

docker tag mcr.microsoft.com/k8s/core/pause:1.2.0 kubeletwin/pause

[FrankAtHexagon]

After some more investigation, I think I found the likely cause of the issue. Either it was:

• Installing Docker Desktop on Windows Server instead of Docker EE, or...
• Not enabling Windows containers before the setup on Windows Server

Regardless, I think the documentation makes it implicit that Docker EE is required thanks to this link.

[bangzhuzhu]

@joedborg follow the whole step, but can't join the cluster.
And in the file kubelet.out:
Running kubelet service. Using configured nodename: vm05 DNS: 10.96.0.10 Auto-detecting node IP, looking for interface named like 'vEthernet (Ethernet*' ... Waiting for interface named like 'vEthernet (Ethernet*' ... Waiting for interface named like 'vEthernet (Ethernet*' ...

[bangzhuzhu]

there is a error msg about, hope it helps:
Error from server (NotFound): namespaces "calico-system" not found
Calico running in kube-system namespace

[hassanshamshir]

@organom I have install microk8s cluster successfully on window hyper-V and now want to add window node into microk8s cluster which have by default linux master node. But how can I move config file from linux node to window node using terminal?
Both machines can't communicate with each other I have checked it by ping command.

[organom]

@hassanshamshir You first need to guarantee that both machines can reach each other, if they can't then no point on trying to make k8s work because it'll not.
If you're trying to test the connectivity by using ping, check if ping is enabled and allowed on the machines (I believe windows firewall may block it).

Then try to read/understand Hyper-V networking and virtual switches. Should be easy to make 2 VMs communicate to each other. VM to host communication could be more tricky to archive.
I don't have too much experience with Hyper-V, so I unfortunately can't help much on that side (on my side I generally run VirtualBox or Unraid)

[hassanshamshir]

@organom can you have an experience of creating cluster using VirtualBox?

[hassanshamshir]

@organom I have created a window VM which have window server2019.

[hassanshamshir]

@organom When I run ping google.com from both machine then they ping it fine.

[hassanshamshir]

I have successfully added window node into my microk&8 cluster using hyper-V in window.

[hassanshamshir]

@biiiipy @organom Now I have defined all the steps.
Steps to Follow:
1: Install microk8s single node cluster in window it will create a VM in window hyper-V Manager after installation.
Official documentation
Note: Need to install calicoctl on microk8s-VM
Run below multipass command in cmd/powershell
multipass shell microk8s-vm
Then run this command
cd /usr/local/bin/
https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install
2: From microk8s VM perform this action
Export kubeconfig
To access the cluster, calicoctl will need a copy of the kubeconfig from MicroK8s VM. This can be made in any location - this example copies it to the default path:
Access terminal in microk8s VM by running below command in Window PowerShell.
multipass shell microk8s-vm
sudo -i
mkdir -p ~/.kube
microk8s config > ~/.kube/config
Run below commad from microk8s-vm
DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl ipam configure --strictaffinity=true
Note: check your config file path is correct

3: Create a second VM in window hyper-V Manager with OS window server 2019 and install docker inside it.
install docker
https://learn.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=dockerce

4: Install components on the Windows node

You are now ready to install Calico on Windows VM. This will also install Kubernetes components required for a working node.
All code snippets here should be run in PowerShell running as Administrator.

5: Create directory for Kubernetes
All Kubernetes components will be installed into this directory.

mkdir C:\k
In here, place the kubeconfig file previously exported from MicroK8s. Be careful as some editors may try to append file extensions. You can test that it has been named correctly by trying to print the contents after you’ve saved it:
New-Item C:\k\config
Install choco and nano in window-vm
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
choco install -y nano
nano C:\k\config
**Copy config file content from microk8s-VM and Paste it in window VM using nano text-editor if you don't have then install it using chocolaty **
6: Install Calico Using Powershell script

First, download the script.

Invoke-WebRequest https://docs.projectcalico.org/scripts/install-calico-windows.ps1 -OutFile C:\install-calico-windows.ps1
You can then run the script with the required parameters. Change the -KubeVersion argument to the version noted earlier.
you will need to know the exact version of Kubernetes installed. This can be grabbed from MicroK8s vm.
microk8s kubectl get node
C:\install-calico-windows.ps1 -DownloadOnly yes -KubeVersion k8s-version
Note: You can also explicitly set whether you are running Calico in vxlan or BGP mode by specifying this as an addidional argument, e.g.-CalicoBackend "vxlan"

Register the Calico services.

C:\CalicoWindows\install-calico.ps1
Then register the Kubernetes services so they come up with the node.
install k8s services
C:\CalicoWindows\kubernetes\install-kube-services.ps1
This script won’t start the Kubernetes services. Let’s do that now:

7:Verify that the Calico services are running.

Get-Service -Name CalicoNode
Get-Service -Name CalicoFelix

8: start kubelet/kube-proxy service. Execute following PowerShell commands.
Start-Service -Name kubelet
Start-Service -Name kube-proxy

9:Verify kubelet/kube-proxy services are running.

Get-Service -Name kubelet
Get-Service -Name kube-proxy
Start-Service Docker

Congratulations! You now have a Kubernetes cluster with Calico for Windows and a Linux control node.

Recommended Links
Official Doc
kubernetes.io
Install calicoctl on microk8s and window VM
install docker on window server 2019
install nano using Chocolatey

[neoaggelos]

@hassanshamshir thanks for this, excellent work!

[pedrofragola]

@hassanshamshir Thanks for the steps described, however using the microk8s 1.26 version the kubelet is constantly restarting on the Windows side.

calico-node.log

2023-01-31 04:33:18.117 [INFO][9048] startup/startup.go 427: Early log level set to info
2023-01-31 04:33:18.117 [INFO][9048] startup/utils.go 127: Using NODENAME environment for node name win-9snsgb4cp64
2023-01-31 04:33:18.117 [INFO][9048] startup/utils.go 139: Determined node name: win-9snsgb4cp64
2023-01-31 04:33:18.117 [INFO][9048] startup/startup.go 94: Starting node win-9snsgb4cp64 with version v3.25.0
2023-01-31 04:33:18.118 [INFO][9048] startup/startup.go 106: Skipping datastore connection test
2023-01-31 04:33:18.131 [INFO][9048] startup/startup.go 470: Building new node resource Name="win-9snsgb4cp64"
2023-01-31 04:33:18.131 [INFO][9048] startup/startup.go 485: Initialize BGP data
2023-01-31 04:33:18.140 [INFO][9048] startup/autodetection_methods.go 103: Using autodetected IPv4 address on interface vEthernet (Ethernet Instance 0): 192.168.8.205/24
2023-01-31 04:33:18.140 [INFO][9048] startup/startup.go 561: Node IPv4 changed, will check for conflicts
2023-01-31 04:33:18.143 [INFO][9048] startup/startup.go 701: No AS number configured on node resource, using global value
2023-01-31 04:33:18.147 [WARNING][9048] startup/node.go 74: Operation Create is not supported on Node type
2023-01-31 04:33:18.147 [ERROR][9048] startup/startup.go 201: Unable to set node resource configuration error=operation Create is not supported on Node(win-9snsgb4cp64)
2023-01-31 04:33:18.147 [WARNING][9048] startup/utils.go 49: Terminating
Calico node initialisation failed, will retry...

Note: With version 1.19 everything worked.

I need to test other versions.