Kubernetes windows node not creating network

[kevinbayes]

I followed the scripts provided by msdn and was unable to create a cbr0 network interface required to install a Windows node.

Looking at the start-kubelet.ps1 script I assume that the script should create the cbr0, but it seems to not work correctly and therefore you cannot follow the AddRoutes commands to the windows nodes.

Are there any known issues or manual instructions on how to add this interface?

Getting Started:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows
Network Topology:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/configuring-host-gateway-mode

Kubernetes:
Kubernetes 1.9.2
System details are below:

OS Name:                   Microsoft Windows Server Standard
OS Version:                10.0.16299 N/A Build 16299
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                00396-02142-95046-AA798
Original Install Date:     25/01/2018, 10:09:06 AM
System Boot Time:          25/01/2018, 10:52:10 AM
System Manufacturer:       Dell Inc.
System Model:              OptiPlex 9020
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~1700 Mhz
BIOS Version:              Dell Inc. A03, 17/09/2013
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
Total Physical Memory:     16,326 MB
Available Physical Memory: 14,258 MB
Virtual Memory: Max Size:  19,270 MB
Virtual Memory: Available: 17,277 MB
Virtual Memory: In Use:    1,993 MB
Hotfix(s):                 1 Hotfix(s) Installed.
                           [01]: KB4054517
Network Card(s):           2 NIC(s) Installed.
                           [01]: Intel(R) Ethernet Connection I217-LM
                                 Connection Name: Ethernet
                                 DHCP Enabled:    Yes
                                 DHCP Server:     172.20.1.6
                                 IP address(es)
                                 [01]: 172.20.40.126
                                 [02]: fe80::d8d5:fce8:baef:3eca
                           [02]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (nat)
                                 DHCP Enabled:    Yes
                                 DHCP Server:     255.255.255.255
                                 IP address(es)
                                 [01]: 192.168.96.1
                                 [02]: fe80::b50d:fd19:b328:1fe6
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

[elee3]

I am having this issue as well.

[funky81]

Have you do manual installation of kubernetes like instructions in the URL above ? Mine having works perfectly after execute startKubulet.ps1, startKuberproxy and script to network (network section).

[elee3]

Line in the AddRoutes.ps1 script such as

$na = Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*"

just do not work since there is no such network adapter in a fresh installation of Windows Server 2016. How did you get it to work?

[funky81]

Have you tried in Powershell?

[elee3]

yes i have tried running in powershell. it didnt work because of the reason i mentioned above

[JMesser81]

@elee3 @funky81 - we just updated the scripts. Can you please try again?

[funky81]

Hi @JMesser81 , i've tried your latest script from Linux and windows nodes. Somehow it didn't worked. I tried to ping to outside cluster from win-webserver (example) pod.

My masterIP (192.168.34.0/24) is outside my worker node IP (192.168.91.0/24). Pod CIDR is 10.168.0.0/16.

Another info, I'm using Centos 7.4 with firewalld, SElinux disable also with Swapoff.

Right now I try to make node (windows) and management node (Centos) as in one network. Also tried to replace firewalld with iptables since generate-routes.sh involve iptables instead of firewalld.

Any solutions for this? I tried hard to make things work but it still doesn't worked.

[Jaspreetc]

@JMesser81
Hi Jason
I am having the same problem with the latest script.
i can list the pods , the Kubernetes is working but for some reason its not listing cbr0 network

$kubectl get pods --all-namespaces
kube-system kube-addon-manager-server1 1/1 Running 0 15m
kube-system kube-apiserver-server1 1/1 Running 0 15m
kube-system kube-controller-manager-server1 1/1 Running 0 14m
kube-system kube-etcd-server1 1/1 Running 0 14m
kube-system kube-scheduler-server1 1/1 Running 0 15m

In the script its trying to use network plugin as "kubenet" i am not sure how kubernetes will come to know about the network plugins, are they specified in the "container image"
Since we have not specified anything specific location for CNI plugins.

$ps aux
root 45250 3.5 2.8 1636748 114956 pts/1 Sl 12:22 0:41 ./bin/hyperkube kubelet --kubeconfig=/root/.kube/config --pod-infra-container-image=gcrio.azureedge.net/google_containers/pause-amd64:3.0 --address=0.0.0.0 --allow-privileged=true --enable-server --enable-debugging-handlers --pod-manifest-path=/root/kube/manifest --cluster-dns=11.0.0.10 --cluster-domain=cluster.local --node-labels=role=master --hairpin-mode=promiscuous-bridge --container-runtime=docker --v=6 --fail-swap-on=false --network-plugin=kubenet

[Jaspreetc]

@JMesser81
do we need to add the below switch somewhere in the shell script
https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet

kubenet
Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments.

Kubenet creates a Linux bridge named cbr0 and creates a veth pair for each pod with the host end of each pair connected to cbr0. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. cbr0 is assigned an MTU matching the smallest MTU of an enabled normal interface on the host.

The plugin requires a few things:

The standard CNI bridge, lo and host-local plugins are required, at minimum version 0.2.0. Kubenet will first search for them in /opt/cni/bin. Specify cni-bin-dir to supply additional search path. The first found match will take effect.
Kubelet must be run with the --network-plugin=kubenet argument to enable the plugin
Kubelet should also be run with the --non-masquerade-cidr= argument to ensure traffic to IPs outside this range will use IP masquerade.

The node must be assigned an IP subnet through either the --pod-cidr kubelet command-line option or the --allocate-node-cidrs=true --cluster-cidr= controller-manager command-line options.

[calebpalmer]

I'm in the same boat. I can't add the windows routes because there is no vEthernet (cbr0).

[JMesser81]

@daschott - can you take a look at this? We recently updated our HOW TO guide and David can provide an update (or even better, post on our repo) :-)

[daschott]

Here is the most recent How-to guide for deploying a mixed-OS Kubernetes cluster using Flannel. These instructions will work on either Windows Server, version 1803 or Windows Server 2019. These are also compliant with the way a Linux master is initialized using kubeadm, making deployment of not just Windows, but also Linux easier than before.

The benefit of using Flannel as a network management solution is simpler configuration and scalability, as it programs routes automatically for you. This is in contrast to our (current) public documentation that is trickier to deploy, but offers more control due to manual route management.

[calebpalmer]

I followed the instructions again. Thank you very much for updating them. I'm running into this issue running start.ps1 to join the windows node. It just keeps spinning with this error:

"E0618 09:52:23.273306 1188 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.26.64.195:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dfrd-vtapp-kub03&limit=500&resourceVersion=0: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")"

A couple of differences to note:

• Master node is running kubernetes 1.10.4 on CentOS 7.5.1804
• The windows node is 10.0.16299 N/A Build 16299 (1709)

[calebpalmer]

Please ignore my last comment. I accidentally copied the wrong config file. The windows node was joined! Thanks!