Support Windows in host-gw and VXLAN modes #833

rakelkar · 2017-10-04T05:07:24Z

Windows RS3 release introduces support for Kubernetes networking, however setting it up without a Flannel like component is very difficult.The Windows Server RS3 release supports host-gw as L2Bridge and vxlan in Overlay modes - this issue proposes supporting Flannel on windows for host-gw and vxlan modes and links to a proposed PR.

See here for details on new Windows support for K8s networking: http://blog.kubernetes.io/2017/09/windows-networking-at-parity-with-linux.html

Possible Solution

Proposed PR: #832
Related CNI changes: containernetworking/plugins#76

Your Environment

Operating System and version: Windows RS3

This uses new Windows Host Networking Service (HNS) features to configure networking for the host-gw mode Fixes flannel-io#833

rakelkar · 2017-10-18T18:14:24Z

containernetworking/plugins#77

rakelkar · 2017-10-18T18:14:52Z

containernetworking/plugins#76

cspwizard · 2017-12-06T09:58:43Z

Hi!

Are there any plans when this would be available for use on Windows K8S systems?

Thanks!

rakelkar · 2017-12-06T14:51:16Z

Yes we are working towards making this available... Would you like to try early bits to see if it works for you?

…

On Dec 6, 2017 1:58 AM, "Konstantin Pozdniakov" ***@***.***> wrote: Hi! Are there any plans when this would be available for use on Windows K8S systems? Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#833 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADQphXAyihTEssi399tHY5veDl_Y-kkgks5s9mVXgaJpZM4PtGg_> .

rjmorse · 2017-12-06T14:57:34Z

Count me in as a tester of early bits as well please.

cspwizard · 2017-12-06T15:01:26Z

@rakelkar yes, sure!

rakelkar · 2017-12-07T00:40:15Z

@cspwizard @rjmorse
I've posted some slides that describe how flannel works on windows and how to set it up on the k8s sig-windows slack channel... have a look and give it a shot. We're working with the Flannel maintainers to get this merged step by step :) currently stuck at #891

dmitryzykov · 2018-01-15T17:45:26Z

Hello @rakelkar
I've tried to test host-gw and VXLAN modes for Windows 1709 nodes, but unfortunately, it doesn't work at the end.
What I did after reading slack talks, watching your demo video and presentation from slack:

ran kubernetes v1.9.0 cluster with flannel in VXLAN and then in host-gw mode.
compiled all required windows binaries from: containernetworking/plugins#85 and #832 and v1.9.0 kubernetes repo

├── admin.conf
├── cni
│   ├── config
│   │   └── cni-conf.json
│   ├── flannel.exe
│   ├── host-local.exe
│   ├── win-l2bridge.exe
│   └── win-overlay.exe
├── etc
│   └── kube-flannel
│       └── net-conf.json
├── flanneld.exe
├── kubectl.exe
├── kubelet.exe
└── kube-proxy.exe

started binaries

c:\k\kubelet.exe --hostname-override=windows1 --v=6 `
    --pod-infra-container-image=kubeletwin/pause --resolv-conf="" `
    --allow-privileged=true --enable-debugging-handlers `
    --cluster-dns=10.233.0.3 --cluster-domain=cluster.local `
    --kubeconfig=c:\k\admin.conf --hairpin-mode=promiscuous-bridge `
    --image-pull-progress-deadline=20m --cgroups-per-qos=false `
    --enforce-node-allocatable="" `
    --network-plugin=cni --cni-bin-dir="c:\k\cni" --cni-conf-dir "c:\k\cni\config"

$env:NODE_NAME = "windows1"
.\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr=1

windows node sucessfully joined cluster:

NAME       STATUS    ROLES     AGE       VERSION
linux4     Ready     node      19m       v1.9.0+coreos.0
linux5     Ready     node      19m       v1.9.0+coreos.0
linux6     Ready     node      18m       v1.9.0+coreos.0
master1    Ready     master    19m       v1.9.0+coreos.0
master2    Ready     master    19m       v1.9.0+coreos.0
windows1   Ready     <none>    4m        v1.9.0-dirty

created POD, which stuck in ContainerCreating status

kubectl get pods
NAME                             READY     STATUS              RESTARTS   AGE
win-webserver-6f5c5c676d-8mt77   0/1       ContainerCreating   0          10m

in pod events i found that:

  Warning  FailedCreatePodSandBox  <invalid> (x6 over 25s)  kubelet, windows1  Failed create pod sandbox.
  Normal   SandboxChanged          <invalid> (x6 over 25s)  kubelet, windows1  Pod sandbox changed, it will be killed and re-created.

In kubelet windows logs was error:

E0115 08:42:00.545300   56084 kuberuntime_manager.go:647] createPodSandbox for pod "win-webserver-6f5c5c676d-8mt77_default(e5deb513-fa12-11e7-a02b-00155d010712)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "win-webserver-6f5c5c676d-8mt77_default" network: HNS failed with error : The parameter is incorrect.
E0115 08:42:00.546236   56084 pod_workers.go:186] Error syncing pod e5deb513-fa12-11e7-a02b-00155d010712 ("win-webserver-6f5c5c676d-8mt77_default(e5deb513-fa12-11e7-a02b-00155d010712)"), skipping: failed to "CreatePodSandbox" for "win-webserver-6f5c5c676d-8mt77_default(e5deb513-fa12-11e7-a02b-00155d010712)" with CreatePodSandboxError: "CreatePodSandbox for pod \"win-webserver-6f5c5c676d-8mt77_default(e5deb513-fa12-11e7-a02b-00155d010712)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"win-webserver-6f5c5c676d-8mt77_default\" network: HNS failed with error : The parameter is incorrect. "
I0115 08:42:00.546236   56084 server.go:231] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"win-webserver-6f5c5c676d-8mt77", UID:"e5deb513-fa12-11e7-a02b-00155d010712", APIVersion:"v1", ResourceVersion:"2640", FieldPath:""}): type: 'Warning' reason: 'FailedCreatePodSandBox' Failed create pod sandbox.

This error was in VXLAN mode.
my CNI configs:

bash-4.2# cat cni-conf.json
{
   "name": "vxlan0",
   "type": "flannel",
   "delegate": {
      "type": "win-overlay"
   }
}
bash-4.2# cat net-conf.json
{
  "Network": "10.244.0.0/16",
  "Backend": {
    "name": "vxlan0",
    "type": "vxlan",
    "vni": 4096
  }
}

and flannel output:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0115 08:37:11.927665   57700 main.go:411] Searching for interface using 10.2.51.61
I0115 08:37:12.198777   57700 main.go:482] Using interface with name Ethernet and address 10.2.51.61
I0115 08:37:12.198777   57700 main.go:499] Defaulting external address to interface address (10.2.51.61)
I0115 08:37:12.209776   57700 kube.go:130] Waiting 10m0s for node controller to sync
I0115 08:37:12.209776   57700 kube.go:283] Starting kube subnet manager
I0115 08:37:13.212567   57700 kube.go:137] Node controller sync successful
I0115 08:37:13.212567   57700 main.go:234] Created subnet manager: Kubernetes Subnet Manager - windows1
I0115 08:37:13.214566   57700 main.go:237] Installing signal handlers
I0115 08:37:13.215568   57700 main.go:347] Found network config - Backend type: vxlan
I0115 08:37:13.215568   57700 vxlan_windows.go:111] VXLAN config: {name:vxlan0 macPrefix:0E-2A VNI:4096 Port:4789 GBP:false DirectRouting:false}
I0115 08:37:13.269572   57700 vxlan_windows.go:181] Attempting to create HNS network, request: {"Name":"vxlan0","Subnets":[{"AddressPrefix":"10.244.0.0/16","GatewayAddress":"10.244.0.1","Policies":[{"Type":"VSID","VSID":4096}]}],"Type":"Overlay"}
E0115 08:37:14.385312   57700 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.2.51.61:51724->10.2.51.51:6443: wsarecv: An established connection was aborted by the software in your host machine.
E0115 08:37:14.388313   57700 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:284: Failed to watch *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=2227&timeoutSeconds=582&watch=true: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable network.
I0115 08:37:15.358324   57700 vxlan_windows.go:186] Created HNS network [vxlan0] as &{Id:ab461c56-1327-44bf-aa7e-1a985c84b76e Name:vxlan0 Type:Overlay NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-15-5D-10-A0-00 EndMacAddress:00-15-5D-10-AF-FF}] Subnets:[{AddressPrefix:10.244.0.0/16 GatewayAddress:10.244.0.1 Policies:[[123 32 34 84 121 112 101 34 32 58 32 34 86 83 73 68 34 44 32 34 86 83 73 68 34 32 58 32 52 48 57 54 32 125]]}] DNSSuffix: DNSServerList: DNSServerCompartment:4 ManagementIP:10.2.51.61 AutomaticDNS:false}
I0115 08:37:15.364326   57700 main.go:294] Wrote subnet file to /run/flannel/subnet.env
I0115 08:37:15.364326   57700 main.go:298] Running backend.
I0115 08:37:15.364326   57700 main.go:316] Waiting for all goroutines to exit
I0115 08:37:15.364326   57700 vxlan_network_windows.go:56] Watching for new subnet leases
E0115 08:37:15.393328   57700 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:284: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
E0115 08:37:16.396338   57700 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:284: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
E0115 08:37:17.398268   57700 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:284: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
I0115 08:37:44.830206   57700 vxlan_network_windows.go:123] Subnet added: 10.233.64.0/24 [29464486400 ns]
I0115 08:38:35.936004   57700 vxlan_network_windows.go:123] Subnet added: 10.233.65.0/24 [51103984000 ns]
I0115 08:39:50.522490   57700 vxlan_network_windows.go:123] Subnet added: 10.233.66.0/24 [74584420300 ns]
I0115 08:41:26.992390   57700 vxlan_network_windows.go:123] Subnet added: 10.233.67.0/24 [96467647200 ns]
I0115 08:43:12.912303   57700 vxlan_network_windows.go:123] Subnet added: 10.233.68.0/24 [105917484400 ns]

and network config:

PS C:\k> Get-HNSNetwork


ActivityId             : ba2b92fe-f41b-450d-9a36-7700d9e02265
CurrentEndpointCount   : 0
DNSServerCompartment   : 4
DrMacAddress           : 00-15-5D-7B-F9-50
Extensions             : {@{Id=e7c3b2f0-f3c5-48df-af2b-10fed6d72e7a; IsEnabled=False},
                         @{Id=e9b59cfa-2be1-4b21-828f-b6fbdbddc017; IsEnabled=True},
                         @{Id=ea24cd6c-d17a-4348-9190-09f0d5be83dd; IsEnabled=False}}
ID                     : ab461c56-1327-44bf-aa7e-1a985c84b76e
LayeredOn              : 60f6e4a4-bf78-479e-b987-c0c9ec4040a2
MacPools               : {@{EndMacAddress=00-15-5D-10-AF-FF; StartMacAddress=00-15-5D-10-A0-00}}
ManagementIP           : 10.2.51.61
MaxConcurrentEndpoints : 0
Name                   : vxlan0
Policies               : {}
Resources              : @{AllocationOrder=1; Allocators=System.Object[];
                         ID=ba2b92fe-f41b-450d-9a36-7700d9e02265; PortOperationTime=0; State=1;
                         SwitchOperationTime=0; VfpOperationTime=0;
                         parentId=d2471075-c1e1-4dc4-bb0d-e0134efd004c}
State                  : 1
Subnets                : {@{AddressPrefix=10.244.0.0/16; GatewayAddress=10.244.0.1; Policies=System.Object[]}}
TotalEndpoints         : 0
Type                   : Overlay
Version                : 21474836481

So it stuck with kubelet error network: HNS failed with error : The parameter is incorrect. and i couldn't solve this.

After that, I've reinstalled cluster in host-gw mode
my CNI configs (i've tried both br0 and cbr0 names) :

bash-4.2# cat cni-conf.json
{
   "name": "cbr0",
   "type": "flannel",
   "delegate": {
      "type": "win-l2bridge"
   }
}
bash-4.2# cat net-conf.json
{
  "Network": "10.244.0.0/16",
  "Backend": {
    "name": "cbr0",
    "type": "host-gw",
    "vni": 4096
  }
}

and flannel is failed to run:
when I started it for the first time:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0114 12:10:07.263566   51624 main.go:411] Searching for interface using 10.2.51.61
I0114 12:10:07.709211   51624 main.go:482] Using interface with name Ethernet and address 10.2.51.61
I0114 12:10:07.709211   51624 main.go:499] Defaulting external address to interface address (10.2.51.61)
I0114 12:10:07.717207   51624 kube.go:130] Waiting 10m0s for node controller to sync
I0114 12:10:07.717207   51624 kube.go:283] Starting kube subnet manager
I0114 12:10:08.717246   51624 kube.go:137] Node controller sync successful
I0114 12:10:08.717246   51624 main.go:234] Created subnet manager: Kubernetes Subnet Manager - windows1
I0114 12:10:08.720248   51624 main.go:237] Installing signal handlers
I0114 12:10:08.720248   51624 main.go:347] Found network config - Backend type: host-gw
I0114 12:10:08.744258   51624 hostgw_windows.go:147] Attempting to create HNS network, request: {"DNSServerList
":"","Name":"cbr0","Subnets":[{"AddressPrefix":"10.233.69.0/24","GatewayAddress":"10.233.69.1"}],"Type":"l2brid
ge"}
E0114 12:10:09.426250   51624 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 1
0.2.51.61:51473->10.2.51.51:6443: wsarecv: An established connection was aborted by the software in your host m
achine.
E0114 12:10:09.428263   51624 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:284: Failed to wa
tch *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=13593&timeoutSeconds=582&watch=true: dia
l tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable network.
I0114 12:10:10.058264   51624 hostgw_windows.go:153] Created HNS network [cbr0] as &{Id:a92467b9-7e34-44bf-ae5e
-07ca4703b45e Name:cbr0 Type:l2bridge NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-
15-5D-CD-90-00 EndMacAddress:00-15-5D-CD-9F-FF}] Subnets:[{AddressPrefix:10.233.69.0/24 GatewayAddress:10.233.6
9.1 Policies:[]}] DNSSuffix: DNSServerList: DNSServerCompartment:0 ManagementIP: AutomaticDNS:false}
I0114 12:10:10.065256   51624 hostgw_windows.go:182] Attempting to create HNS endpoint [&{Id: Name:cbr0_ep Virt
ualNetwork:a92467b9-7e34-44bf-ae5e-07ca4703b45e VirtualNetworkName: Policies:[] MacAddress: IPAddress:10.233.69
.2 DNSSuffix: DNSServerList: GatewayAddress: EnableInternalDNS:false DisableICC:false PrefixLength:0 IsRemoteEn
dpoint:false}]
I0114 12:10:10.104265   51624 hostgw_windows.go:188] Created bridge endpoint [cbr0_ep] as &{Id:b4aaf069-e5f9-40
5f-96de-191b625650a4 Name:cbr0_ep VirtualNetwork:a92467b9-7e34-44bf-ae5e-07ca4703b45e VirtualNetworkName:cbr0 P
olicies:[[123 32 34 84 121 112 101 34 32 58 32 34 76 50 68 114 105 118 101 114 34 32 125]] MacAddress:00-15-5D-
CD-98-11 IPAddress:10.233.69.2 DNSSuffix: DNSServerList: GatewayAddress:10.233.69.1 EnableInternalDNS:false Dis
ableICC:false PrefixLength:24 IsRemoteEndpoint:false}
I0114 12:10:10.335260   51624 hostgw_windows.go:194] Attached bridge endpoint [cbr0_ep] to host
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0xc0 pc=0x1177fcc]

goroutine 1 [running]:
github.com/coreos/flannel/backend/hostgw.(*HostgwBackend).RegisterNetwork(0xc042176500, 0x6e70068, 0xc0423b4600
, 0xc042142910, 0xc042176500, 0x0, 0x0, 0xa0042599d01)
        /root/gopath/src/github.com/coreos/flannel/backend/hostgw/hostgw_windows.go:198 +0x171c
main.main()
        /root/gopath/src/github.com/coreos/flannel/main.go:277 +0x7a4

and for all the next:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0114 12:12:25.132420   51192 main.go:411] Searching for interface using 10.2.51.61
I0114 12:12:25.383257   51192 main.go:482] Using interface with name vEthernet (Ethernet) and address 10.2.51.6
1
I0114 12:12:25.383257   51192 main.go:499] Defaulting external address to interface address (10.2.51.61)
I0114 12:12:25.393256   51192 kube.go:130] Waiting 10m0s for node controller to sync
I0114 12:12:25.393256   51192 kube.go:283] Starting kube subnet manager
I0114 12:12:26.393277   51192 kube.go:137] Node controller sync successful
I0114 12:12:26.393277   51192 main.go:234] Created subnet manager: Kubernetes Subnet Manager - windows1
I0114 12:12:26.395298   51192 main.go:237] Installing signal handlers
I0114 12:12:26.396283   51192 main.go:347] Found network config - Backend type: host-gw
I0114 12:12:26.419268   51192 hostgw_windows.go:114] Found existing HNS network [&{Id:a92467b9-7e34-44bf-ae5e-0
7ca4703b45e Name:cbr0 Type:l2bridge NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-15
-5D-CD-90-00 EndMacAddress:00-15-5D-CD-9F-FF}] Subnets:[{AddressPrefix:10.233.69.0/24 GatewayAddress:10.233.69.
1 Policies:[]}] DNSSuffix: DNSServerList: DNSServerCompartment:0 ManagementIP:10.2.51.61 AutomaticDNS:false}]
I0114 12:12:26.428269   51192 hostgw_windows.go:162] Found existing HNS bridge endpoint [&{Id:b4aaf069-e5f9-405
f-96de-191b625650a4 Name:cbr0_ep VirtualNetwork:a92467b9-7e34-44bf-ae5e-07ca4703b45e VirtualNetworkName:cbr0 Po
licies:[[123 32 34 84 121 112 101 34 32 58 32 34 76 50 68 114 105 118 101 114 34 32 125]] MacAddress:00-15-5D-C
D-98-11 IPAddress:10.233.69.2 DNSSuffix: DNSServerList: GatewayAddress:10.233.69.1 EnableInternalDNS:false Disa
bleICC:false PrefixLength:24 IsRemoteEndpoint:false}]
I0114 12:12:26.449273   51192 hostgw_windows.go:194] Attached bridge endpoint [cbr0_ep] to host
E0114 12:12:26.810274   51192 main.go:279] Error registering network: unable to enable forwarding on [vEthernet
 (Ethernet)], error: failed to enable forwarding on [vEthernet (Ethernet)], error: exit status 1. cmd: int ipv4
 set int "vEthernet (Ethernet)" for=en. stdout: The parameter is incorrect.


I0114 12:12:26.810274   51192 main.go:327] Stopping shutdownHandler...

I appreciate any help to run it. Also, I can provide any details of my setup. Thank you.

rakelkar · 2018-01-16T02:25:41Z

@kdomanski for the l2bridge issue - this is a known issue... @jroggeman made a fix for this in the underlying gonetsh library, but we didn;t update the PR (since it was closed). I need to create a new PR that includes this fix.

rakelkar · 2018-01-16T02:27:26Z

For overlay - when did you build the overlay plugin? @banatm made a fix in https://github.com/rakelkar/plugins/tree/windowsCni - can you make sure you have the fix?

rakelkar · 2018-01-16T03:53:32Z

For L2Bridge try #921 that should have the fix for:
"Error registering network: unable to enable forwarding on [vEthernet (Ethernet)], error: failed to enable forwarding on [vEthernet (Ethernet)], error: exit status 1. cmd: int ipv4 set int "vEthernet (Ethernet)" for=en. stdout: The parameter is incorrect."

dmitryzykov · 2018-01-16T21:05:23Z

@rakelkar
I'm trying to build it now from https://github.com/rakelkar/plugins/tree/windowsCni, but I can't find valid deps:

-bash-4.2# go get -d github.com/rakelkar/plugins/
package github.com/rakelkar/plugins: no Go files in /root/gopath/src/github.com/rakelkar/plugins
-bash-4.2# cd $GOPATH/src/github.com/rakelkar/plugins/
-bash-4.2# git checkout windowsCni
Branch windowsCni set up to track remote branch windowsCni from origin.
Switched to a new branch 'windowsCni'
-bash-4.2# export GOOS=windows GOARCH=amd64
-bash-4.2# go get ./...
package github.com/containernetworking/plugins/pkg/hns: cannot find package "github.com/containernetworking/plugins/pkg/hns" in any of:
	/root/go/src/github.com/containernetworking/plugins/pkg/hns (from $GOROOT)
	/root/gopath/src/github.com/containernetworking/plugins/pkg/hns (from $GOPATH)
-bash-4.2# go build -o /root/windows/cni/win-overlay.exe -pkgdir "$GOPATH/pkg" ./plugins/main/windows/overlay
plugins/main/windows/overlay/overlay_windows.go:30:2: cannot find package "github.com/containernetworking/plugins/pkg/hns" in any of:
	/root/gopath/src/github.com/rakelkar/plugins/vendor/github.com/containernetworking/plugins/pkg/hns (vendor tree)
	/root/go/src/github.com/containernetworking/plugins/pkg/hns (from $GOROOT)
	/root/gopath/src/github.com/containernetworking/plugins/pkg/hns (from $GOPATH)

If I get hns from containernetworking/plugins#85 it doesn't compile well

-bash-4.2# go build -o /root/windows/cni/win-overlay.exe -pkgdir "$GOPATH/pkg" ./plugins/main/windows/overlay
# github.com/rakelkar/plugins/plugins/main/windows/overlay
plugins/main/windows/overlay/overlay_windows.go:87:85: cannot use func literal (type func() (*"github.com/rakelkar/plugins/vendor/github.com/Microsoft/hcsshim".HNSEndpoint, error)) as type hns.EndpointMakerFunc in argument to hns.ProvisionEndpoint
plugins/main/windows/overlay/overlay_windows.go:95:45: cannot use r (type "github.com/containernetworking/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result) as type "github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result in argument to "github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types/current".NewResultFromResult:
	"github.com/containernetworking/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result does not implement "github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result (wrong type for GetAsVersion method)
		have GetAsVersion(string) ("github.com/containernetworking/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result, error)
		want GetAsVersion(string) ("github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result, error)
plugins/main/windows/overlay/overlay_windows.go:132:36: cannot use hnsNetwork (type *"github.com/rakelkar/plugins/vendor/github.com/Microsoft/hcsshim".HNSNetwork) as type *"github.com/containernetworking/plugins/vendor/github.com/Microsoft/hcsshim".HNSNetwork in argument to hns.ConstructResult
plugins/main/windows/overlay/overlay_windows.go:137:26: cannot use result (type *"github.com/containernetworking/plugins/vendor/github.com/containernetworking/cni/pkg/types/current".Result) as type "github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result in argument to "github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".PrintResult:
	*"github.com/containernetworking/plugins/vendor/github.com/containernetworking/cni/pkg/types/current".Result does not implement "github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result (wrong type for GetAsVersion method)
		have GetAsVersion(string) ("github.com/containernetworking/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result, error)
		want GetAsVersion(string) ("github.com/rakelkar/plugins/vendor/github.com/containernetworking/cni/pkg/types".Result, error)

so where is proper location for hns?

dmitryzykov · 2018-01-17T18:51:14Z

@rakelkar
I've tried host-gw from #921

bash-4.2# go get -d github.com/coreos/flannel
bash-4.2# cd $GOPATH/src/github.com/coreos/flannel/
bash-4.2#
bash-4.2# git fetch origin pull/921/head:rakelkar/windowsHostGw180115
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 2), reused 5 (delta 2), pack-reused 0
Unpacking objects: 100% (6/6), done.
From https://github.com/coreos/flannel
 * [new ref]         refs/pull/921/head -> rakelkar/windowsHostGw180115
bash-4.2# git checkout rakelkar/windowsHostGw180115
Switched to branch 'rakelkar/windowsHostGw180115'
bash-4.2#
bash-4.2# go get -d github.com/Microsoft/hcsshim
bash-4.2# go get -d github.com/rakelkar/gonetsh/netroute
bash-4.2# go get -d github.com/rakelkar/gonetsh/netsh
bash-4.2#
bash-4.2# curl https://raw.githubusercontent.com/coreos/flannel/master/Makefile > Makefile
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10820  100 10820    0     0  46837      0 --:--:-- --:--:-- --:--:-- 47043
bash-4.2# make dist/flanneld.exe
GOOS=windows go build -o dist/flanneld.exe \
  -ldflags '-s -w -X github.com/coreos/flannel/version.Version=v0.9.0-84-g1cb9412 -extldflags "-static"'

but I have the same error:
At the first run:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0117 10:27:15.872672   61908 main.go:417] Searching for interface using 10.2.51.61
I0117 10:27:16.123660   61908 main.go:488] Using interface with name Ethernet and address 10.2.51.61
I0117 10:27:16.123660   61908 main.go:505] Defaulting external address to interface address (10.2.51.61)
I0117 10:27:16.136660   61908 kube.go:131] Waiting 10m0s for node controller to sync
I0117 10:27:16.136660   61908 kube.go:294] Starting kube subnet manager
I0117 10:27:17.137377   61908 kube.go:138] Node controller sync successful
I0117 10:27:17.137377   61908 main.go:235] Created subnet manager: Kubernetes Subnet Manager - windows1
I0117 10:27:17.138381   61908 main.go:238] Installing signal handlers
I0117 10:27:17.138381   61908 main.go:353] Found network config - Backend type: host-gw
I0117 10:27:17.162379   61908 hostgw_windows.go:147] Attempting to create HNS network, request: {"DNSServerList
":"","Name":"cbr0","Subnets":[{"AddressPrefix":"10.233.69.0/24","GatewayAddress":"10.233.69.1"}],"Type":"l2brid
ge"}
E0117 10:27:17.985358   61908 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 1
0.2.51.61:61216->10.2.51.51:6443: wsarecv: An established connection was aborted by the software in your host m
achine.
E0117 10:27:17.987375   61908 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to wa
tch *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=6487&timeoutSeconds=582&watch=true: dial
 tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable network.
I0117 10:27:18.669142   61908 hostgw_windows.go:153] Created HNS network [cbr0] as &{Id:ea7c6139-2b72-449c-afd5
-4b804860dd83 Name:cbr0 Type:l2bridge NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-
15-5D-5F-10-00 EndMacAddress:00-15-5D-5F-1F-FF}] Subnets:[{AddressPrefix:10.233.69.0/24 GatewayAddress:10.233.6
9.1 Policies:[]}] DNSSuffix: DNSServerList: DNSServerCompartment:0 ManagementIP: AutomaticDNS:false}
I0117 10:27:18.676144   61908 hostgw_windows.go:182] Attempting to create HNS endpoint [&{Id: Name:cbr0_ep Virt
ualNetwork:ea7c6139-2b72-449c-afd5-4b804860dd83 VirtualNetworkName: Policies:[] MacAddress: IPAddress:10.233.69
.2 DNSSuffix: DNSServerList: GatewayAddress: EnableInternalDNS:false DisableICC:false PrefixLength:0 IsRemoteEn
dpoint:false}]
I0117 10:27:18.724142   61908 hostgw_windows.go:188] Created bridge endpoint [cbr0_ep] as &{Id:23d91650-0e11-43
08-8377-c06ae4554533 Name:cbr0_ep VirtualNetwork:ea7c6139-2b72-449c-afd5-4b804860dd83 VirtualNetworkName:cbr0 P
olicies:[[123 32 34 84 121 112 101 34 32 58 32 34 76 50 68 114 105 118 101 114 34 32 125]] MacAddress:00-15-5D-
5F-11-0E IPAddress:10.233.69.2 DNSSuffix: DNSServerList: GatewayAddress:10.233.69.1 EnableInternalDNS:false Dis
ableICC:false PrefixLength:24 IsRemoteEndpoint:false}
E0117 10:27:18.994143   61908 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to li
st *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A
 socket operation was attempted to an unreachable host.
I0117 10:27:19.022139   61908 hostgw_windows.go:194] Attached bridge endpoint [cbr0_ep] to host
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0xc0 pc=0x117892c]

goroutine 1 [running]:
github.com/coreos/flannel/backend/hostgw.(*HostgwBackend).RegisterNetwork(0xc042674160, 0x71b0000, 0xc04203f900
, 0xc042131090, 0xc042674160, 0x0, 0x0, 0xa004259bd01)
        /root/gopath/src/github.com/coreos/flannel/backend/hostgw/hostgw_windows.go:198 +0x171c
main.main()
        /root/gopath/src/github.com/coreos/flannel/main.go:278 +0x7a7

The second run:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0117 10:29:16.131574   56212 main.go:417] Searching for interface using 10.2.51.61
I0117 10:29:16.385577   56212 main.go:488] Using interface with name vEthernet (Ethernet) and address 10.2.51.61
I0117 10:29:16.386578   56212 main.go:505] Defaulting external address to interface address (10.2.51.61)
I0117 10:29:16.395587   56212 kube.go:131] Waiting 10m0s for node controller to sync
I0117 10:29:16.395587   56212 kube.go:294] Starting kube subnet manager
I0117 10:29:17.397165   56212 kube.go:138] Node controller sync successful
I0117 10:29:17.397165   56212 main.go:235] Created subnet manager: Kubernetes Subnet Manager - windows1
I0117 10:29:17.399167   56212 main.go:238] Installing signal handlers
I0117 10:29:17.399167   56212 main.go:353] Found network config - Backend type: host-gw
I0117 10:29:17.421162   56212 hostgw_windows.go:114] Found existing HNS network [&{Id:ea7c6139-2b72-449c-afd5-4b804860dd83 Name:cbr0 Type:l2bridge NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-15-5D-5F-10-00 EndMacAddress:00-15-5D-5F-1F-FF}] Subnets:[{AddressPrefix:10.233.69.0/24 GatewayAddress:10.233.69.1 Policies:[]}] DNSSuffix: DNSServerList: DNSServerCompartment:0 ManagementIP:10.2.51.61 AutomaticDNS:false}]
I0117 10:29:17.429174   56212 hostgw_windows.go:162] Found existing HNS bridge endpoint [&{Id:23d91650-0e11-4308-8377-c06ae4554533 Name:cbr0_ep VirtualNetwork:ea7c6139-2b72-449c-afd5-4b804860dd83 VirtualNetworkName:cbr0 Policies:[[123 32 34 84 121 112 101 34 32 58 32 34 76 50 68 114 105 118 101 114 34 32 125]] MacAddress:00-15-5D-5F-11-0E IPAddress:10.233.69.2 DNSSuffix: DNSServerList: GatewayAddress:10.233.69.1 EnableInternalDNS:false DisableICC:false PrefixLength:24 IsRemoteEndpoint:false}]
I0117 10:29:17.436165   56212 hostgw_windows.go:194] Attached bridge endpoint [cbr0_ep] to host
E0117 10:29:17.793563   56212 main.go:280] Error registering network: unable to enable forwarding on [vEthernet (Ethernet)], error: failed to enable forwarding on [vEthernet (Ethernet)], error: exit status 1. cmd: int ipv4 set int "vEthernet (Ethernet)" for=en. stdout: The parameter is incorrect.


I0117 10:29:17.793563   56212 main.go:333] Stopping shutdownHandler...

rakelkar · 2018-01-18T01:43:29Z

Here is how I build CNI (not a GO expert so feel free to critique my method).
(on a windows machine...)
cd $GOPATH\github.com
mkdir containernetworking
cd containernetworking
git clone https://github.com/rakelkar/plugins
go build -o overlay.exe github.com\containernetworking\plugins\plugins\main\windows\overlay

rakelkar · 2018-01-18T03:18:30Z

Figured out forwarding issue and crash issue.. raised pull request in underlying lib for the forwarding issue: rakelkar/gonetsh#6

rakelkar · 2018-01-18T20:08:50Z

@dmitryzykov I've updated #921 with fixes - but to use it you still need to update gonetsh (see comment on the PR)

rakelkar · 2018-01-19T20:31:54Z

@madhanrm the host-gw fails for me with HNS errors that I dont understand. We've made fixes for netsh etc. to work that @dmitryzykov saw - but unable to figure out HNS errors. Can you give it a try when you get a chance? To build you need #921 but also need latest from https://github.com/rakelkar/gonetsh (to do that just delete the vendored gonetsh - and go will use the copy you cloned locally)

dmitryzykov · 2018-01-20T19:34:42Z

@rakelkar I've tried vxlan (CNI from latest https://github.com/rakelkar/plugins/ windowsCni branch and flannel from #922 ) and I got this errors in kubelet at windows node:

E0120 11:20:50.232260   12812 cni.go:259] Error adding network: Error while ProvisionEndpoint(fbe0d3bfae648e2d715f3b2d8a04a355ff11c767ae301b19fe847a69a78de379_vxlan0,774d8ed7-7299-4665-9141-2a9135933c92,fbe0d3bfae648e2d715f3b2d8a04a355ff11c767ae301b19fe847a69a78de379) :Error while ipam.ExecAdd: failed to allocate for range 0: no IP addresses available in range set: 10.233.69.1-10.233.69.254

kubelet at linux master node started with --service-cluster-ip-range=10.233.0.0/18

flanneld output:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0120 11:17:31.026000    6608 main.go:417] Searching for interface using 10.2.51.61
I0120 11:17:31.283997    6608 main.go:488] Using interface with name Ethernet and address 10.2.51.61
I0120 11:17:31.283997    6608 main.go:505] Defaulting external address to interface address (10.2.51.61)
I0120 11:17:31.295000    6608 kube.go:131] Waiting 10m0s for node controller to sync
I0120 11:17:31.295000    6608 kube.go:294] Starting kube subnet manager
I0120 11:17:32.295801    6608 kube.go:138] Node controller sync successful
I0120 11:17:32.295801    6608 main.go:235] Created subnet manager: Kubernetes Subnet Manager - windows1
I0120 11:17:32.299810    6608 main.go:238] Installing signal handlers
I0120 11:17:32.300803    6608 main.go:353] Found network config - Backend type: vxlan
I0120 11:17:32.301805    6608 vxlan_windows.go:111] VXLAN config: {name:vxlan0 macPrefix:0E-2A VNI:4096 Port:4789 GBP:false DirectRouting:false}
I0120 11:17:32.331802    6608 vxlan_windows.go:181] Attempting to create HNS network, request: {"Name":"vxlan0","Subnets":[{"AddressPrefix":"10.233.64.0/18","GatewayAddress":"10.233.64.1","Policies":[{"Type":"VSID","VSID":4096}]}],"Type":"Overlay"}
E0120 11:17:33.806779    6608 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.2.51.61:49922->10.2.51.51:6443: wsarecv: An established connection was aborted by the software in your host machine.
E0120 11:17:33.810778    6608 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to watch *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=269379&timeoutSeconds=582&watch=true: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable network.
I0120 11:17:34.739796    6608 vxlan_windows.go:186] Created HNS network [vxlan0] as &{Id:774d8ed7-7299-4665-9141-2a9135933c92 Name:vxlan0 Type:Overlay NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-15-5D-C0-A0-00 EndMacAddress:00-15-5D-C0-AF-FF}] Subnets:[{AddressPrefix:10.233.64.0/18 GatewayAddress:10.233.64.1 Policies:[[123 32 34 84 121 112 101 34 32 58 32 34 86 83 73 68 34 44 32 34 86 83 73 68 34 32 58 32 52 48 57 54 32 125]]}] DNSSuffix: DNSServerList: DNSServerCompartment:3 ManagementIP:10.2.51.61 AutomaticDNS:false}
I0120 11:17:34.752798    6608 main.go:300] Wrote subnet file to /run/flannel/subnet.env
I0120 11:17:34.752798    6608 main.go:304] Running backend.
I0120 11:17:34.753799    6608 main.go:322] Waiting for all goroutines to exit
I0120 11:17:34.753799    6608 vxlan_network_windows.go:56] Watching for new subnet leases
E0120 11:17:34.852820    6608 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
E0120 11:17:35.854811    6608 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
E0120 11:17:36.855828    6608 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
I0120 11:17:59.823878    6608 vxlan_network_windows.go:123] Subnet added: 10.233.66.0/24 [25067727500 ns]
I0120 11:18:48.529974    6608 vxlan_network_windows.go:123] Subnet added: 10.233.67.0/24 [48704449400 ns]
I0120 11:19:55.874976    6608 vxlan_network_windows.go:123] Subnet added: 10.233.68.0/24 [67344095300 ns]
I0120 11:21:21.508992    6608 vxlan_network_windows.go:123] Subnet added: 10.233.64.0/24 [85630857700 ns]

net-conf.json

  "Network": "10.233.64.0/18",
  "Backend": {
    "name": "vxlan0",
    "type": "vxlan",
    "vni": 4096
  }
}

P.S. I'll test host-gw soon.

dmitryzykov · 2018-01-21T14:47:18Z

@rakelkar I tried host-gw (CNI from latest https://github.com/rakelkar/plugins/ windowsCni branch, flannel from #921 and latest gonetsh from https://github.com/rakelkar/gonetsh) and I got this errors in kubelet at windows node:

E0121 05:33:43.353929    1504 cni.go:259] Error adding network: network cbr0 is of an unexpected type: l2bridge
I0121 05:33:43.356931    1504 cni_windows.go:47] GetPodNetworkStatus result <nil>
E0121 05:33:43.356931    1504 cni_windows.go:49] error while adding to cni network: network cbr0 is of an unexpected type: l2bridge
W0121 05:33:43.356931    1504 docker_sandbox.go:340] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "win-webserver-64bf4ffc9d-fn9pn_default": network cbr0 is of an unexpected type: l2bridge
I0121 05:33:43.356931    1504 plugins.go:426] Calling network plugin cni to tear down pod "win-webserver-64bf4ffc9d-fn9pn_default"

flanneld output:

PS C:\k> .\flanneld.exe  --kubeconfig-file="c:\k\admin.conf" --iface="10.2.51.61" --ip-masq=1 --kube-subnet-mgr
=1
>>
I0121 05:25:19.341026    5708 main.go:417] Searching for interface using 10.2.51.61
I0121 05:25:24.353253    5708 main.go:488] Using interface with name Ethernet and address 10.2.51.61
I0121 05:25:24.353253    5708 main.go:505] Defaulting external address to interface address (10.2.51.61)
I0121 05:25:24.385279    5708 kube.go:131] Waiting 10m0s for node controller to sync
I0121 05:25:24.386279    5708 kube.go:294] Starting kube subnet manager
I0121 05:25:25.386575    5708 kube.go:138] Node controller sync successful
I0121 05:25:25.386575    5708 main.go:235] Created subnet manager: Kubernetes Subnet Manager - windows1
I0121 05:25:25.386575    5708 main.go:238] Installing signal handlers
I0121 05:25:25.387576    5708 main.go:353] Found network config - Backend type: host-gw
I0121 05:25:25.693582    5708 hostgw_windows.go:148] Attempting to create HNS network, request: {"DNSServerList":"","Name":"cbr0","Subnets":[{"AddressPrefix":"10.233.69.0/24","GatewayAddress":"10.233.69.1"}],"Type":"l2bridge"}
E0121 05:25:27.397324    5708 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.2.51.61:49731->10.2.51.51:6443: wsarecv: An established connection was aborted by the software in your host machine.
E0121 05:25:27.401242    5708 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to watch *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=2231&timeoutSeconds=582&watch=true: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable network.
E0121 05:25:28.402557    5708 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
I0121 05:25:28.547573    5708 hostgw_windows.go:156] Created HNS network [cbr0] as &{Id:0391977a-dc64-4c66-8c2f-3b27f06c2ac4 Name:cbr0 Type:l2bridge NetworkAdapterName: SourceMac: Policies:[] MacPools:[{StartMacAddress:00-15-5D-E2-D0-00 EndMacAddress:00-15-5D-E2-DF-FF}] Subnets:[{AddressPrefix:10.233.69.0/24 GatewayAddress:10.233.69.1 Policies:[]}] DNSSuffix: DNSServerList: DNSServerCompartment:0 ManagementIP: AutomaticDNS:false}
I0121 05:25:28.557564    5708 hostgw_windows.go:185] Attempting to create HNS endpoint [&{Id: Name:cbr0_ep VirtualNetwork:0391977a-dc64-4c66-8c2f-3b27f06c2ac4 VirtualNetworkName: Policies:[] MacAddress: IPAddress:10.233.69.2 DNSSuffix: DNSServerList: GatewayAddress: EnableInternalDNS:false DisableICC:false PrefixLength:0 IsRemoteEndpoint:false}]
I0121 05:25:28.566559    5708 hostgw_windows.go:191] Created bridge endpoint [cbr0_ep] as &{Id:aab73359-c58e-4738-be4e-c9f90d41bdcb Name:cbr0_ep VirtualNetwork:0391977a-dc64-4c66-8c2f-3b27f06c2ac4 VirtualNetworkName:cbr0 Policies:[[123 32 34 84 121 112 101 34 32 58 32 34 76 50 68 114 105 118 101 114 34 32 125]] MacAddress:00-15-5D-E2-DF-4C IPAddress:10.233.69.2 DNSSuffix: DNSServerList: GatewayAddress:10.233.69.1 EnableInternalDNS:false DisableICC:false PrefixLength:24 IsRemoteEndpoint:false}
I0121 05:25:28.782560    5708 hostgw_windows.go:196] Attached bridge endpoint [cbr0_ep] to host
I0121 05:25:29.198563    5708 hostgw_windows.go:210] Enabled forwarding on [vEthernet (Ethernet)] index [3]
E0121 05:25:29.418568    5708 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A socket operation was attempted to an unreachable host.
I0121 05:25:29.816580    5708 hostgw_windows.go:210] Enabled forwarding on [vEthernet (cbr0_ep)] index [13]
I0121 05:25:29.821574    5708 main.go:300] Wrote subnet file to /run/flannel/subnet.env
I0121 05:25:29.821574    5708 main.go:304] Running backend.
I0121 05:25:29.823576    5708 main.go:322] Waiting for all goroutines to exit
I0121 05:25:29.823576    5708 hostgw_network_windows.go:52] Watching for new subnet leases
I0121 05:25:29.872577    5708 hostgw_network_windows.go:86] Subnet added: 10.233.66.0/24 via 10.2.51.66
W0121 05:25:29.872577    5708 hostgw_network_windows.go:89] Ignoring non-host-gw subnet: type=vxlan
I0121 05:25:31.476818    5708 hostgw_network_windows.go:86] Subnet added: 10.233.67.0/24 via 10.2.51.51
W0121 05:25:31.476818    5708 hostgw_network_windows.go:89] Ignoring non-host-gw subnet: type=vxlan
I0121 05:25:32.513120    5708 hostgw_network_windows.go:86] Subnet added: 10.233.68.0/24 via 10.2.51.65
W0121 05:25:32.513120    5708 hostgw_network_windows.go:89] Ignoring non-host-gw subnet: type=vxlan
I0121 05:25:33.310125    5708 hostgw_network_windows.go:86] Subnet added: 10.233.64.0/24 via 10.2.51.52
W0121 05:25:33.311125    5708 hostgw_network_windows.go:89] Ignoring non-host-gw subnet: type=vxlan
I0121 05:25:34.094133    5708 hostgw_network_windows.go:86] Subnet added: 10.233.65.0/24 via 10.2.51.64
W0121 05:25:34.094133    5708 hostgw_network_windows.go:89] Ignoring non-host-gw subnet: type=vxlan
E0121 05:25:51.422050    5708 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:295: Failed to list *v1.Node: Get https://10.2.51.51:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.2.51.51:6443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

my configs:

-bash-4.2# cat cni-conf.json
{
   "name": "cbr0",
   "type": "flannel",
   "delegate": {
      "type": "win-l2bridge"
   }
}
-bash-4.2# cat net-conf.json
{
  "Network": "10.233.64.0/18",
  "Backend": {
    "name": "cbr0",
    "type": "host-gw",
    "vni": 4096
  }
}

===
Then for test I've renamed cbr0 to br0 and win-l2bridge to l2bridge in confs, removed cbr0 by Get-HNSNetwork|Remove-HNSNetwork and for sure rebooted windows node
but flannel creates cbr0, not br0 as expected and kubelet is trying to reach br0 and can't find it.

I0121 06:17:55.575883    6420 main.go:353] Found network config - Backend type: host-gw
I0121 06:17:55.616888    6420 hostgw_windows.go:148] Attempting to create HNS network, request: {"DNSServerList":"","Name":"cbr0","Subnets":[{"AddressPrefix":"10.233.69.0/24","GatewayAddress":"10.233.69.1"}],"Type":"l2bridge"}

ActivityId             : 715b7752-48e6-40da-9f7f-7121df2553b7
CurrentEndpointCount   : 1
DNSServerList          :
Extensions             : {@{Id=e7c3b2f0-f3c5-48df-af2b-10fed6d72e7a; IsEnabled=False},
                         @{Id=e9b59cfa-2be1-4b21-828f-b6fbdbddc017; IsEnabled=True},
                         @{Id=ea24cd6c-d17a-4348-9190-09f0d5be83dd; IsEnabled=False}}
ID                     : 312e9973-24a3-4b54-8c32-2efddc3fbe52
LayeredOn              : 7ff7d9b1-35f8-4352-92c4-256869f173ce
MacPools               : {@{EndMacAddress=00-15-5D-69-3F-FF; StartMacAddress=00-15-5D-69-30-00}}
ManagementIP           : 10.2.51.61
MaxConcurrentEndpoints : 1
Name                   : cbr0
Policies               : {}
Resources              : @{AllocationOrder=0; ID=715b7752-48e6-40da-9f7f-7121df2553b7; PortOperationTime=0; State=1;
                         SwitchOperationTime=0; VfpOperationTime=0; parentId=d1d08645-78ca-4b7b-a653-6bab06aab2a2}
State                  : 1
Subnets                : {@{AddressPrefix=10.233.69.0/24; GatewayAddress=10.233.69.1}}
TotalEndpoints         : 1
Type                   : l2bridge
Version                : 21474836481

===
then I renamed back br0 to cbr0 and keep l2bridge without win- prefix, reset networks.
And i have the same error like at the beginning:

I0121 06:43:17.360215    6944 cni.go:284] Got netns path none
I0121 06:43:17.360215    6944 cni.go:285] Using podns path default
I0121 06:43:17.360215    6944 cni.go:274] About to del CNI network cbr0 (type=flannel)
I0121 06:43:17.370230    6944 kuberuntime_manager.go:641] Creating sandbox for pod "kube-flannel-5rlm9_kube-system(71b3b82d-feae-11e7-92e6-00155d01070e)"
I0121 06:43:17.377221    6944 docker_service.go:441] Setting cgroup parent to: ""
E0121 06:43:17.502213    6944 cni.go:259] Error adding network: network cbr0 is of an unexpected type: l2bridge
I0121 06:43:17.502213    6944 cni_windows.go:47] GetPodNetworkStatus result <nil>
E0121 06:43:17.504217    6944 cni_windows.go:49] error while adding to cni network: network cbr0 is of an unexpected type: l2bridge
W0121 06:43:17.505221    6944 docker_sandbox.go:340] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "win-webserver-64bf4ffc9d-wlt8n_default": network cbr0 is of an unexpected type: l2bridge
I0121 06:43:17.505221    6944 plugins.go:426] Calling network plugin cni to tear down pod "win-webserver-64bf4ffc9d-wlt8n_default"

nlowe · 2018-05-07T14:46:57Z

Is there more up-to-date documentation for testing this? I was able to join a windows node to my cluster in VXLAN mode and even deploy a windows pod to it. The pod starts up but has no network connectivity. I can't ping the gateway, cluster DNS, or any internal / external services, but I can get pod logs and an exec shell from my workstation or one of my nodes via kubectl. Linux pods successfully resolve the IP of the service exposing the windows pods but can't interact with them.

jinxiao · 2018-05-15T07:25:56Z

@nlowe i think i have the same problem, and the kubelet logs showed that :

Calling network plugin cni to set up pod "win-webserver-7b87fb99b4-q9nld_default"
I0515 15:12:30.174829 9952 cni.go:284] Got netns path none
I0515 15:12:30.174829 9952 cni.go:285] Using podns path default
I0515 15:12:30.175692 9952 cni.go:256] About to add CNI network cbr0 (type=flannel)
E0515 15:12:30.327684 9952 cni.go:259] Error adding network: unsupported platform request
E0515 15:12:30.327684 9952 cni.go:227] Error while adding to cni network: unsupported platform request
I0515 15:12:30.369685 9952 kubelet.go:2102] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=true reason: message:

seanknox · 2018-06-27T00:47:45Z

Is there more up-to-date documentation for testing this? I was able to join a windows node to my cluster in VXLAN mode and even deploy a windows pod to it. The pod starts up but has no network connectivity. I can't ping the gateway, cluster DNS, or any internal / external services, but I can get pod logs and an exec shell from my workstation or one of my nodes via kubectl. Linux pods successfully resolve the IP of the service exposing the windows pods but can't interact with them.

Is IP forwarding enabled on the nodes and masters?

atomaras · 2018-11-01T19:29:10Z

We are having problems with flannel on windows. Flannel leaks IPs and I am forced to cleanup the cni folder and rejoin my windows nodes. We are also getting a bunch of CNI errors about not being able to remove endpoints and finally service discovery stops working.

stale · 2023-01-26T13:04:59Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

rakelkar added a commit to rakelkar/flannel that referenced this issue Oct 5, 2017

backend: Support host-gw backed on Windows

9eb1ff4

This uses new Windows Host Networking Service (HNS) features to configure networking for the host-gw mode Fixes flannel-io#833

rakelkar added a commit to rakelkar/flannel that referenced this issue Oct 5, 2017

backend: Support host-gw backed on Windows

4e73b4c

This uses new Windows Host Networking Service (HNS) features to configure networking for the host-gw mode Fixes flannel-io#833

rakelkar changed the title ~~Support Windows in L2Bridge (host-gw) mode~~ Support Windows in host-gw and VXLAN modes Oct 18, 2017

tomdee added the platform/windows label Oct 20, 2017

tomdee added the kind/enhancement label Nov 3, 2017

This was referenced Nov 28, 2017

flannel: Get flannel running on windows #891

Merged

network: move ipmasq management into platform-specific files #854

Merged

backend: Get flannel building on windows with stubs #879

Merged

This was referenced Jan 16, 2018

host-gw: add windows support #921

Closed

vxlan: add windows support #922

Closed

tomdee added platform/windows and removed platform/windows labels Oct 5, 2018

stale bot added the wontfix label Jan 26, 2023

stale bot closed this as completed Feb 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Windows in host-gw and VXLAN modes #833

Support Windows in host-gw and VXLAN modes #833

rakelkar commented Oct 4, 2017 •

edited

Loading

rakelkar commented Oct 18, 2017

rakelkar commented Oct 18, 2017

cspwizard commented Dec 6, 2017

rakelkar commented Dec 6, 2017 via email

rjmorse commented Dec 6, 2017

cspwizard commented Dec 6, 2017

rakelkar commented Dec 7, 2017

dmitryzykov commented Jan 15, 2018

rakelkar commented Jan 16, 2018

rakelkar commented Jan 16, 2018 •

edited

Loading

rakelkar commented Jan 16, 2018

dmitryzykov commented Jan 16, 2018 •

edited

Loading

dmitryzykov commented Jan 17, 2018

rakelkar commented Jan 18, 2018

rakelkar commented Jan 18, 2018

rakelkar commented Jan 18, 2018

rakelkar commented Jan 19, 2018

dmitryzykov commented Jan 20, 2018

dmitryzykov commented Jan 21, 2018

nlowe commented May 7, 2018

jinxiao commented May 15, 2018 •

edited

Loading

seanknox commented Jun 27, 2018

atomaras commented Nov 1, 2018 •

edited

Loading

stale bot commented Jan 26, 2023

Support Windows in host-gw and VXLAN modes #833

Support Windows in host-gw and VXLAN modes #833

Comments

rakelkar commented Oct 4, 2017 • edited Loading

Possible Solution

Your Environment

rakelkar commented Oct 18, 2017

rakelkar commented Oct 18, 2017

cspwizard commented Dec 6, 2017

rakelkar commented Dec 6, 2017 via email

rjmorse commented Dec 6, 2017

cspwizard commented Dec 6, 2017

rakelkar commented Dec 7, 2017

dmitryzykov commented Jan 15, 2018

rakelkar commented Jan 16, 2018

rakelkar commented Jan 16, 2018 • edited Loading

rakelkar commented Jan 16, 2018

dmitryzykov commented Jan 16, 2018 • edited Loading

dmitryzykov commented Jan 17, 2018

rakelkar commented Jan 18, 2018

rakelkar commented Jan 18, 2018

rakelkar commented Jan 18, 2018

rakelkar commented Jan 19, 2018

dmitryzykov commented Jan 20, 2018

dmitryzykov commented Jan 21, 2018

nlowe commented May 7, 2018

jinxiao commented May 15, 2018 • edited Loading

seanknox commented Jun 27, 2018

atomaras commented Nov 1, 2018 • edited Loading

stale bot commented Jan 26, 2023

rakelkar commented Oct 4, 2017 •

edited

Loading

rakelkar commented Jan 16, 2018 •

edited

Loading

dmitryzykov commented Jan 16, 2018 •

edited

Loading

jinxiao commented May 15, 2018 •

edited

Loading

atomaras commented Nov 1, 2018 •

edited

Loading