Azure CNI never uses HNSv2 with Docker on Windows.

[adelina-t]

In AzureCNI, the check on weather to use HNS v1 or v2 is done with this helper function:

azure-container-networking/network/network_windows.go

Lines 41 to 56 in aed0d71

	// UseHnsV2 indicates whether to use HNSv1 or HNSv2
	// HNSv2 should be used if the NetNs is a valid GUID and if the platform
	// has HCN which supports HNSv2 API.
	func UseHnsV2(netNs string) (bool, error) {
	// Check if the netNs is a valid GUID to decide on HNSv1 or HNSv2
	useHnsV2 := false
	var err error
	if _, err = uuid.Parse(netNs); err == nil {
	useHnsV2 = true
	if err = hcn.V2ApiSupported(); err != nil {
	log.Printf("HNSV2 is not supported on this windows platform")
	}
	}
	return useHnsV2, err
	}

Notice that this first parses the netNS param to check if it's a GUID and only then checks if the platform supports HNSv2.

The netNS param when UseHnsV2 is called is when creating a new endpoint is epInfo.NetNsPath which we get from the CNI call args:

azure-container-networking/cni/network/network.go

Line 582 in eba4207

NetNsPath: args.Netns,

Problem is that in the case of docker, netns will be None the first time CNI is called to create the endpoint and thus UseHnsV2 will always return false. The subsequent CNI Add calls will have a netns specified, but by that point it is irelevant as the endpoint has already been created.

For example, consider this bit from azure-cni logs here:

We get the first ADD command, and you can clearly see that Netns is empty.Notice ContainerID, this is the InfraContainer.

2020/10/22 17:48:45 [784] [cni-net] Processing ADD command with args {ContainerID:6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8 Netns:none IfName:eth0
Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx;K8S_POD_INFRA_CONTAINER_ID=6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8 Path:c:\k\azurecni\bin StdinData:{"AdditionalArgs
":[{"Name":"EndpointPolicy","Value":{"ExceptionList":["10.240.0.0/12","10.240.0.0/12","10.0.0.0/8"],"Type":"OutBoundNAT"}},{"Name":"EndpointPolicy","Value":{"DestinationPrefix":"10.0.0.0/16","NeedEncap":true
,"Type":"ROUTE"}}],"bridge":"azure0","capabilities":{"dns":true,"portMappings":true},"cniVersion":"0.3.0","dns":{"Nameservers":["10.0.0.10","168.63.129.16"],"Search":["svc.cluster.local"]},"ipam":{"type":"az
ure-vnet-ipam"},"mode":"bridge","name":"azure","runtimeConfig":{"dns":{"servers":["10.0.0.10"],"searches":["default.svc.cluster.local","svc.cluster.local","cluster.local"],"options":["ndots:5"]},"portMapping
s":[{"hostPort":8123,"containerPort":80,"protocol":"tcp","hostIP":"127.0.0.1"}]},"type":"azure-vnet"}}.

CNI creates the endpoint, note NetNsPath:none. Thus, it will use HNSv1.

DNS:{Suffix:default.svc.cluster.local,svc.cluster.local,cluster.local Servers:[10.0.0.10] Options:[ndots:5]} IPAddresses:[{IP:10.240.0.50 Mask:fff00000}] IPsToRouteViaHost:[] InfraVnetIP:{IP:<nil>
Mask:<nil>} Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} Src:<nil> Gw:10.240.0.1 Protocol:0 DevName: Scope:0 Priority:0}] Policies:[{Type:EndpointPolicy Data:[123 34 69 120 99 101 112 116 105 111 110 76 105 115
116 34 58 91 34 49 48 46 50 52 48 46 48 46 48 47 49 50 34 44 34 49 48 46 50 52 48 46 48 46 48 47 49 50 34 44 34 49 48 46 48 46 48 46 48 47 56 34 93 44 34 84 121 112 101 34 58 34 79 117 116 66 111 117 110
100 78 65 84 34 125]} {Type:EndpointPolicy Data:[123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 48 46 48 46 48 46 48 47 49 54 34 44 34 78 101 101 100 69 110 99 97 112 34
58 116 114 117 101 44 34 84 121 112 101 34 58 34 82 79 85 84 69 34 125]} {Type:EndpointPolicy Data:[123 34 84 121 112 101 34 58 34 78 65 84 34 44 34 80 114 111 116 111 99 111 108 34 58 34 116 99 112 34 44
34 73 110 116 101 114 110 97 108 80 111 114 116 34 58 56 48 44 34 69 120 116 101 114 110 97 108 80 111 114 116 34 58 56 49 50 51 44 34 86 73 80 34 58 34 49 50 55 46 48 46 48 46 49 34 125]}] Gateways:[]
EnableSnatOnHost:false EnableInfraVnet:false EnableMultiTenancy:false EnableSnatForDns:false AllowInboundFromHostToNC:false AllowInboundFromNCToHost:false NetworkContainerID: PODName:nginx
PODNameSpace:default Data:map[] InfraVnetAddressSpace: SkipHotAttachEp:false IPV6Mode: VnetCidrs: ServiceCidrs:} in network azure.

2020/10/22 17:48:46 [784] [net] Created endpoint &{Id:6ad647ea-eth0 HnsId:7496f7e9-3409-4cb4-9486-e89523d977a1 SandboxKey:6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8 IfName:eth0
HostIfName: MacAddress:00:15:5d:d2:6c:f8 InfraVnetIP:{IP:<nil> Mask:<nil>} LocalIP: IPAddresses:[{IP:10.240.0.50 Mask:fff00000}] Gateways:[10.240.0.1]
DNS:{Suffix:default.svc.cluster.local,svc.cluster.local,cluster.local Servers:[10.0.0.10] Options:[ndots:5]} Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} Src:<nil> Gw:10.240.0.1 Protocol:0 DevName: Scope:0
Priority:0}] VlanID:0 EnableSnatOnHost:false EnableInfraVnet:false EnableMultitenancy:false AllowInboundFromHostToNC:false AllowInboundFromNCToHost:false NetworkContainerID: NetworkNameSpace: ContainerID:
PODName: PODNameSpace: InfraVnetAddressSpace: NetNs:none}.

Subsequent ADD command for for InfraContainer. NetNs is again empty.

2020/10/22 17:48:47 [1920] [cni-net] Processing ADD command with args {ContainerID:6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8 Netns:none IfName:eth0
Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx;K8S_POD_INFRA_CONTAINER_ID=6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8 Path:c:\k\azurecni\bin StdinData:{"AdditionalArgs
":[{"Name":"EndpointPolicy","Value":{"ExceptionList":["10.240.0.0/12","10.240.0.0/12","10.0.0.0/8"],"Type":"OutBoundNAT"}},{"Name":"EndpointPolicy","Value":{"DestinationPrefix":"10.0.0.0/16","NeedEncap":true
,"Type":"ROUTE"}}],"bridge":"azure0","capabilities":{"dns":true,"portMappings":true},"cniVersion":"0.3.0","dns":{"Nameservers":["10.0.0.10","168.63.129.16"],"Search":["svc.cluster.local"]},"ipam":{"type":"az
ure-vnet-ipam"},"mode":"bridge","name":"azure","runtimeConfig":{"portMappings":[{"hostPort":8123,"containerPort":80,"protocol":"tcp","hostIP":"127.0.0.1"}]},"type":"azure-vnet"}}.

ADD command for WorkContainer. Finally it has a netns referencing the InfraContainer, as expected.

2020/10/22 17:48:55 [3776] [cni-net] Processing ADD command with args {ContainerID:f533175419b952dc0ced8e891e5dd6766c0e10076144d8c3b0e3c38b9172cc9f
Netns:container:6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8 IfName:eth0
Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx;K8S_POD_INFRA_CONTAINER_ID=f533175419b952dc0ced8e891e5dd6766c0e10076144d8c3b0e3c38b9172cc9f Path:c:\k\azurecni\bin StdinData:{"AdditionalArgs
":[{"Name":"EndpointPolicy","Value":{"ExceptionList":["10.240.0.0/12","10.240.0.0/12","10.0.0.0/8"],"Type":"OutBoundNAT"}},{"Name":"EndpointPolicy","Value":{"DestinationPrefix":"10.0.0.0/16","NeedEncap":true
,"Type":"ROUTE"}}],"bridge":"azure0","capabilities":{"dns":true,"portMappings":true},"cniVersion":"0.3.0","dns":{"Nameservers":["10.0.0.10","168.63.129.16"],"Search":["svc.cluster.local"]},"ipam":{"type":"az
ure-vnet-ipam"},"mode":"bridge","name":"azure","runtimeConfig":{"portMappings":[]},"type":"azure-vnet"}}.

It's intuitive by the formating of netns we see in the logs where it exists, that it's expected to be a reference to the sandbox container ID. For confirmation, if we look exactly in the implementation for dockershim, we can see clearly how it is being set.

https://github.com/kubernetes/kubernetes/blob/433c3d57cca8bd59889e126df3e5e2975d530972/pkg/kubelet/dockershim/docker_service.go#L367-L376

And the helper function, here:

https://github.com/kubernetes/kubernetes/blob/fe1aeff2d2341e3d9a553534c814ad40f8219e35/pkg/kubelet/dockershim/helpers_windows.go#L178-L183

As you can see, it returns the HostConfig.NetworkMode field from the inspected container, which in the case of the Sandbox is None.

{
        "Id": "6ad647ea8218c1a666b10706209708fd2b9ac846fd7e7f4d1106a379da65bad8",
        "Created": "2020-10-22T17:48:42.847337Z",
        "Path": "cmd",
        "Args": [
            "/S",
            "/C",
            "pauseloop.exe"
        ],
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "5",
                    "max-size": "50m"
                }
            },
            "NetworkMode": "none"
            ...

The solution here would of course trivial, just remove the netns parsing check here:

azure-container-networking/network/network_windows.go

Lines 41 to 56 in aed0d71

	// UseHnsV2 indicates whether to use HNSv1 or HNSv2
	// HNSv2 should be used if the NetNs is a valid GUID and if the platform
	// has HCN which supports HNSv2 API.
	func UseHnsV2(netNs string) (bool, error) {
	// Check if the netNs is a valid GUID to decide on HNSv1 or HNSv2
	useHnsV2 := false
	var err error
	if _, err = uuid.Parse(netNs); err == nil {
	useHnsV2 = true
	if err = hcn.V2ApiSupported(); err != nil {
	log.Printf("HNSV2 is not supported on this windows platform")
	}
	}
	return useHnsV2, err
	}

However, I'm not quite sure I understand why it's done in the first place or if it actually is a necessary check.

This is blocking this PR here: #689

[adelina-t]

Since K8s sig-windows intends on transitioning to ContainerD for Windows Nodes, there is no need for this working with Dockershim. Closing.