Unable to create transparent network with defined subnet on nested Windows Server Core host

[riverar]

Description

Attempting to create a transparent network with a user-defined subnet on a nested Windows Server Core Insider Build 16278 host (virtual machine) results in obscure error.

Handler for POST /v1.30/networks/create returned error: HNS failed with error:
The object already exists.

Verified the host doesn't have any networks created with overlapping subnets. I faintly recall hearing nested hosts only support nat but I can't seem to find docs on this. At any rate, would expect a slightly friendlier error.

Steps to reproduce the issue:

1. Spin up WSC 16278 in Hyper-V w/ nested virtualization enabled
2. docker network create -d transparent --subnet x.x.x.x repro
3. Note failure in daemon logs

Describe the results you received:
Docker log snippet:

time="2017-09-08T01:32:54.315353500-07:00" level=debug msg="Calling POST /v1.30/networks/create"
time="2017-09-08T01:32:54.316363300-07:00" level=debug msg="form data: {\"Attachable\":false,\"CheckDuplicate\":true,\"ConfigFrom\":null,\"ConfigOnly\":false,\"Driver\":\"transparent\",\"EnableIPv6\":false,\"IPAM\":{\"Config\":[{\"Subnet\":\"192.168.1.0/24\"}],\"Driver\":\"default\",\"Options\":{}},\"Ingress\":false,\"Internal\":false,\"Labels\":{},\"Name\":\"external\",\"Options\":{\"com.docker.network.windowsshim.dnssuffix\":\"corp.redacted\"},\"Scope\":\"\"}"
time="2017-09-08T01:32:54.321342600-07:00" level=debug msg="Allocating IPv4 pools for network external (b8e7f7e5552a3101dd033399c00eccfb488e064d19470065c311d000c9248822)"
time="2017-09-08T01:32:54.322348400-07:00" level=debug msg="RequestPool(LocalDefault, 192.168.1.0/24, , map[], false)"
time="2017-09-08T01:32:54.322348400-07:00" level=debug msg="RequestAddress(192.168.1.0/24, <nil>, map[RequestAddressType:com.docker.network.gateway])"
time="2017-09-08T01:32:54.325331500-07:00" level=debug msg="HNSNetwork Request ={\"Name\":\"b8e7f7e5552a3101dd033399c00eccfb488e064d19470065c311d000c9248822\",\"Type\":\"transparent\",\"Subnets\":[{\"AddressPrefix\":\"192.168.1.0/24\",\"GatewayAddress\":\"192.168.1.0\"}],\"DNSSuffix\":\"corp.redacted\"} Address Space=[{192.168.1.0/24 192.168.1.0 []}]"
time="2017-09-08T01:32:54.326347000-07:00" level=debug msg="[POST]=>[/networks/] Request : {\"Name\":\"b8e7f7e5552a3101dd033399c00eccfb488e064d19470065c311d000c9248822\",\"Type\":\"transparent\",\"Subnets\":[{\"AddressPrefix\":\"192.168.1.0/24\",\"GatewayAddress\":\"192.168.1.0\"}],\"DNSSuffix\":\"corp.redacted\"}"
time="2017-09-08T01:32:55.649081600-07:00" level=debug msg="releasing IPv4 pools from network external (b8e7f7e5552a3101dd033399c00eccfb488e064d19470065c311d000c9248822)"
time="2017-09-08T01:32:55.649081600-07:00" level=debug msg="ReleaseAddress(192.168.1.0/24, 192.168.1.0)"
time="2017-09-08T01:32:55.684066400-07:00" level=debug msg="ReleasePool(192.168.1.0/24)"
time="2017-09-08T01:32:55.685080000-07:00" level=error msg="Handler for POST /v1.30/networks/create returned error: HNS failed with error : The object already exists. "

Event Log:

HNS-Network-Create :- 
 Network id = '{28c409af-0c3d-4d6e-b416-fa0fe8d12ad5}'.
 Network type = 'TRANSPARENT'.
  Result code = '0x80071392'. 

Trace snippet:

547	None	2017-09-08T00:53:25.9587655	0.0000021	5820	2816	Microsoft_Windows_Host_Network_Service	Enter function: Function=HNS::Service::Network::BaseNetwork::StringToType,Line=69	
548	None	2017-09-08T00:53:25.9587678	0.0000023	5820	2816	Microsoft_Windows_Host_Network_Service	Exit function: Function=HNS::Service::Network::BaseNetwork::StringToType,Line=75,success=0	
549	None	2017-09-08T00:53:25.9587697	0.0000019	5820	2816	Microsoft_Windows_Host_Network_Service	Exit function: Function=HNS::Service::Network::BaseNetwork::GetNetworkType,Line=216,success=0	
550	None	2017-09-08T00:53:25.9587713	0.0000016	5820	2816	Microsoft_Windows_Host_Network_Service	Exit function: Function=HNS::Service::Network::BaseNetwork::GetNetworkType,Line=114,success=0	
551	None	2017-09-08T00:53:25.9587731	0.0000018	5820	2816	Microsoft_Windows_Host_Network_Service	HNSNetworkCreated: networkId=0953129c-8323-4737-b995-ac663a601c0c,networkType=1,resultCode=-2147019886	
552	None	2017-09-08T00:53:25.9587774	0.0000043	5820	2816	Etw	{bbccf6c1-6cd1-48c4-80ff-839482e37671}, EventID: 0, ProcessID: 5820, ThreadID: 2816, Size: 1310	
553	None	2017-09-08T00:53:25.9587774	0.0000000	5820	2816	Microsoft_Windows_Host_Network_Service	HNS-Network-Create :- 
 Network id = '0953129c-8323-4737-b995-ac663a601c0c'.
 Network type = 'TRANSPARENT'.
  Result code = '0x80071392'.	

Describe the results you expected:
Success, or a documented error about transparent networks not being supported in nested scenarios. (Is this a thing?)

Output of docker version:

Client:
 Version:      17.07.0-ce-rc1
 API version:  1.30 (downgraded from 1.31)
 Go version:   go1.8.3
 Git commit:   8c4be39
 Built:        Wed Jul 26 05:19:44 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.1-ee-2
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   8e43158
 Built:        Wed Aug 23 21:25:53 2017
 OS/Arch:      windows/amd64
 Experimental: false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 16
Server Version: 17.06.1-ee-2
Storage Driver: windowsfilter
 Windows: 
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 16278 (16278.1000.amd64fre.rs3_release.170825-1441)
Operating System: Windows Server Datacenter
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 3.999GiB
Name: containerhost-insider
ID: [redacted]
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 19
 System Time: 2017-09-08T00:47:31.1961294-07:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Hyper-V host: Windows Server 2016 14393.1593
Container host, virtualized: Windows Server Core Insider 16278.1000

cc: @jhowardmsft @PatrickLang @neilpeterson @msabansal

[riverar]

Upon closer investigation, appears HNS in this scenario was attempting to set a default gateway matching my subnet (e.g. 192.168.1.0). Not sure what that's about (sounds like a strange default) but explicitly specifying the --gateway resolves the issue.

[msabansal]

@riverar Oh ya i actually found out that this doesn't work without specifying the gateway very recently :(. Unfortunately it was too late in the development cycle to fix it for this milestone. Will get it fixed in the next milestone.

[thaJeztah]

@msabansal is that an issue in Windows, or Docker?

[msabansal]

Windows fix.

[riverar]

Did this make it into RS4 or?

[riverar]

Did this make it into RS5 or?

I ask because now it seems to have gotten worse. Creating a network with --gateway= specified results in:

DEBU[2019-01-07T21:46:58.321797300-08:00] Binding a resolver on network test gateway 192.168.1.1
ERRO[2019-01-07T21:46:58.322798300-08:00] Resolver Setup/Start failed for container test, "error in opening name server socket listen udp 192.168.1.1:53: bind: The requested address is not valid in its context."
DEBU[2019-01-07T21:46:59.325438400-08:00] Binding a resolver on network test gateway 192.168.1.1
ERRO[2019-01-07T21:46:59.325438400-08:00] Resolver Setup/Start failed for container test, "error in opening name server socket listen udp 192.168.1.1:53: bind: The requested address is not valid in its context."
DEBU[2019-01-07T21:47:00.342605200-08:00] Binding a resolver on network test gateway 192.168.1.1
ERRO[2019-01-07T21:47:00.342605200-08:00] Resolver Setup/Start failed for container test, "error in opening name server socket listen udp 192.168.1.1:53: bind: The requested address is not valid in its context."

The network does get created and seems to work. More testing required.
(Windows Server 1809 17763.195)

[thaJeztah]

@madhanrm perhaps you know?