Hns: POST after DELETE doesn't work without a delay #95

m-kostrzewa · 2016-12-13T16:10:40Z

Deleting a HNS network and then creating another one immediately after doesn't work. Here's the error message:

Expected error:
    <*errors.errorString | 0xc0422f18b0>: {
        s: "HNS failed with error : Element not found. ",
    }
    HNS failed with error : Element not found.
not to have occurred

Here's a test case:

subnets1 := []hcsshim.Subnet{
    {
        AddressPrefix:  "172.100.0.0/20",
        GatewayAddress: "172.100.0.1",
    },
}
configuration1 := &hcsshim.HNSNetwork{
    Name:    "TestNetworkName1",
    Type:    "transparent",
    Subnets: subnets1,
}

subnets2 := []hcsshim.Subnet{
    {
        AddressPrefix:  "172.200.0.0/20",
        GatewayAddress: "172.200.0.1",
    },
}
configuration2 := &hcsshim.HNSNetwork{
    Name:    "TestNetworkName2",
    Type:    "transparent",
    Subnets: subnets2,
}

It("doesn't work if there's no delay after DELETE", func() {
    configBytes1, err := json.Marshal(configuration1)
    Expect(err).ToNot(HaveOccurred())
    response, err := hcsshim.HNSNetworkRequest("POST", "", string(configBytes1))
    Expect(err).ToNot(HaveOccurred())
    hnsID := response.Id

    _, err = hcsshim.HNSNetworkRequest("DELETE", hnsID, "")
    Expect(err).ToNot(HaveOccurred())

    //time.Sleep(time.Second * 20) // 20 second timeout "fixes" the issue

    configBytes2, err := json.Marshal(configuration2)
    Expect(err).ToNot(HaveOccurred())
    response, err = hcsshim.HNSNetworkRequest("POST", "", string(configBytes2))
    Expect(err).To(HaveOccurred()) // !!! ERROR

    _, err = hcsshim.HNSNetworkRequest("GET", hnsID, "")
    Expect(err).To(HaveOccurred()) // but can't GET the deleted network either
})

Note that sleeping for 20 seconds after deleting a network seems to "fix" the issue. 10 second timeout is not enough.

update: I repeated this in Powershell, so this may be problem with HNS. I posted an issue here: MicrosoftDocs/Virtualization-Documentation#516

The text was updated successfully, but these errors were encountered:

msabansal · 2016-12-13T19:15:38Z

This is probably because of a delay in applying bindings to the external network adapter when the transparent network is deleted. Will look into this

msabansal · 2016-12-13T20:17:30Z

I tried this on an RS2 build and it doesn't seem to be a problem. I will try reproducing this on Windows Server 2016.

PS C:\Users\Administrator> 1..10 | % { Remove-ContainerNetwork -name transparent -Force; New-ContainerNetwork -name tra
nsparent -Mode Transparent; }

Name Id Subnets Mode SourceMac DNSServers DNSSuffix

transparent 53765384-11a1-42f7-ab32-28424273895f {} Transparent
transparent c2fbbf8d-9b9e-4a0d-aca9-c188dc2b1a7e {} Transparent
transparent f5dc263e-803f-482d-8246-5d81b262d7a6 {} Transparent
transparent e2c48589-e004-4bb6-9183-018dc4053d79 {} Transparent
transparent f31d1a78-41f0-4118-b2a2-59eef6ae3fc7 {} Transparent
transparent 6645bbf2-c0cf-47f0-91d3-8d7edd74c2f3 {} Transparent
transparent 04dc5700-6b85-44cc-b3b0-5444e2d8e26b {} Transparent
transparent e0974078-8602-4170-9858-27edd7249918 {} Transparent
transparent eaef6a0c-5ee0-4f4a-aee3-32a8ef39310f {} Transparent
transparent 55998ed7-e204-4d69-aee3-7e2651c0ff56 {} Transparent

m-kostrzewa · 2016-12-14T07:33:47Z

Forgot to mention, I'm using Server 2016.

msabansal · 2016-12-16T01:03:27Z

@m-kostrzewa Ok it seems that this is happening because our logic dictates binding transparent networks to Network adapters with IPv4 addresses. When the network is deleted it takes some time for the reinitialized adapter to acquire IP which leads to the failure as we can't select the adapter till it has acquired the IP.

One of the workaround would be to specify the network adapter by specifying the name using the com.docker.network.windowsshim.interface option

m-kostrzewa · 2016-12-16T08:59:34Z

PS > New-ContainerNetwork -Name net1 -Mode transparent -NetworkAdapterName Ethernet

Name Id                                   Subnets Mode        SourceMac DNSServers DNSSuffix
---- --                                   ------- ----        --------- ---------- ---------
net1 2f6e5b1f-dd62-4a2a-ab26-459452b82fb9 {}      Transparent


PS > Remove-ContainerNetwork -Name net1; New-ContainerNetwork -Name net2 -Mode transparent -NetworkAdapterName Ethernet

Confirm
Remove-ContainerNetwork will remove the container network "net1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y
New-ContainerNetwork : Element not found.
At line:1 char:37
+ ... -Name net1; New-ContainerNetwork -Name net2 -Mode transparent -Networ ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

PS > Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet                  Realtek PCIe FE Family Controller            17 Up           34-17-EB-93-15-C9       100 Mbps
Wi-Fi                     Intel(R) Dual Band Wireless-AC 3160          13 Not Present  F4-06-69-1E-80-34          0 bps

Hmm.. Can't get it to work...

m-kostrzewa · 2016-12-19T17:50:01Z

I did a similar experiment:

PS C:\Users\mk> New-ContainerNetwork -name net -Mode Transparent -NetworkAdapterName Ethernet; Remove-ContainerNetwork -name net; 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -
NetworkAdapterName Ethernet; Get-NetAdapter | format-table; Start-Sleep -s 1; }


Confirm
Remove-ContainerNetwork will remove the container network "net".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
Name Id                                   Subnets Mode        SourceMac DNSServers DNSSuffix
---- --                                   ------- ----        --------- ---------- ---------
net  126aff1f-42c0-415b-8e13-defed91895eb {}      Transparent
New-ContainerNetwork : Element not found.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork




Name                         InterfaceDescription              ifIndex Status       MacAddress        LinkSpeed
----                         --------------------              ------- ------       ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up           00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Disconnected 34-17-EB-93-15-C9     0 bps


New-ContainerNetwork : Element not found.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status       MacAddress        LinkSpeed
----                         --------------------              ------- ------       ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up           00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Disconnected 34-17-EB-93-15-C9     0 bps


New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps



Name Id                                   Subnets Mode        SourceMac DNSServers DNSSuffix
---- --                                   ------- ----        --------- ---------- ---------
net  91fb8fff-5810-489f-8f58-242873bbdd97 {}      Transparent



Name                         InterfaceDescription                ifIndex Status MacAddress        LinkSpeed
----                         --------------------                ------- ------ ----------        ---------
vEthernet (HNSTransparent)   Hyper-V Virtual Ethernet Adapter #2      25 Up     34-17-EB-93-15-C9  100 Mbps
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter         27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller         2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : {Object Exists} An attempt was made to create an object and the object name already existed.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork


Name                         InterfaceDescription                ifIndex Status MacAddress        LinkSpeed
----                         --------------------                ------- ------ ----------        ---------
vEthernet (HNSTransparent)   Hyper-V Virtual Ethernet Adapter #2      25 Up     34-17-EB-93-15-C9  100 Mbps
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter         27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller         2 Up     34-17-EB-93-15-C9  100 Mbps


New-ContainerNetwork : {Object Exists} An attempt was made to create an object and the object name already existed.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

So it seems like you get "element not found" when Ethernet is in Disconnected state, and "unspecified error" when it's in Up state, but has no IP yet?

@msabansal, could you describe in a little bit more detail how to perform the workaround you mentioned? This doesn't seem to work:

PS C:\Users\mk> docker network rm $(docker network ls -q)
Error response from daemon: nat is a pre-defined network and cannot be removed
Error response from daemon: none is a pre-defined network and cannot be removed
PS C:\Users\mk> docker network create -d transparent -o com.docker.network.windowsshim.interface="Ethernet" net1; docker network rm net1; docker network create -d transparent -o com.docker.network.windowsshim.interface="Ethernet" net2;
2368224e9344b4d7dd7efe5dfb2cd446b2fc83d75ff6d746156ba23cad5d8cdc
net1
Error response from daemon: HNS failed with error : Element not found.

natalieparellano · 2019-03-19T18:28:48Z

We ran into a similar issue with a similar reproduction. We encountered this on Windows Server 1709 and 1803, but it looks to be resolved in 2019.

Changing base layer order

m-kostrzewa mentioned this issue Dec 13, 2016

HNS Delete+Create transparent network MicrosoftDocs/Virtualization-Documentation#516

Closed

m-kostrzewa closed this as completed Dec 19, 2016

m-kostrzewa reopened this Dec 19, 2016

m-kostrzewa mentioned this issue Feb 15, 2017

connectex: A socket operation was attempted to an unreachable error AND HNS Unspecified error #108

Open

dcantah pushed a commit to dcantah/hcsshim that referenced this issue Mar 17, 2021

Merge pull request microsoft#95 from Microsoft/dns_resolv_conf

7f54f58

Changing base layer order

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hns: POST after DELETE doesn't work without a delay #95

Hns: POST after DELETE doesn't work without a delay #95

m-kostrzewa commented Dec 13, 2016 •

edited

Loading

msabansal commented Dec 13, 2016

msabansal commented Dec 13, 2016

m-kostrzewa commented Dec 14, 2016

msabansal commented Dec 16, 2016

m-kostrzewa commented Dec 16, 2016

m-kostrzewa commented Dec 19, 2016

natalieparellano commented Mar 19, 2019

Hns: POST after DELETE doesn't work without a delay #95

Hns: POST after DELETE doesn't work without a delay #95

Comments

m-kostrzewa commented Dec 13, 2016 • edited Loading

msabansal commented Dec 13, 2016

msabansal commented Dec 13, 2016

m-kostrzewa commented Dec 14, 2016

msabansal commented Dec 16, 2016

m-kostrzewa commented Dec 16, 2016

m-kostrzewa commented Dec 19, 2016

natalieparellano commented Mar 19, 2019

m-kostrzewa commented Dec 13, 2016 •

edited

Loading