WIP: Enable swarm integration tests for Windows

[olljanat]

https://docs.docker.com/ee/ucp/admin/configure/join-nodes/join-windows-nodes-to-cluster/ says

Only worker nodes are supported on Windows, and all manager nodes in the cluster must run on Linux.

but there looks to be some users who would like to use Swarm on Windows only environment (e.g. docker/for-win#1476 (comment) ) so let's try and see if we can get these integration tests working on Windows too.

Related to:

• integration package is not ran on the windowsRS1 CI #36748
• Windows: Start of enabling tests under integration #37715

TODO:

• Finalize and get Windows CI: Enable more integration tests #38469 merged.
• Make tests working
• Include tests on integration/config/config_test.go
• Include tests on integration/network/inspect_test.go
• Include tests on integration/network/service_test.go
• Include tests on integration/secret/secret_test.go
• Include tests on integration/service/...

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@3967999). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #38432   +/-   ##
=========================================
  Coverage          ?   36.54%           
=========================================
  Files             ?      608           
  Lines             ?    45040           
  Branches          ?        0           
=========================================
  Hits              ?    16458           
  Misses            ?    26300           
  Partials          ?     2282

[olljanat]

Closing this now because #38469 need to be finalized first and that cannot be done before build scripts are updated

[thaJeztah]

Fingers crossed 🤞

[olljanat]

@StefanScherer for some reason win-RS5 is missing from pipeline time to time. I see it with this one too https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-38432/12/pipeline

but after a while it can came back to visible...

[StefanScherer]

Hi @olljanat This is how the Jenkins blue ocean plugin draws the current pipeline.
It somehow uses a previous build to guess how the pipeline will look like, but it's interpreted step by step. I also see that when I add new stages then the first build doesn't show it until it's actually running.
You have to be patient to finally see it. 😅

[olljanat]

Short status update: Looks that I got needed logic implemented that on Windows we can run swarm tests against of daemon started by hack\ci\windows.ps1 but it looks to be that at least ingress removal from HNS during swarm leave is async which why Docker and HNS gets out of sync quite soon and tests starts failing.

I included ipconfig command to swarm init so it is easier see which state it is.

First then passes nicely:

[2020-09-21T00:01:37.606Z] === RUN   TestInspectNetwork
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] Windows IP Configuration
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] Ethernet adapter Ethernet 3:
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z]    Connection-specific DNS Suffix  . : 22lddyrsr24ufjkj5c0fzjspmc.dx.internal.cloudapp.net
[2020-09-21T00:01:37.606Z]    Link-local IPv6 Address . . . . . : fe80::b4d7:5d93:18e7:c872%14
[2020-09-21T00:01:37.606Z]    IPv4 Address. . . . . . . . . . . : 10.0.0.4
[2020-09-21T00:01:37.606Z]    Subnet Mask . . . . . . . . . . . : 255.255.255.0
[2020-09-21T00:01:37.606Z]    Default Gateway . . . . . . . . . : 10.0.0.1
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] Ethernet adapter vEthernet (nat):
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z]    Connection-specific DNS Suffix  . : 
[2020-09-21T00:01:37.606Z]    Link-local IPv6 Address . . . . . : fe80::2d6c:5d3d:62ad:d68e%10
[2020-09-21T00:01:37.606Z]    IPv4 Address. . . . . . . . . . . : 172.28.144.1
[2020-09-21T00:01:37.606Z]    Subnet Mask . . . . . . . . . . . : 255.255.240.0
[2020-09-21T00:01:37.606Z]    Default Gateway . . . . . . . . . : 
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] === RUN   TestInspectNetwork/full_network_id
[2020-09-21T00:01:37.606Z] === RUN   TestInspectNetwork/partial_network_id
[2020-09-21T00:01:37.606Z] === RUN   TestInspectNetwork/network_name
[2020-09-21T00:01:37.606Z] === RUN   TestInspectNetwork/network_name_and_swarm_scope
[2020-09-21T00:01:37.606Z] --- PASS: TestInspectNetwork (17.62s)
[2020-09-21T00:01:37.606Z]     --- PASS: TestInspectNetwork/full_network_id (0.01s)
[2020-09-21T00:01:37.606Z]     --- PASS: TestInspectNetwork/partial_network_id (0.02s)
[2020-09-21T00:01:37.606Z]     --- PASS: TestInspectNetwork/network_name (0.01s)
[2020-09-21T00:01:37.606Z]     --- PASS: TestInspectNetwork/network_name_and_swarm_scope (0.05s)

But already second one fails because earlier swarm leave was most probably still running and swarm init failed:

[2020-09-21T00:01:37.606Z] === RUN   TestServiceWithDefaultAddressPoolInit
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] Windows IP Configuration
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] Ethernet adapter Ethernet 3:
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z]    Connection-specific DNS Suffix  . : 22lddyrsr24ufjkj5c0fzjspmc.dx.internal.cloudapp.net
[2020-09-21T00:01:37.606Z]    Link-local IPv6 Address . . . . . : fe80::b4d7:5d93:18e7:c872%14
[2020-09-21T00:01:37.606Z]    IPv4 Address. . . . . . . . . . . : 10.0.0.4
[2020-09-21T00:01:37.606Z]    Subnet Mask . . . . . . . . . . . : 255.255.255.0
[2020-09-21T00:01:37.606Z]    Default Gateway . . . . . . . . . : 10.0.0.1
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] Ethernet adapter vEthernet (nat):
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z]    Connection-specific DNS Suffix  . : 
[2020-09-21T00:01:37.606Z]    Link-local IPv6 Address . . . . . : fe80::2d6c:5d3d:62ad:d68e%10
[2020-09-21T00:01:37.606Z]    IPv4 Address. . . . . . . . . . . : 172.28.144.1
[2020-09-21T00:01:37.606Z]    Subnet Mask . . . . . . . . . . . : 255.255.240.0
[2020-09-21T00:01:37.606Z]    Default Gateway . . . . . . . . . : 
[2020-09-21T00:01:37.606Z] 
[2020-09-21T00:01:37.606Z] --- FAIL: TestServiceWithDefaultAddressPoolInit (16.37s)
[2020-09-21T00:01:37.606Z]     service_test.go:420: timeout hit after 15s: running task count at 0 waiting for 1 (total tasks: 3)
[2020-09-21T00:01:37.606Z]     main_test.go:32: assertion failed: error is not nil: Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.: failed to remove network szhfjkzafqui9g0sszh247xcb
[2020-09-21T00:01:37.606Z] FAIL
[2020-09-21T00:01:37.606Z] FAIL	github.com/docker/docker/integration/network	34.222s

Then after a while whole "Ethernet 3" adapter is missing as ingress creation on Windows will re-create it and it is middle of that process:

[2020-09-21T00:01:37.608Z] === RUN   TestCreateWithDuplicateNetworkNames
[2020-09-21T00:01:37.608Z] 
[2020-09-21T00:01:37.608Z] Windows IP Configuration
[2020-09-21T00:01:37.608Z] 
[2020-09-21T00:01:37.608Z] 
[2020-09-21T00:01:37.608Z] Ethernet adapter vEthernet (nat):
[2020-09-21T00:01:37.608Z] 
[2020-09-21T00:01:37.608Z]    Connection-specific DNS Suffix  . : 
[2020-09-21T00:01:37.608Z]    Link-local IPv6 Address . . . . . : fe80::2d6c:5d3d:62ad:d68e%10
[2020-09-21T00:01:37.608Z]    IPv4 Address. . . . . . . . . . . : 172.28.144.1
[2020-09-21T00:01:37.608Z]    Subnet Mask . . . . . . . . . . . : 255.255.240.0
[2020-09-21T00:01:37.608Z]    Default Gateway . . . . . . . . . : 
[2020-09-21T00:01:37.608Z] 
[2020-09-21T00:01:37.608Z] --- FAIL: TestCreateWithDuplicateNetworkNames (0.05s)
[2020-09-21T00:01:37.608Z]     swarm_windows.go:20: assertion failed: error is not nil: Error response from daemon: could not find the system's IP address: initializing swarm

Now I'm using loopback adapter for swarm tests so tests does not cause Jenkins loosing access to server and increased daemon.ReleaseIngress() timeout to 60 seconds but still tests was failing even locally on my dev machine without that static 10 wait before swarm init which proofs that some parts are async.

As Linux tests are done differently by always starting new daemon it is very hard to say if this is just Windows specific or general issue but most probably need look Windows implementation on libnetwork is made...