VerifyWithHttpHealthCheckBlocksDependentResources failing consistently - could not create the network as all available subnet ranges from the default pool are allocated, retrying...

[danmoseley]

related? #7215

started failing consistently between d51f233..9b06baf

System.TimeoutException : The operation at //tests/Aspire.Hosting.Tests/HealthCheckTests.cs:88 timed out after reaching the limit of 120000ms.
Stack trace
at Microsoft.AspNetCore.InternalTesting.AsyncTestHelpers.TimeoutAfter(Task task, TimeSpan timeout, String filePath, Int32 lineNumber) in //tests/Shared/AsyncTestHelpers.cs:line 149
at Aspire.Hosting.Tests.HealthCheckTests.VerifyWithHttpHealthCheckBlocksDependentResources() in /_/tests/Aspire.Hosting.Tests/HealthCheckTests.cs:line 88
--- End of stack trace from previous location ---

[2025-02-10T20:34:49] Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler Error: could not create the network as all available subnet ranges from the default pool are allocated, retrying... {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 3, "Network": "default-aspire-network-lg6cnbqum4", "error": "docker command 'InspectNetworks' returned with non-zero exit code 1: command output: Stdout: '\n' Stderr: 'Error: No such network: default-aspire-network-lg6cnbqum4\n'\nobject could not allocate required resources\nnetwork subnet pool full"}
| [2025-02-10T20:34:52] Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler Error: could not create the network as all available subnet ranges from the default pool are allocated, retrying... {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 4, "Network": "default-aspire-network-0c3gsf10a0", "error": "docker command 'InspectNetworks' returned with non-zero exit code 1: command output: Stdout: '\n' Stderr: 'Error: No such network: default-aspire-network-0c3gsf10a0\n'\nobject could not allocate required resources\nnetwork subnet pool full"}

There was a healthcheck related change in this interval though, if the errors above are unrelated --
commit 849f74a Eventhubs health check (retargeted for main) (#4139)

@karolz-ms

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=954078
Build error leg or test failing: Aspire.Hosting.Tests.HealthCheckTests.VerifyWithHttpHealthCheckBlocksDependentResources
Pull request: N/A

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "could not create the network as all available subnet ranges from the default pool are allocated",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=954078
Error message validated: [could not create the network as all available subnet ranges from the default pool are allocated]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 2/18/2025 7:04:05 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[danmoseley]

also Aspire.Hosting.Tests.WaitForTests.WaitForBehaviorWaitOnDependencyFailure is often failing with this message. Between them the build is constantly red.

[danmoseley]

run with more logging shows ~20 entries in docker network ls after the tests run, 0 before. Is that expected?

https://dev.azure.com/dnceng-public/public/_build/results?buildId=953760&view=logs&j=80af4256-dfad-5412-889b-4c79f0868294&t=64cae19a-af0e-53c1-0d35-d4c23b4b293c

[danmoseley]

Seems like next step is getting docker version updated

[karolz-ms]

@danmoseley updating Docker is a good idea but I do not think it is under our control and I do not think it will help in this case.

So many leftover Docker networks is definitively not normal. Most likely cause is that a bunch of test are trying to use containers via DCP, which tries to allocate a dedicated Docker network for each tests. Then there is a bunch of failures, and probably the test infrastructure kills DCP instances and that prevents the networks from being cleaned up.

In addition to figuring out what caused these failures to start appearing, I think we need to make it easier to associate Docker networks with tests that create them (e.g. via a label or something).

[karolz-ms]

I took a closer look at the logs from the failed run. Looks like Docker at some point indeed "locks up" and starts failing the docker network rm command (DCP makes numerous attempts, but all end up with exit code 125). So updating Docker probably IS the best bet, but as far as I know, the Docker is part of AzDO infrastructure and it won't be easy to update it unless we want to have own test runner machine image.

[afscrome]

Docker's default IP range is a /24, and it gives each network it's own /20 - only allowing for 32 unique network subnets. You may want to adjust this config in the docker daemon.json to use smaller subnets, or a larger base range to allow for more networks.

(Edit, docs are a bit unclear over the default network size, but tweaking default-address-pools to allow for more networks should help if running out of address space)

{
  "bip": "172.17.0.0/16",
  "default-address-pools": [
    {
      "base": "172.17.0.0/16",
      "size": 24
    }
  ]
}

[radical]

@karolz-ms should this be closed now?

[karolz-ms]

Yes, it was assigned to me to be closed. Long story short, the issue turned out to be a quite old Docker CLI that was not supporting invocation flags we were expecting.