Some hosting tests failing on Azdo with docker+network issues

[radical]

Failing build on an unrelated PR #10053.

A bunch of test runs failed on helix.

Looking at Hosting.MongoDb, it has lot of entries like the following before the whole job times out:

fail: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0] could not create the network as all available subnet ranges from the default pool are allocated, retrying... {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 52, "Network": "default-aspire-network-vskli6mtis", "error": "docker command 'CreateNetwork' returned with non-zero exit code 1\nobject could not allocate required resources\nnetwork subnet pool full\nError response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network"} [+44/x3/?0] Aspire.Hosting.MongoDB.Tests.dll (net8.0|x64) - Aspire.Hosting.MongoDB.Tests.MongoDbFunctionalTests.VerifyWaitForOnMongoBlocksDependentResources (19m 36s)

could not create the network as all available subnet ranges from the default pool are allocated, retrying...

docker command 'CreateNetwork' returned with non-zero exit code 1
object could not allocate required resources
network subnet pool full
Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Looking at Hosting.Milvus, I see lot of

fail: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0] could not create a network {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 2, "Network": "default-aspire-network-fcj9i4r2t0", "error": "docker command 'InspectNetworks' returned with non-zero exit code 1\nobject not found\nnetwork not found\nError response from daemon: network default-aspire-network-fcj9i4r2t0 not found\nnot all requested objects were returned\nonly 0 out of 1 networks were successfully inspected\ndocker command 'CreateNetwork' returned with non-zero exit code 1\nerror\nError response from daemon: failed to update bridge store for object type *bridge.networkConfiguration: open /datadisks/disk1/docker/network/files/local-kv.db: no such file or directory"}:

docker command 'InspectNetworks' returned with non-zero exit code 1
object not found
network not found
Error response from daemon: network default-aspire-network-fcj9i4r2t0 not found
not all requested objects were returned
only 0 out of 1 networks were successfully inspected
docker command 'CreateNetwork' returned with non-zero exit code 1
error
Error response from daemon: failed to update bridge store for object type *bridge.networkConfiguration: open /datadisks/disk1/docker/network/files/local-kv.db: no such file or directory"

Looking at some of the other failures they seem to be of the same form.

This happened only on Azdo. And the changes in the PR are based on 223ff6a from main.

cc @karolz-ms @davidfowl

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[davidfowl]

@danegsta @karolz-ms @dbreshears It's back.

[danegsta]

That local-kv.db: no such file or directory seems to indicate a corrupt Docker setup on the machine; that file is where Docker stores network state (running networks, containers connected to networks, etc.), so if it’s corrupted or unreachable, it would prevent network commands working.

Are these tests failing consistently on azdo?

[radical]

Are these tests failing consistently on azdo?

No. But this failed on multiple helix agents as part of the same build.

Also, this has the same network failures but no error about local-kv.db.

[danegsta]

Okay, I'm assuming the local-kv.db error was an unrelated point in time issue on that machine. What's odd is that the last change we made to networks in DCP was a couple weeks ago? Did these tests just start failing in the last day or so?

[radical]

This failed on a fresh main run today - https://dev.azure.com/dnceng-public/public/_build/results?buildId=1081427&view=logs&j=acc6f692-388e-5b31-5997-e154da29f5b3&t=59fafa2a-4be0-5f74-c845-0779cc54f4a3

[danegsta]

@radical looking at the logs, I'm suspicious at least one of the helix machines has a corrupt Docker install; multiple errors reading files from /datadisks/disk1/docker/ (both related to network operations and pulling images:

[apphost] info: TestProject.AppHost.Resources.redis[0]
[apphost]       11: 2025-06-30T00:44:05.1000000Z open /datadisks/disk1/docker/tmp/GetImageBlob1849834936: no such file or directory
[apphost] info: TestProject.AppHost.Resources.postgres[0]
[apphost]       17: 2025-06-30T00:44:05.2770000Z open /datadisks/disk1/docker/tmp/GetImageBlob788583834: no such file or directory

[radical]

I'm suspicious at least one of the helix machines has a corrupt Docker install

The agents seem to be reset now, and the tests don't fail. Closing this issue.
Validated on https://dev.azure.com/dnceng-public/public/_build/results?buildId=1083181&view=results .