Docker Networks are not being cleaned up (again)

[kijanawoodard]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I followed #7215 and applied the fix when it was released. Everything seemed better.

Now using dotnet 9 + aspire 9.3, the problem appears to be back.
I had zero problems on a project with just sql server, but once I added redis, I'm getting "failed to create network" at least once a day.

The ironic part is i'm not even using redis yet past setting it up in Program.cs.

fail: Aspire.Hosting.Dcp.dcpctrl.NetworkReconciler[0]
      could not create a network        {"NetworkName": {"name":"default-aspire-network"}, "Reconciliation": 2, "Network": "default-aspire-network-1b9lq4ehg4", "error": "object not found\nnetwork not found\ndocker command 'CreateNetwork' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Error response from daemon: all predefined address pools have been fully subnetted\n'"}

Expected Behavior

Restarting AppHost should not exhaust docker networks.

Steps To Reproduce

app host:

<PackageReference Include="Aspire.Hosting.Redis" Version="9.3.0" />

var redis = builder.AddRedis("redis")
    .WithImageTag("8.0.2")
    .WithRedisInsight()
    .WithDataVolume("redis-data")
    .WithPersistence(TimeSpan.FromSeconds(1));

web:

<PackageReference Include="Aspire.StackExchange.Redis" Version="9.3.0" />

builder.AddRedisClient(connectionName: "redis");

Exceptions (if any)

No response

.NET Version info

No response

Anything else?

No response

[davidfowl]

Are you using sorbet watch by any chance?

[danegsta]

@kijanawoodard can you run docker network ls and report how many Aspire networks you have (vs how many total networks are running)?

[kijanawoodard]

I just cleaned, but i'll run it next time it crashes or later today.

 docker network list
NETWORK ID     NAME      DRIVER    SCOPE
1529d1b84162   bridge    bridge    local
8102fa3331bb   host      host      local
c6d6351c9731   none      null      local

[kijanawoodard]

for anyone who gets here via search, here is a hammer of a workaround. it could be more targeted, but I keep this in my cli history to keep moving.

docker container stop $(docker container ls -aq); docker container prune -f; docker network prune -f

[kijanawoodard]

@davidfowl

Are you using sorbet watch by any chance?

Yes. I'm using dotnet watch.

Side note: aspire run experience is nice. Waiting for aspire watch to go full time on aspire cli.

[danegsta]

If you’re having to forcibly remove containers as well as networks, that makes me wonder if this isn’t a similar issue to what we were running into with dotnet watch simply terminating all child processes in its job group (which meant cleanup never actually ran). @karolz-ms do you remember which dotnet release they fixed that in?

[kijanawoodard]

@kijanawoodard can you run docker network ls and report how many Aspire networks you have (vs how many total networks are running)?

My other computer has a list:

 docker network ls             in pwsh at 11:39:07
NETWORK ID     NAME                                DRIVER    SCOPE
31b0717b9640   bridge                              bridge    local
9810f8ef27d5   default-aspire-network-0udb9mj208   bridge    local
8a2bcb90805c   default-aspire-network-2svtl0poc4   bridge    local
505d450e55cd   default-aspire-network-3fpscjb5u8   bridge    local
12d3fa6a164f   default-aspire-network-6h0gkmemr4   bridge    local
8907f6772814   default-aspire-network-7c3rtktr68   bridge    local
39148c7127e4   default-aspire-network-9l7u7ltueo   bridge    local
651fb0101dd8   default-aspire-network-29uqv9ombc   bridge    local
aad4eeeb61c2   default-aspire-network-288as54fa4   bridge    local
9481681670a5   default-aspire-network-669iesih8o   bridge    local
896a294c2f09   default-aspire-network-f4vkicu38g   bridge    local
bffb1e4fc367   default-aspire-network-g8q7jhab9g   bridge    local
f35a64431c10   default-aspire-network-h3ikkso41k   bridge    local
da8e8ad574ba   default-aspire-network-hbb6143iak   bridge    local
487aa9a811b0   default-aspire-network-ihrvjffc5g   bridge    local
8c5b63358763   default-aspire-network-me82hpl9i8   bridge    local
8827162d6a64   default-aspire-network-msmbu0i9mg   bridge    local
eb1a5583792f   default-aspire-network-qjrl5m1p1o   bridge    local
627a45c84234   default-aspire-network-sqfh8c358o   bridge    local
510bb3dce3a3   default-aspire-network-thoa1bomgs   bridge    local
00914c23e380   default-aspire-network-ucvutc5lik   bridge    local
95dcfe77a7db   host                                host      local
e129594a0f5e   none                                null      local

[kijanawoodard]

If you’re having to forcibly remove containers as well as networks, that makes me wonder if this isn’t a similar issue to what we were running into with dotnet watch simply terminating all child processes in its job group (which meant cleanup never actually ran). @karolz-ms do you remember which dotnet release they fixed that in?

I removed the prune from my hammer. 😅

docker container stop $(docker container ls -aq);docker network prune -f
43d0dfe18de0
Deleted Networks:
default-aspire-network-3fpscjb5u8
default-aspire-network-msmbu0i9mg
default-aspire-network-thoa1bomgs
default-aspire-network-6h0gkmemr4
default-aspire-network-sqfh8c358o
default-aspire-network-qjrl5m1p1o
default-aspire-network-0udb9mj208
default-aspire-network-7c3rtktr68
default-aspire-network-2svtl0poc4
default-aspire-network-f4vkicu38g
default-aspire-network-ucvutc5lik
default-aspire-network-ihrvjffc5g
default-aspire-network-669iesih8o
default-aspire-network-9l7u7ltueo
default-aspire-network-hbb6143iak
default-aspire-network-h3ikkso41k
default-aspire-network-29uqv9ombc
default-aspire-network-288as54fa4
default-aspire-network-g8q7jhab9g
default-aspire-network-me82hpl9i8

docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
31b0717b9640   bridge    bridge    local
95dcfe77a7db   host      host      local
e129594a0f5e   none      null      local

[danegsta]

@kijanawoodard do you end up with abandoned Aspire containers in addition to networks? No need to run until the networks are exhausted for this check; I’m mostly curious if you run it say three times, do we leave three runs worth of both network and container resources behind (or is it intermittent)?

[davidfowl]

@kijanawoodard what version of the dotnet sdk

[kijanawoodard]

@kijanawoodard what version of the dotnet sdk

@davidfowl This machine is:

dotnet --version              in pwsh at 13:20:29
9.0.204

I'll upgrade to 9.0.300 in a bit.

[kijanawoodard]

@kijanawoodard do you end up with abandoned Aspire containers in addition to networks? No need to run until the networks are exhausted for this check; I’m mostly curious if you run it say three times, do we leave three runs worth of both network and container resources behind (or is it intermittent)?

@danegsta the containers aren't hanging around (except for the sql one I have setup .WithLifetime(ContainerLifetime.Persistent).

I did notice that ctrl-c seems fine.
But ctrl-r adds a new aspire-network each time.