fix: make project network external, fixes #5193

[stasadev]

Summary

• fix: ensure no duplicates for docker networks, fixes #5193 #5508

The Issue

• Multiple ambiguous docker networks created #5193

Sometimes when you start a project, multiple project networks are created (e.g. ddev-npg_default) which prevents the project from starting.

Error response from daemon: network ddev-npg_default is ambiguous (16 matches found on name)

It doesn't happen all the time, but it's really annoying.

How This PR Solves The Issue

The idea is that if the project's docker network is created before doing the usual stuff, it can prevent the problem from occurring.

Let's make the project's docker network external (just like the ddev_default global network).

Manual Testing Instructions

This PR must not break existing features to be considered something valuable to merge.

There is no reliable way to test this, because the issue is related to docker-compose (not DDEV itself), and it occurs randomly.

Automated Testing Overview

Related Issue Link(s)

• Multiple ambiguous docker networks created #5193

Release/Deployment Notes

[github-actions]

Download the artifacts for this pull request:

• all-ddev-executables.zip
• ddev-linux-amd64.zip
• ddev-linux-arm64.zip
• ddev-macos-amd64.zip
• ddev-macos-arm64.zip
• ddev-windows-amd64-installer.zip

See Testing a PR

[stasadev]

I removed $ from Warning: No resource found to remove$ because there was extra output from docker-compose with this command:

$ project=test-5305 && mkdir $project && cd $project && ddev config --auto --omit-containers=db && ddev start && ddev stop && ddev restart && ddev delete --omit-snapshot --yes && cd .. && rm -rf $project
...
Starting test-5305...
...
Project test-5305 has been stopped. 
Restarting project test-5305... 
Warning: No resource found to remove for project "ddev-test-5305".
...

[rfay]

Rebased to hopefully fix colima problems.

[rfay]

Rebased again hoping for colima success.

[jonaseberle]

It seems we are working around a docker bug here if the implicit project networks get duplicated with the same name (which should not be possible!).

I haven't found regressions so far.

But maybe help me understand: Where do we create that external project network now that we are not letting docker-compose doing that? It must be in our code but I have not found it.

[rfay]

I haven't looked closely at this, but another way to handle this, instead of using external network, is to just remove any dangling network before running docker-compose up. That would leave the network responsibility in the docker-compose yaml and be less intrusive.

[stasadev]

Where do we create that external project network now that we are not letting docker-compose doing that?

I put the creation of the project's network inside EnsureDdevNetwork() - where ddev_default is created.

another way to handle this, instead of using external network, is to just remove any dangling network before running docker-compose up

The problem is that there is no dangling network, the duplicates are created only when docker-compose up is running.

A comment from the related issue:

All 16 created in the same second with different subnets.

[jonaseberle]

Thanks @stasadev for the explanation!

[stasadev]

I have been using this PR for a week and have not noticed any problems.

During this time, I did not see duplicate networks.

Again, this is a workaround, but it would be nice to have it here.

[rfay]

I manually tested this and looked at networks before and after and it seemed right to me.

I added just a single commit to add links to the PRs and issues so this won't be incomprehensible in the future when somebody stumbles across it in the code.

@stasadev would you mind testing this version for another week, as it now has an updated docker-compose in it?

[stasadev]

Sure, I will use it for another week.

Meanwhile, yesterday I went to the stable v1.22.1 to confirm that I have this problem with networks - yes, I still see duplicate networks.

[rfay]

I sure wish we understood why this can happen to you so often, but is not often-reported.

[rfay]

I'm sure you're already doing this, but when testing make sure to use the "native" bundled version of docker-compose.

[stasadev]

I sure wish we understood why this can happen to you so often, but is not often-reported.

Added a new comment here, hard to tell, but the problem may come from the Docker Engine.

I'm sure you're already doing this, but when testing make sure to use the "native" bundled version of docker-compose.

Yes, I use the bundled version.

---

Tried to grab the latest artifacts in the second comment, but they are not updated.

Something is wrong with Add download link to PR after PR Build.

Found the latest binaries here.

[rfay]

We'll do a point release this week, so if you think this is going to be OK we'll pull when you're ready again. Or we can wait for the next point release.

We could also wait for moby/moby#46251 but that one won't roll out to all environments for quite some time, even though it will show up sooner on Linux and Docker Desktop.

[stasadev]

I've been testing it for a few days and haven't noticed any issues with docker-compose v2.21.0.

But let's postpone this PR for another point release.

I see activity in moby/moby#46251 and want to check it out, hopefully it will be merged soon (I'll build Docker from master).

[rfay]

Thanks! I'm going to move this back to draft then, and we'll rebase as time goes on.

[rfay]

I did find one minor piece of fallout with this, but it has to do with going back to regular version after testing this one:

failed to start d10: composeCmd failed to run 'COMPOSE_PROJECT_NAME=ddev-d10 docker-compose -f /Users/rfay/workspace/d10/.ddev/.ddev-docker-compose-full.yaml up -d', action='[up -d]', err='exit status 1', stdout='', stderr='time="2023-09-11T11:25:04-06:00" level=warning msg="a network with name ddev-d10_default exists but was not created by compose.\nSet external: true to use an existing network" network ddev-d10_default was found but has incorrect label com.docker.compose.network set to ""'

People who understand what's going on know to delete the network, but it would be baffling for others. Not sure if or what we could do or if we should do anything.

[stasadev]

Ah yes, that could be a problem. DDEV offers to do ddev poweroff only after the upgrade.

But I can't guarantee that project will be stopped before switching the version.

This requires adding extra code to check and remove the external network at ddev start.

But again, that won't prevent this error from occurring if someone decides to downgrade.

[rfay]

Yeah, let's just think about it. One thing we could do now is

• Always tag networks like we already do on containers and should start doing on images
• ddev poweroff can delete all tagged networks.

[stasadev]

Yes, looks like part of a larger task with Docker resources management.
So many things to control, but it would definitely improve the user experience.

[stasadev]

I see activity in moby/moby#46251 and want to check it out, hopefully it will be merged soon (I'll build Docker from master).

My update here:

I'm not testing anything here yet, despite it being merged into master.

The PR has not yet been merged into the https://github.com/moby/moby/commits/24.0 branch.

I tried a build from master, but ddev start is very slow for me there. I believe the Arch Linux build is not optimized for it (the master branch is something like what we'll see in Docker 25).

[stasadev]

I was going to do the same thing with labels, and on a larger scale (with images and other docker entities).

But I haven't had time to do it yet. @jonaseberle we think alike here 😄

[rfay]

So sad to say this after as long as you've been working on this, but it will introduce a situation where if we have any kind of problem with the release it's introduced in, we won't easily be able to tell people "just go back to previous" without quite a bit of pain.

So let's introduce a simple all-network cleanup before we introduce this, and clean up all known networks and all "ddev" tagged networks. So with that little item added in the upcoming release, we can be confident that we can offer people backward compatibility. Then we can get this into the next one.

This also gives @jonaseberle, a total expert on networks, time to use these artifacts, which is also a win.

[rfay]

Put this one back to draft mode again, just do I don't get confused along the way :)

[stasadev]

I will rebase and resolve conflicts here later

• feat: add output for removed network, for #5305 #5436

[stasadev]

Fixed conflicts and rebased.

[stasadev]

Another rebase after v1.22.4 release.

I am going to check it out after tests and I believe we can finally merge it.

Thanks to @rfay for the careful planning and @jonaseberle for the testing and ideas on how to make it better.

[rfay]

Given the amount of effort on this one and good checking I think it should be good to go now, can be pulled when you think it's good.

[rfay]

I also did a prerelease that can be used for testing this, https://github.com/ddev/ddev/releases/tag/v1.22.5-alpha1