Multiple ambiguous docker networks created #5193
Comments
|
I've just seen that DDEV v1.22 has been released since I started writing this ticket. I've upgraded and will report if it still happens. |
|
DDEV v1.22.0 uses the Traefik router by default, whereas previous versions used the nginx-proxy router by default (but had Traefik as an option). Looking forward to your results. Are you using https://github.com/torenware/ddev-viteserve ? |
|
I see
It looks like you have a docker problem. And I see you're using Orbstack. I've spent a bit of time with Orbstack, but we haven't run a complete set of tests for it, and it doesn't have automated tests. So I would suspect Orbstack. Also, do try DDEV v1.22.0 and the Traefik router, but it sure looks like you have docker trouble. And please do see if you can recreate on Colima or Docker Desktop. |
rfay
changed the title
|
Hi @rfay until 36 hours ago I was using Docker Desktop and getting an identical problem, with the spec: I thought I might have a Docker problem so I installed OrbStack with this DDEV project the only set of containers/networks in it. The problem happened just the same, despite Orbstack having its own network stack implementation. Since I updated to ddev version 1.220.0 10 hours ago it hasn't happened, so fingers crossed that traefik has sorted it. I'll be back on this project next week so will wait and see if it remains sorted! |
|
You were getting the identical problem with failing to remove the docker network?
I sure wouldn't expect to see that the same way on two different docker providers. I wonder if you're running a separate container using that network that is not managed by DDEV so DDEV doesn't stop the container? I too hope you don't have the problem any more but you may have to develop a full reproduction case that shows what happens here, because this sounds really unusual. One thing to do is to study |
|
Okay so... I've been working on another project that uses ddev all day. It's been running fine. I go to start the npg project this evening to do a bit of development and... I mean how??? How are so many network providers proliferating?
And yes there really are all those bridged networks available. Aside: ddev-api has not been run for weeks and is dangling. Let's inspect them: All 16 created in the same second with different subnets. |
|
Nobody seems to be able to recreate it or figure out what's doing it. Appears to be related to Docker Desktop? But now you're on OrbStack? Or did you switch back? |
|
Still on orbstack. I was seeing the same on Docker Desktop (I'm the most recent comment on #4958 ) |
|
DDEV deletes its networks on What's your workflow? Do you possibly do something that doesn't include "stop"? Perhaps repeatedly doing I don't really understand why this odd behavior is happening to just a scattered few people. As you know it's fixable with |
|
And yes running vite on the host and spinning up its in-built dev server on port 5173 may be coincidence rather than cause. I thought I had a reproducer there.
I do |
rfay
changed the title
|
There's actually nothing wrong with I sleep/poweroff laptop all the time without even stopping projects. Never have seen this on macOS or WSL2. Obviously it happens! I just experimented with a few sites and saw only correct behavior with networks being deleted on stop. Maybe it does have something to do with the vite service, that would seem strange. Please do occasional |
|
After a poweroff which removed all the networks, I've done a dozen starts/stops in the last 5 minutes and it's not happened. I plug into an USB-C Anker hub connected to ethernet at my desk and then use wifi the rest of the time. I'll see if it happens without the network interfaces changing (I assume they do) - been trying that out now but nothing bad happening whichever way |
|
It looks to me like the suspicious place for this is https://github.com/ddev/ddev/blob/master/pkg/dockerutil/dockerutils.go#L36-L78 and ddev/pkg/dockerutil/dockerutils.go Lines 266 to 275 in 7294710 The only thing I can think is if NetExists() isn't working correctly. |
|
Oh, it can also be a dangling container which is using the network and prevents it from being deleted. I think we saw that in an earlier dump from you? You'll want to watch carefully for that, and when it happens, do a |
|
|
Hiya, I have continued to experiment and write up notes. I have uncovered 2 OrbStack issues but not yet got reproducers to be able to report them to Orbstack. See what you think in case one of them involves ddev. Issue 1: PhpStorm and Orbstack interplayThis only happens with particular PhpStorm projects. Presumably ones that try to run something inside the Docker Container automatically. It doesn't involve the ddev plugin as I've tried it installed and removed. What happens is when my ddev project is not running and I open PhpStorm, it does the following: docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dbcd138aabc7 ddev/ddev-ssh-agent:v1.22.0-built "/entry.sh ssh-agent" 12 hours ago Up 12 hours (healthy) ddev-ssh-agent
f3fde3ba6d5f 422dec05bb87 "/bin/sh" 11 days ago Created phpstorm_helpers_PS-231.9225.18
> docker network ls
NETWORK ID NAME DRIVER SCOPE
ccb41cd50c43 bridge bridge local
012df10b2e38 ddev_default bridge local
d97be534fb66 host host local
b159031a67f9 none null local
### I open my PhpStorm project
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16df75968a0e ddev/ddev-webserver:v1.22.0-npg-built "/pre-start.sh" 4 seconds ago Created ddev-npg-web
dbcd138aabc7 ddev/ddev-ssh-agent:v1.22.0-built "/entry.sh ssh-agent" 12 hours ago Up 12 hours (healthy) ddev-ssh-agent
f3fde3ba6d5f 422dec05bb87 "/bin/sh" 11 days ago Created phpstorm_helpers_PS-231.9225.18
> docker network ls
NETWORK ID NAME DRIVER SCOPE
ccb41cd50c43 bridge bridge local
5954c8413085 ddev-npg_default bridge local
32eeba2c017d ddev-npg_default bridge local
5942e2cae323 ddev-npg_default bridge local
cadcd530431b ddev-npg_default bridge local
cbca75a1f7ae ddev-npg_default bridge local
5c507064878d ddev-npg_default bridge local
62e74d9579c1 ddev-npg_default bridge local
93129e5da6a4 ddev-npg_default bridge local
66ecd98e027a ddev-npg_default bridge local
cb46373f3e67 ddev-npg_default bridge local
2eba9c1129cb ddev-npg_default bridge local
9d8c1f425c46 ddev-npg_default bridge local
63fad463bf4d ddev-npg_default bridge local
c1eb0f28b8e9 ddev-npg_default bridge local
bc6e4aa78f88 ddev-npg_default bridge local
63ac3f654513 ddev-npg_default bridge local
012df10b2e38 ddev_default bridge local
d97be534fb66 host host local
b159031a67f9 none null localTo fix this problem I can run > ddev stop
Project npg is already stopped.
Container ddev-npg-web Stopped
Container ddev-npg-web Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Network ddev-npg_default Removed
Project npg has been stopped.
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dbcd138aabc7 ddev/ddev-ssh-agent:v1.22.0-built "/entry.sh ssh-agent" 12 hours ago Up 12 hours (healthy) ddev-ssh-agent
f3fde3ba6d5f 422dec05bb87 "/bin/sh" 11 days ago Created phpstorm_helpers_PS-231.9225.18
> docker network ls
NETWORK ID NAME DRIVER SCOPE
ccb41cd50c43 bridge bridge local
012df10b2e38 ddev_default bridge local
d97be534fb66 host host local
b159031a67f9 none null localThis is a nuisance but now I know the fix, I can work around it. Issue 2: Subfolders, mutagen and project confusionI only discovered this one late last night so haven't yet had a chance to reproduce with Docker Desktop to confirm it's an OrbStack issue (I'll edit this message when I do) I had an already run This got me into a broken state, because I got (message from a reproducer this morning): I suspect I tried a few things and in the end powered off ddev and then manually deleted the volume Then I ran This time I ran Starting npg...
Using custom mysql configuration: [/Users/peter/npg/.ddev/mysql/my-npg.cnf]
Custom configuration is updated on restart.
If you don't see your custom configuration taking effect, run 'ddev restart'.
Container ddev-npg-elasticsearch Running
Container ddev-npg-web Running
Container ddev-npg-db Recreate
Container ddev-npg-db Recreated
Container ddev-npg-db Started
Container ddev-router Running
Successfully started npg
Project can be reached at https://npg.ddev.site https://127.0.0.1:32802and more importantly, the web container still works afterwards. Aside: startup timeEven on a successful run the containers start and then there's a 15s+ pause before the "Successfully started" message is printed. Is there any way I can see what's going on in that time? |
|
Great work studying it!
or if desperate and you really want to read lots of stuff
and then you can see exactly what's going on. I think your study here is going to get us somewhere... that it may have to do with PhpStorm. When PhpStorm uses the .ddev/.ddev-docker-compose-full.yaml, it's doing it without any input from DDEV. DDEV thinks it's already done its job (or perhaps a project hasn't started yet), but PhpStorm is using raw docker-compose. So maybe that's our path. Thank you for the great study and debugging, I think we're going to get somewhere! With regard to your but `ddev start definitely knows how to find the correct project, doesn't get confused about this. |
|
Thanks Randy, will check tomorrow if I've got more than one |
|
Interestingly, the I see this workaround is mentioned here: Maybe if the project default network was created beforehand, it could resolve this issue 🤔 |
|
Thanks for spotting that issue @stasadev - I wonder why this doesn't happen more. The project network could probably be created externally. It makes some things awkward and the docker-compose externally becomes harder (as with PhpStorm's plugin for ddev) - An experimental PR would be welcome. |
|
So far, so good.
PhpStorm's plugin sees my database with the project's external network. I'll be using the binary from #5305 with my projects for a week or so to see if there are any issues and will report my results here. |
|
Thanks @stasadev - have you been able to demonstrate this problem in your own environment? |
|
It just happens from time to time. |
|
Great that you see it! Have you figured out anything about your workflow that might trigger it? Remind me your OS and environment... |
|
My workflow:
Usually such a problem occurs:
|
|
I wonder if it could have something to do with the Arch build of docker or something. But assuming the problem is docker-compose, that wouldn't matter because we provide our own docker-compose (I assume you use the defaults, and this report is from your PR, which would be using v2.20.3.) |
Seeing as other people have this problem on Mac, I'd rather generalize the problem. Also, when I see 100+ likes on a comment about duplicated docker networks (even if not everyone who liked it uses DDEV), it makes me think this problem is common: I remember having such a problem with: Then it was all good with: And started again with: It is a mystery to me where the cause of the problem lies, because there are also many dependencies for the mentioned programs. |
|
I am using the PHPStorm extension - and the problem happens for me on MacOS AND Linux (with PHPStorm + plugin). I did not get to try the updated release, sorry - but this is probably a good lead. edit: my plugin is now disabled. Will report if this keeps happening |
|
I mostly have DDEV integration disabled (I only use it when I need a database connection). Regarding @pbowyer's description - I agree that it could also be related to the DDEV integration, because I haven't seen that many (16 matches) duplicate networks at once. My maximum is 2 matches. |
|
I've had as many as 12 before. |
|
I finally tested @stasadev work today using the latest nightly on ARM64 with OrbStack as the backend, and so far it seems to have fixed the problem. I'll keep testing. |
|
I had this problem happen to me again today, using a nightly build. Command outputs slightly edited to hide the company I'm working for: From this it looks to have also happened to another project, proj2, that I worked on last week but didn't start the environment for. Edit: this time |
|
@pbowyer thanks for the feedback, I think I understand the problem better now. I have a few ideas to fix this since I can now control the project network:
I will create a PR for this today or tomorrow. Edit: I decided not to use |
|
I've created a new PR #5508 where I've done everything I can to resolve this issue once and for all. The idea is to run I've created a reproducible broken binary in the Manual Testing Instructions, so anyone who wants to test duplicate networks can easily do so. And then use a fixed binary from the bot comment to see how these duplicates will be removed. |
|
😁 @stasadev Thank you! 😁 I opened "proj3" in PhpStorm, a project which has always caused trouble. Your patch worked around the problem and I would dearly love Docker/OrbStack (which I use)/PhpStorm to stop creating unhealthy containers and duplicate networks, but thanks to you I don't have to think about it each time I run |
|
I do think it's possible that this issue is caused by the PhpStorm extension doing the regular So it may be that the actual fix is for ddev to use some kind of synchronization to make sure only one command can be run at a time. |
|
A related fix is short-cutting @nico-loeber would love your feedback on this. Is And to all of you: Does this behavior go away if you turn off the polling of status in the PhpStorm extension? |
|
I've disabled the PhpStorm DDEV extension and will text next week and see what happens. |
|
You don't actually have to disable the extension, just its polling behavior. 
|
|
It's not the DDEV plugin for PHPStorm. I'd kept the plugin disabled, and today I opened a project in PHPStorm before running |
|
Remind me what DDEV version you were using for that test? Please say more about "today I opened a project in PHPStorm before running ddev start and duplicate networks and a "Created" (soon to be in an unhealthy state) container appeared". Are you saying that you had nothing running, and you just opened a project in PhpStorm and things started happening? Could. you please do a |
|
Hi @rfay, that was using DDEV was running and I finished working on one project, closed its PhpStorm window and ran I will certainly run the docker commands before and after next time. The problem won't repeat now (just tried it) - it always needs a delay, which makes me think it's linked to something PhpStorm rechecks after a certain amount of time - and tries to do inside the container e.g. checking for composer package updates. |
|
This ticket looks like it could be the same issue: https://youtrack.jetbrains.com/issue/WI-53533/Autostart-docker-containers. I'll test out the workarounds in it. |
|
Wow, if PhpStorm is trying to start containers that could cause all kinds of havoc. |
|
@pbowyer could you try the artifacts from #5533 (comment) ? I decided that making the project networks external was too intrusive and found a way to make them internal as before, but with the same duplicate check. I hope this change will have the same effect with a simpler approach. |
|
I tested Docker 25.0.0-beta.1 The only difference I've noticed is that with Docker 24 you can have duplicates and with Docker 25 you don't and you'll still see a different error this time if a duplicate tries to be created. Docker 24.0.7, DDEV v1.22.4 Docker 25.0.0-beta.1, DDEV v1.22.4 ⬆️ I only tested the behavior of Docker 25 programmatically with https://github.com/fsouza/go-dockerclient. I believe that managing the project network with DDEV v1.22.5 will be useful as it will save the user from messing around with errors. |
|
But docker 25 doesn't solve the problem, just reports it? |
That's only my assumption, but I'm going to test it using Docker 25.0.0-beta.1 and DDEV v1.22.4, maybe I am wrong. |
Is there an existing issue for this?
Output of
ddev debug testExpand `ddev debug test` diagnostic information
Expected Behavior
I have a PHP site running within ddev, and on my host machine I run vite with a default configuration (so bound to localhost:5173).
I'd expect them to run alongside each other, but running Vite's dev server with HMR seems to kill DDEV.
Actual Behavior
502: Unresponsive/broken ddev back-end site.
This is the ddev-router container: The back-end webserver at the URL you specified is not responding. You may want to use "ddev restart" to restart the site.
The network hangs around and if I do a
ddev stopSteps To Reproduce
So, this doesn't reproduce 100% of the time but over the last couple of days this is the closest I can get to a reproducer.
When the HTTP 502 happens I try the following:
Trying to restart leads to:
Once I run that I can now run
ddev startand it works.Now I run
npm run devagain in my subfolder:Reload my PHP page a couple of times, and ddev goes back to returning the HTTP 502.
Anything else?
I thought it might be a bug in Docker, so I installed OrbStack with this DDEV setup the only set of containers/networks in it. The problem happens just the same, which is why I'm posting here as it could be to do with DDEV instead.
vite.config.ts:
And the minimal parts of my package.json:
The text was updated successfully, but these errors were encountered: