New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose up fails if network attached to container is removed #5745

Closed
adambro opened this Issue Mar 2, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@adambro

adambro commented Mar 2, 2018

Container start fails, because network it was attached to has been removed. For some reason docker-compose tries to remove that container attached network first, which results with an error.

Steps to reproduce:

docker-compose up
docker network rm dockercomposenetworkrmfails_default
docker-compose up
@adambro

This comment has been minimized.

adambro commented Mar 2, 2018

The confusing part is: docker-compose created a new network first, but then tried to re-start already existing container attached to removed network. I'd expect it to use this newly created network instead. I've figured out what's happening only after comparing network Id hash, because network name was the same (generated automatically).

Maybe compose can try fail-safe network rm instead?

Minimal docker-compose.yml to replicate:

version: '3'

services:
  container:
    image: rwgrim/docker-noop
@shin-

This comment has been minimized.

Member

shin- commented Mar 2, 2018

Thanks for the report! I think there are several things to note here:

  1. First and foremost, you shouldn't delete networks managed by Compose unless you're doing through Compose (typically with docker-compose down). Outside of this particular problem, it will certainly lead to a slew of other issues down the line.
  2. Containers always connect to networks using the network ID which is guaranteed to be unique, whereas no such guarantee exists for names.
  3. Your container is being restarted because no changes in configuration were detected, and the recreation of the network is considered to be an independent event. You could actually workaround the issue by using the --force-recreate flag.
  4. There's a case to be made for checking the network containers will connect to still exists, and reconnecting to the new network if necessary, but see item 1: this probably shouldn't happen in the first place, and I think raising an error is equally valid in this instance.
@adambro

This comment has been minimized.

adambro commented Mar 2, 2018

Thanks for detailed explanation. Indeed, it happened because I've run docker network prune some time before.

What I don't get still is why it tries to remove Network that does not exist any more? I guess that caused "not found" error. I haven't found any other instance of such error message.

@shin-

This comment has been minimized.

Member

shin- commented Mar 2, 2018

It's not due to a removal attempt. Compose tells the engine to start , which the engine attempts and recognizes that the network it is configured to be connected to doesn't exist anymore, preventing the operation from succeeding. Compose simply reports that error.

@adambro

This comment has been minimized.

adambro commented Mar 4, 2018

You're right @shin- it's a message from failed container start command:

ERROR: for container  Cannot start service container: network 8a131ba522472461e38b5189f5fd489b22d4d7a850a6dcd64a227ceb687c35a6 not found

I agree with your points since I better understood how compose tries to bring up old containers instead of creating new ones every time. Thanks for explanation, appreciate that. It seems to be that way by design, so nothing more to do here I guess.

@adambro adambro closed this Mar 4, 2018

@dazinator

This comment has been minimized.

dazinator commented May 17, 2018

It would be nice to have some "recreate if necessary" flag. I.e we want to ensure docker-compose up does whatever it takes to bring the environment up and if that means it needs to create a network then so be it!

@jlyman

This comment has been minimized.

jlyman commented Jun 25, 2018

I'd second the suggestion from @dazinator for a "recreate if necessary" flag. I ran a docker network prune command just like @adambro, and since I run several different Docker setups for different projects, I didn't realize that pruning while working on one large project was going to negatively affect the others. (My fault for not reading and thinking through the warning from network prune more when I did so.) But a flag like that would be a nice way of getting a project back on its feet quickly.

@DonAurelio

This comment has been minimized.

DonAurelio commented Sep 3, 2018

I was running the ubuntu version of docker holded in this respository (docker.io 18.06.0). I removed that version an install docker.io using apt-get install docker.io. So now I have this version:

Client:
Version: 17.12.1-ce
API version: 1.35
Go version: go1.10.1
Git commit: 7390fc6
Built: Wed Apr 18 01:23:11 2018
OS/Arch: linux/amd64

Server:
Engine:
Version: 17.12.1-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.10.1
Git commit: 7390fc6
Built: Wed Feb 28 17:46:05 2018
OS/Arch: linux/amd64
Experimental: false

Containers running with docker-compose doesn't have the file resolv.conf with the dns consiured as expected, but, each container can access Internet. I think that change solve mi problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment