add --remove-orphans to docker-compose command?

[vicyap]

I got the following error while using the plugin. Does it make sense to add --remove-orphans to the docker-compose command?

Exception: Command 'docker-compose -f "<hidden>" -p "pytest7" down -v' returned 1: """Stopping <hidden> ... 
E           Stopping <hidden>      ... 
E           
Stopping <hidden> ... done
Stopping <hidden>      ... done
Found orphan containers (<hidden>, <hidden>, <hidden>) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.

[butla]

@vicyap But did you do anything with the compose file or the containers during the run of pytest?
Do you have some way of reproducing the problem?

I used this plugin quite extensively and didn't see that problem before.

[vicyap]

Hmm. I have seen this problem intermittently. We have a set of VMs connected to Jenkins which runs our CI.

My current theory is that a job was aborted and left containers running. Then when the job re-runs it produces this error.

I'll try to reproduce it.

[butla]

Ok. I didn't use --remove-orphans before and I'm not sure how that affects other workflows with Compose. So I'd be cautious about adding that. But if we can change something in the code that would prevent the situation from occuring, i.e. killing the containers when the tests are killed, that would be safer, I guess. But I think there should be a reproducible flow before the code is changed.

[vicyap]

I agree, I don't know the impact of just adding in --remove-orphans. I'm not that familiar with docker so I'll need to research how orphaned containers occur in the first place. I'll post back here once I have a reproducible flow for this error.

[butla]

I'm guessing it's got something to do with changing the compose file while a previous instance is running

[AndreLouisCaron]

Yes, this option is useful after removing services from a compose file. Example:

a:
  image: redis

b:
  image: redis

Run docker-compose up -d. Then, remove b and run docker-compose down -v and you will get output like this:

Stopping desktop_a_1 ... done
WARNING: Found orphan containers (desktop_b_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Removing desktop_a_1 ... done

I don't see how this could happen during normal operation. You could probably trigger this on long builds by editing your docker-compose.yml file while the build is running, but since you mention this happening on CI builds, that's unlikely.

My current theory is that a job was aborted and left containers running. Then when the job re-runs it produces this error. I'll try to reproduce it.

Hint: you might want to look for builds from multiple branches where one branch contains a service that is not defined in other branches. Think of this sequence:

1. Start build for branch A, with service not defined in branch B
2. Make sure that docker-compose up -d has run and kill the build for branch A
3. Start build for branch B and let it complete.

Hint: concurrent builds on the same machine might also be an issue, especially if the compose project name is fixed across branches. Think of this sequence:

1. Start build for branch A
2. Start build for branch B that has one service not defined in branch A
3. Build for branch A completes and runs docker-compose down -v, which detects and orphaned container.

IMO, you might consider using a random compose project name to avoid conflicts in both cases, then run a cron job on your build machine to clean up orphaned containers once in a while to account for jobs that get killed.

Note: if editing docker-compose.yml file while the build is running is actually an issue for you, I guess we could copy the file to a temporary location somewhere else to prevent this from occurring.

[butla]

So... is there anything that needs to be done regarding this issue, or should it be closed?

[AndreLouisCaron]

Since there is no response from @vicyap, I will close this issue. Feel free to open a new issue if you run into this issue again.