Container crashed, cannot stop, cannot rm, lead to docker daemon crash #20997
Comments
|
@S-YOU Can you check |
|
@coolljt0725, thanks for the tip. docker ps was showing Running, so it might be true there. |
thaJeztah
added
the
kind/bug
|
@coolljt0725 do you have thoughts how we can (at least) prevent a panic in that situation? |
|
@thaJeztah It's similar to the issues we have been meet before, showed in |
|
btw, looks like I can replicate the error. let me know if you need any more info or particular logs. Here is some status And its only happen with docker-compose up. I have mysql and data container linked, total 3 services. I have been using docker run each container, like more than 10 times, was no problem, only problem when docker-compose bring all container up altogether, not sure it is a race condition. |
|
which version of compose are you running? |
|
|
I tried to narrow down to single service, still crashing. This crash This does not crash So, looks like something in my application triggering it? or docker-compose compatibility problem? |
|
One difference between compose and docker in your repro-case is compose would use a different name (e.g. |
|
I just tried to use --name myproject_app_1 on docker run, it didn't crash, not sure it is same as what compose does. more like compose issue? should I raise issue in docker-compose repository? |
|
@S-YOU yes, I don't think that would hurt; it's possible there's an issue in "docker", which happens to be triggered because of the way docker compose calls the API. |
|
ok, I have narrowed it down to particular case.
could crash the docker daemon when running through docker-compose up |
|
Right, that sounds like a possible race condition between the command being run, and libnetwork updating the |
|
@thaJeztah it is not a race-condition. libnetwork updates @S-YOU thanks for narrowing down the scenario. Yes. forcefully clearing out the libnetwork manages the container's @sanimej might know more details on this. |
|
@mavenugo, Thanks for the details. Actual approach is to write something like My container has nginx and apache2, both listen on port 80. And also the setup scripts I am migrating to docker is using hostname -f, and hostname -d, so I had to set some hostnames more into /etc/hosts manually. I cannot use May be there is better way to do that, I can live with running docker run directly. |
|
Some more info This is OK This cause crash, same as "\n" alone |
|
@S-YOU thanks for the info and yes, I missed the fact that you are appending to the hosts file and yes, the bug is in parsing the /etc/hosts file and an empty new-line is the issue. We will get it fixed in the etchosts package. BTW. regarding your use-case, we do support
I guess the above workaround will avoid the need to manually update the |
|
@mavenugo, thanks for the info about --ip, I didnt know I can specify my own. |
|
@S-YOU panic is because of a memory access in the code without proper bounds check. The reason it happens only with compose could be because of the way compose brings up the containers. Will fix this issue.. Can you please explain what doesn't work with the |
|
@sanimej, thanks for analyzing issue and fix. The reason that I think why -h does not work is because of the image has nginx and apache2, both listen on port 80, apache2 on 127.0.0.1, and nginx on outgoing ip, 172.17.0.x in my case. If I set -h www.domain.com, apache start listening on outgoing interface and nginx fail to load. So, its not a issue with docker, in my opinion, but the nature of contents. |
|
@S-YOU why not put those in separate containers, and use (e.g.) https://github.com/jwilder/nginx-proxy ? |
|
@thaJeztah, thanks for the advice and definitely yes, and as far as I understand, nginx docker can just pluggable with environment variables + conf.d mount. EDIT: Looks like nginx-proxy image is more customizable than official nginx image, thanks for the input. |
|
@S-YOU well, it's based on the official nginx image, but adds automatic configuration (which is very nice). If you have a fixed configuration, then you can still consider using the standard nginx image, and adding your own configuration to that. A fairly recent project, but really nice is https://github.com/containous/traefik, which can act as a replacement for nginx, and has lots of really cool features. It's written in Go, and quite lightweight. |
|
Wow, thats pretty impressive project with web ui and full reverse proxy feature in 20M image. |
|
PS: with docker version |
|
@grexe Please report the panic on a separate issue, thanks! |
|
@grexe can you open a new issue with steps to reproduce, just in case? |
|
done @cpuguy83 and @thaJeztah, please see #32136 |
Output of
docker version:Output of
docker info:Provide additional environment details (AWS, VirtualBox, physical, etc.):
VirtualBox
List the steps to reproduce the issue:
Describe the results you received:
Daemon Crashed, container still there, tried several times, including reboot of server and system.
Describe the results you expected:
Container should stop it
Provide additional info you think is important:
Some logs
The text was updated successfully, but these errors were encountered: