docker stop does not work #552
Comments
|
I just tried here locally and it works fine for me (although I have no real integration tests running, of course). In your case could create a thread dump when it hangs and attach it to this issue ? |
|
Thanks for the effort. my dump looks like this: Thread dump at 0:15.093.616
|
|
Thanks. It looks like, that your Docker daemon doesn't close the connect. The Maven plugin is stuck in read of a post request to How long have you waited until to killed the maven call ? You can try to tune the |
|
it will wait forever I tried to play around with the kill parameter already the container itself is stopped at this time so just the feedback seems not to work |
|
That's really strange. Do you have a chance to test a different Docker daemone ? (e.g. with Docker machine or from a VM). BTW your are running on which plattform (OX X, Windows, Linux) ? |
|
Could you also upgrade to d-m-p version 0.16.6 ? |
|
Linux lsb_release -a I will try another docker machine and the update |
|
ok so with Docker version 1.12.2, build bb80604 and plugin version 0.16.8 it is working thanks for your time |
|
I have the same issue here, also with a postgres database. When I look at Docker version: 17.03.1-ce (git commit c6d412e329) Furthermore I don't have any integration tests using this database at the moment. |
|
Does it happen for you all the time that the container cant be stopped or only sporadically ? |
|
It worked one time. So I'd say sporadically it works, but usually not. I also played around with the |
|
Im experiencing the same thing with the postgres container, It is sporadic for me but fails most times. |
|
@benhubert by any chance are you running this postgres image docker-in-docker and/or with gitlab-runner |
|
I found the issue why the container end up with exit code 137. The problem was the log line I waited for. On first startup postgres initializes the database and then restarts it. So The log line The solution was to change the But the original issue - that the maven process hangs - is still not fixed. I run it on my physical machine. No docker-in-docker or gitlab-runner. Here's a thread dump2017-05-04 12:29:45 Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode): |
The interesting thing is that its hangs while reading the response from the Docker daemon when sending the stop command over the Unix socket Could you please try out the following (since unfortunately, I can't reproduce it):
What really puzzles me a bit is that the thread is in state Sorry to bug you with so many questions, but unfortunately, as said, I cant reproduce it. |
It seems that the thread stays |
|
@benhubert thanks a lot for your investigations ! Looks interesting :) sorry, the jnr-unisocket update is not on master (but on https://github.com/rhuss/docker-maven-plugin/tree/pr/update-jnr). However, the snapshot pushed to Maven central contains the update. Could you try it with this version ? |
|
I probably need to check how a proper timeout could be set on the unix socket connection. |
|
I ran it with the branch but this ends up with |
|
Yes, a timeout would be a workaround for this, but then I end up with many docker container running or at least not being removed because sometimes the removal does not work. |
|
Thanks for the feedback. I will have a deeper look into the jnr-unix upgrade. However, I'm not so positive if this upgrade doesn't fix it, because there is very likely not much I can do except maybe adding the timeout. No idea why the Docker daemon doesn't send the response and keeps the connection open (or the client). Would be nice to debug into the Docker daemon, too, but tbh, no idea where to start on the docker side ;-( |
|
I am having this same issue with a MySQL container, but only when I specify a volume for it to use. I was trying to fix some slowness that was happening on Travis by running MySQL on a ramdisk, and then starting running into this issue. I tried attaching the volume to my standard volume rather than ramdisk, and it still happens. I also check and the containers are gone almost immediately after it starts hanging, it just never returns. |
|
@toran414 do youse Unix or TCP sockets for connecting to the Docker daemon ? |
|
Unix |
|
Same problem as everyone here, also with the official Docker version:
|
|
Seems to work when using a TCP socket instead of UNIX socket to connect to docker daemon. |
|
We see the same problem but with different images - with some hanging 90% at the time others work just fine. Unfortunately we can't just switch to TCP sockets so a workaround / fix would be highly appreciated. Looking at |
|
@jljouannic @pepperbob I just pushed an experimental version 0.21-SNAPSHOT to Maven snapshot repository with an update to jnr-unixsocket 0.18 which we use for connecting via Unix Socket. It would be awesome if you could try this plugin version on your CI to check whether it makes any difference. |
|
Still happening on 0.26.0 |
|
@andreasaronsson I tried jnr-unixsocket-0.19 already but I got tons of issues with closed streams. Not sure that's d-m-p doing something wrong or the lib. Going to investigate this further, as it seems that it contains some fix which could be related to this issue. |
|
Also using 0.26.0. |
|
@andreasaronsson thanks for reporting back. When I have on day in row to spend I will debug this very annoying issue again. This should be fixed eventually ! |
|
This could be related to how the postgres container starts postgres twice? On my build, container initializes like this If I wait for the second "database system is ready" log message, which comes from the second postgres startup, like @benhubert:
then stopping hangs. If I wait for the first "database system is ready" log, which comes from the first postgres startup,
(and add a dirty then stopping does not hang. |
|
It's still reproduced in the latest 0.27.2 version of the plugin. Is there any decent solution or a workaround of this problem? |
|
I'm facing this issue too by launching maven within a jenkins docker container that is configured to reuse the host docker. For example: |
|
I'm facing this issue with 0.28.0 as well. My workaround is to run socat and connect to docker with TCP. This setup works for me in OSX and Ubuntu14: |
|
Same problem here with 0.30.0. Is there any news on this topic? |
|
I tested various versions of the plugin while doing integration tests in an endless loop (as the problem doesn't show up with every test run) and found that my use-case works fine with version 0.24.0. With 0.25.0 and above, stopping the container sometimes blocks forever. My use-case is:
|
|
Having the same bug too, when trying to spawn a RabbitMQ side-car for integration testing. My build is sometimes stuck for the stop step of the plugin:
Threaddump of the Java process attached thread-dump.txt (seems stuck with jnr): Maven debug informations just before the process is hanging: Sometimes (maybe ~1% of the time), the build doesn't have the problem, and here are the maven details in theses cases: Also worth noting that the problem doesn't happen with a side-car that is much faster to shutdown (i.e. a redis side-car instead of a rabbitMQ), here are the logs for it: Any news/known work-around for this problem? Please shout if there is more debug information I can provide. |
|
I was having a similar problem with one of my containers. On OSX, (using tcp to communicate with daemon), no problem. On Linux Jenkins slave, (using unix socket to communicate with daemon), docker:stop caused docker container to stop but never completed the maven mojo. The image configuration includes Once I added an integration test which caused additional docker logs output to occur between startup and shutdown, the docker:stop works consistently on both local development and Jenkins hosts. |
|
Disabling connection pooling and passing -Ddocker.maxConnections=1 to maven makes it work for me |
|
apologies celebrated too early, went from 100% reproducing it to like 20% tho, trying to make standalone reproducer but those all magically seem to work as well .. :/ |
|
Can reproduce on 2 machines with https://github.com/selckin/mvn-docker-reproduce |
|
jnr socket with the following (existing) patch applied seems to fix it ! |
|
@selckin cool, very good find ! Now lets push jnr-unixsocket over there to get the fix merged and hope for a new release. |
|
@selckin and all others: jnr-unixsocket 0.25 contains the potential fix and I updated on master to it. Also, I pushed a |
|
@rhuss tested with my build, everything looks good, thanks! |
|
I'm still experiencing this issue with 0.33.0 of the fabric8 docker-maven-plugin. |
|
@bmarcj do you have public build that can reproduce it ? |
|
I'm afraid I've nothing public I can provide. I'm getting this with a MySQL docker image on Ubuntu 18.04, if that's any help. |
|
Hi! @selckin, I'm facing same situation. ' ` |
and others
Hello,
I have a maven project and the docker plug in configuration for it is listed below. Starting works fine.
But after the integration tests are done and the plug in shuts down the container it just hangs with this output forever
[INFO] --- docker-maven-plugin:0.14.1:stop (stop) @ wh-cdc-web ---
08:52:37.421 DB> 2016-09-06 06:52:37 UTC [1-2] LOG: received smart shutdown request
08:52:37.421 DB> 2016-09-06 06:52:37 UTC [10-2] LOG: autovacuum launcher shutting down
The container is not running anymore so it was shutdown, it's just the plugin waiting for something.
PlugIn Version: 0.15.16
Docker version 1.12.1, build 23cf638
Dockerfile.txt
The text was updated successfully, but these errors were encountered: