panic after 30 minutes running #60
Comments
|
Thanks for reporting, i'm on it! |
|
Problem identified: this is when docker API is not reachable, we get a nil object. To reproduce: launch dockerbeat watching a docker daemon and then stop the docker daemon. It crashed dockerbeat. |
This was referenced Jan 15, 2016
|
@pcallewaert all should be ok now. Can you try and give us a feedback? Thankx |
|
It seems the problem is not resolved (tested with latest develop docker container)
|
|
I did not succeed in reproducing this bug.. After 3 hours of dockerbeat running on my personal host (some containers and images) -> no pb. I should try to on a similar environment than yours. @erwanncloarec any idea? |
|
@marminthibaut what is your go version ? I'll build a new environment ; try to reproduce the issue ... |
|
Same as you, running for more than 2 hours with docker image, no problem |
|
The way I see, the error is from the error comes from getContainerEvent at that line of code: Obviously it is not the common.Time that is doing the null pointer, so it is stats that is null. The stats is comming from the chan populated by the docker client library, so we have to invastigate in what case we have a null in the stats. |
|
@erwanncloarec just using this command: @gpolaert imageid is d13cb6762b06c57e9a5d39bb80b2efe23bb4d2124d524de4c3ccbcf78a996040 We do use https://github.com/garethr/garethr-docker for container deployment. We're not sure, but we think it can be related when we deploy/update containers in a puppet run. However, at that point, the docker daemon is never restarted (this i'm pretty sure) so the docker api should be available at all time. It's now running for 19 hours without problems, but we are not really deploying new stuff on it now. |
|
Can you try to reproduce this behaviour with the beta2 version to ensure that we are working on the same version? Docker image: ingensi/dockerbeat:1.0.0-beta2 Thanks |
|
Damn, after 43 hours running on beta 2, it crashed also.. :/ |
|
Hey there I may experience the same error. After some time the following log appears and the container dies: |
|
Thank you for reporting them this bug. I'll look at this in few days. |
|
Hello, I've seen couple panics too. Originally, I thought they were caused by |
|
I'm currently trying to reproduce the issue, but I can't find a way to do it. Any ideas so I can start working on debugging this issue ? |
|
We saw the issue, when we used it in the docker environment. Maybe because there are too many events? I don't know the source of libbeat so that could be an issue. Cheers |
|
I was watching the kibana monitoring when something hit me : the crashs seems to happen when a container is stopped and/or removed. Maybe the crash occurs if we remove a container after container listing but before getting statistics. We're currently investigating on this. |
|
Thanks to @HipsterWhale we can now reproduce the issue: With that script, dockerbeat will eventually stop. |
|
Whoa 😯 Well then it could be a solution to collect all data first and then deliver them? Or how would you deliver partial data? |
ShibaOnTheMoon
added a commit
to ShibaOnTheMoon/dockerbeat
that referenced
this issue
Mar 8, 2016
…terval between listing and statistics retrieving
|
I've made a pull request with the fix for this bug, it will be released in a beta soon. |
|
@HipsterWhale |
|
Thanks @HipsterWhale ! Awesome work I just reopen this issue until @buehler @pcallewaert and @zstyblik report if it's ok (or not). |
|
@HipsterWhale can you explain why this bug occurred? |
|
The crash occurred because the dockerclient return the stats to nil with an error to nil too if the container have been removed in the interval between the listing and the statistics retrieving. I've just added the case to handle this error with a message to the log to say that something abnormal happened. |
|
👍 |
|
I ran the new image from the develop branch. By now, it seems to be working 👍 |
|
Thanks! |
|
Sorry, I didn't have time to give this a test, but it's good to hear it's working. |
|
We've tested it and it seems to be working. Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
We are trying to run develop branch on our servers, but are getting after 30 minutes panics, and the container stops. Seems a deadlock situation..
We are running this command:
docker run -d -v /var/run/docker.sock:/var/run/docker.sock --link elasticsearch-001:elasticsearch ingensi/dockerbeat:developdocker version:
Attached you find the error log
dockerbeat-fail.zip
The text was updated successfully, but these errors were encountered: