Docker daemon dies when fluent driver can not send messages to saturated fluent server #29
Comments
|
Can you create minimum code and environment to reproduce it? |
|
Can not disagree. I will try and update you. |
|
While I am looking to easier method to reproduce panic let us discuss possible solutions. It seems we need to check net object is not nil before calling method. Is it not main reason of panic ? |
|
@nikolaiderzhak your point looks correct. IMO it's particularly problematic under the situation with many communication errors.
The former will work well, but it makes situation a bit more complex (What should we do when |
Something safe. For me as DevOps engineer goal is make sure docker daemon will not die :) I am trying staging setup to reproduce it now. Will try to emulate slow aggregator responses to trigger that chain. So we could try some fixes. If you propose one I cloud quickly test if it works. |
|
I'm sorry that I don't have enough time to work with this right now... but will merge (and try to merge it into Docker) ASAP when reproduction steps and patches are proposed. |
|
Thanks ! I was not able to reproduce it locally, including docker containers with chatty service and lower buffer limits. So next I will try is to replicate production setup with smaller nodes in AWS. We need it to be done anyway for testing purposes. Question to you. I see f.conn.Write() is wrapped to if f.conn == nil already (since 2013) . |
|
@nikolaiderzhak checking |
|
I see. It can be accessed before it initialised.... |
|
Got critical things on plate. Did not create pre-prod to reproduce it yet. Seems as will start working on his next week. Will update you. |
|
I have synched with R&D team and we have plan to use logstash with gelf docker logging driver as interim solution (long term is kafka driver moby/moby#18604).
So I will stick with gelf driver for now. Hope you or other contributor will find time to fix this production critical bug. Otherwise people will go the sam route I did. |
|
Any updates on this? |
|
Nothing yet. I couldn't create tests or scripts to reproduce it. |
|
@tagomoris we have reproduced . We will provide the data soon. |
|
Great to hear that! Then I'll fix it. |
|
That is good news. How soon for a fix? |
|
This is a show-stopper for our fluentd adoption. Any word here on progress? |
|
Please give me any code to show how to reproduce it... |
|
I managed to reproduce it since we have been hitting this more and more recently. I am not able to reproduce it on my machine but I can make it crash every time on an AWS instance running Amazon Linux and Docker 1.11.2. The time it takes to crash is variable though. My test setup is under Docker however since I am not that good in Go(and ruby) but I guess it will be easy enough for someone on the project to replicate the setup outside of Docker. fluent.conf out_down.rb plugin I spin one container running Fluentd and another container with this command:
Update: I also managed to crash docker 1.11.2 on CentOS 7. I found a workaround for people hitting the problem that may acceptable. In fluentd-0.12.26, a config option named buffer_queue_full_action has been introduced. Setting it to drop_oldest_chunk avoid the problem for us so far but with the obvious consequence of dropping some logs. I let you decide if this is acceptable. |
|
@tagomoris I gave a try to writing a program in Go to reproduce and succeeded. I used the same fluent.conf as previously. Here's the Go source code: It generated the same callstack as the crash within docker: |
|
I'm trying just same code & configuration on my Mac, but it's not reproduced yet (in 10 minutes). |
|
@tagomoris So far, I have reproduced it only in VM on AWS. I run Linux in a VM on my local machine and weren't able to reproduce it either. Since we only experienced it in AWS, where we run our stuff, I developed the test case there. It usually takes between 2-10 minutes to crash. |
|
@EricFortin Thank you for great report! I'll try it on AWS too and some patches next week. |
|
@EricFortin Finally, I made it! (w/ Amazon Linux 2016.03.3) |
|
👍 |
|
I updated the code not to panic but retries to send message, with showing err content. package main
import (
"fmt"
"time"
"github.com/fluent/fluent-logger-golang/fluent"
)
func main() {
f, err := fluent.New(fluent.Config{})
if err != nil {
panic(err)
}
data := map[string]string{
"mylog": "This is my json log!",
}
for {
tm := time.Now()
if err := f.PostWithTime("docker.logs", tm, data); err != nil {
fmt.Printf("error: %s\n", err);
continue;
}
}
}With the change of #30, I got:
With this behavior, fluent-logger-golang (and fluentd logging driver of docker) doesn't stop Docket itself. |
|
Nice ! |
|
I created a pull-request for docker to solve this problem! |
|
That was merged. |
|
@EricFortin can you give your configuration to use buffer_queue_full_action. |
|
@hridyeshpant Yes you add that to your elasticsearch plugin config. It is available because this plugin inherits BufferedOutput. I am using flume on my side but since it also inherits from BufferedOutput, I can use this settings. Here's my config: |
Got very chatty 25 nodes in elasticsearch dockerised cluster.
We use fluentd log driver for elasticssearch containers.
So saturation/slowness on fluent server (which stores data on standalone elasticsearch node in logstash format to view with kibana) led to panic and death of random elasticsearch containers.
From /var/log/syslog:
From fluent forwarder log:
Let me know if you need full stack trace or more details. We really eager to fix that.
Because even if I start bigger fluentd server and standalone elasticsearch node I want to make sure
we will not start loosing data nodes due to panics during log spikes.
Related docker ticker: moby/moby#20960
The text was updated successfully, but these errors were encountered: