Lost event with fluentd logging driver #43690
Labels
area/logging
kind/bug
Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
version/20.10
Comments
thaJeztah
added
area/logging
version/20.10
kind/bug
|
Hi @thaJeztah, This issue has been labelled There was an issue with older versions of the fluent-bit implementation not handling SIGTERM properly, but this has been fixed in fluent-bit v1.8.10. (see: fluent/fluent-bit#2610) Step to reproduce: # docker versionStart a fluent-bit container: # docker run -it --rm --name flb -p 24224:24224 fluent/fluent-bit:2.0.10 /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1Start a nc container with --log-opt fluentd-request-ack=true: # docker run -it --rm --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 --log-opt fluentd-request-ack=true --link flb busybox sh -c 'while true; do date; nc -vz flb 24224; sleep 1; done'Stop the fluent-bit container: # date -u; time docker stop -t 10 flb; date -uStart a new fluent-bit container: # docker run -it --rm --name flb -p 24224:24224 fluent/fluent-bit:2.0.10 /fluent-bit/bin/fluent-bit -v -i forward -o stdout -f 1According to the above output, the Docker tried to stop the fluent-bit container at 05:45:45. The fluent-bit caught signal (SIGTERM) at 05:45:45, and stopped at 05:45:46. And the timestamp of the earliest record received after the restart was 05:45:46. We can see that the record with timestamp 05:45:45 has been lost. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Lost event while using fluentd as the logging driver configured with "fluentd-async": "true".
Steps to reproduce the issue:
Test configuration incudes docker configured with fluentd as the logging driver (async môde), fluent bit installed on the same host as a log aggregator and a sample app that actually generates some logs.
For the sample app we use a simple nginx web container. This works perfectly since it simply generates a log event for every received HTTP request:
docker run -it --rm -d -p 8080:80 --name web nginxWe start by verify that our app is alive and reachable. A couple of initial requests are visible in container logs and fluent bit output aswell:
Next, we stop fluent bit and send only one request which is, as expected, visible in container logs, but doesn't get propagated to fluent bit:
If we look at tcpdump output we can see a reset packet as the last step of commumnication. This is, of course, expected since fluent bit is down:
At this point, if no other requests/log events are generated, everything maintains a status quo. Tcpdump will show no traces of any other traffic from docker, even if we restart fluent bit.
Let's continue by reactivating fluent bit. Again, nothing happens until we generate new request(s). After we do generate another request, we can see that it's regularly logged, but that one event, which was sent after fluent bit was stopped, seems to be lost:
Detailed tcpdump output confirms that only one log message is sent from docker's end.
Furthermore, if we repeat previous steps, but send multiple requests while fluent bit is offline, by looking at the tcpdump output we can see that docker starts to periodically probe our log destination while trying to resend logs which where buffered, just as we assume it would do:
After we activate fluent bit again, all buffered logs are resent and captured as expected, except the one that was generated immediately after fluent bit was stopped.
So, it seems that there is this one log message that actually does get lost if our logging destination is unavailable for whatever reason. It doesn't look as much, but I'd assume that this is not expected behaviour.
Describe the results you received:
Lost event in case of log destination outage.
Describe the results you expected:
Docker buffers all log events and preserves them while log destionation is unavailable.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version:Output of
docker info:Docker deamon settings:
Fluent bit configuration:
Additional environment details (AWS, VirtualBox, physical, etc.):
VMware, VirtualBox
The text was updated successfully, but these errors were encountered: