-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fluentd DaemonSet degrades and stops sending logs #87
Comments
Hi @rloporp , thanks for reporting this issue. |
Sure, it's a kOps-managed cluster running on AWS EC2 instances:
We've investigated on our own, and disk IOPS seems to be a bottleneck. We're testing memory-based buffers instead and the results look promising. |
@rloporp thank you. |
About the log size, they are currently truncated to 5000 characters (in terms of byte size, I think they're UTF-8 characters, but I'm not 100% sure) + minimal Sping Boot headers. About the error and Logz.io's response: This is an old error (after one month, I could only retrieve a couple of lines):
Current errors:
Zoom in on the last (
|
Hi @rloporp , |
We found that after some time the
fluentd
DaemonSet degrades and stops sending logs to Logz.io. This degradation may take somewhere between a couple of hours to 2 or 3 days to happen. We haven't been able to run a working DaemonSet for a whole week.Research
We've confirmed on our monitoring systems that the problem happens as soon as the error (
fluentd.output.status.num.errors.gauge
) and retry (fluentd.output.status.retry.count.gauge
) counters are higher than 0. The DaemonSet remains totally healthy in terms of memory and CPU (no throttling).Environment
We've found this behavior with release 1.3.0. We've tried release 1.4.0 and the problem arises faster.
Workaround
When that happens, logs stop being sent to Logz.io and the only known workaround is to restart the affected DaemonSet.
The text was updated successfully, but these errors were encountered: