Backup for broken chunks occupying too much disk space #4295
rustlingwind
started this conversation in
General
Replies: 1 comment 1 reply
-
Just found the feature already existing: #2117 |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi guys,
I just found our AWS EKS (Fargate) pods frequently raised disk space issue and I noticed that the size of /tmp directory of each pod increased quickly.
5 days before:
/# df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 30G 17G 12G 59% /
5 days after:
/# df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 30G 24G 4.8G 83% /
The space was ate by fluentd tmp backup files, the pods age is 6 days, the ate speed is about 1Gi per day.
/tmp/fluent/backup/worker0/out_cloudwatch_logs_containers# du -h --max-depth=1
6.7G .
/tmp/fluent/backup/worker0/out_cloudwatch_logs_containers# ls -l | grep "^-" | wc -l
4948
Is there a way to disable or limit the fluentd backup for broken chunks? Any impact in case disable or limit it?
Or is there a rotation for these backup of broken chunks? So that these tmp files can be cleaned up automatically when size or duration reach a threshold.
Much appreciated if any advice.
P.S. Regarding the tmp backup for broken chunks:
#1952
https://www.fluentd.org/blog/fluentd-v1.2.0-has-been-released
Beta Was this translation helpful? Give feedback.
All reactions