-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
in_tail: Add option file_cache_advise to reduce file cache usage #8422
base: master
Are you sure you want to change the base?
Conversation
fluent-bit.conf
Valgrind & Debug Level Output:
|
a5280ee
to
1c5c24e
Compare
fluent/fluent-bit#8422 Signed-off-by: Richard Treu <richard.treu@sap.com>
Docs PR: fluent/fluent-bit-docs#1291 |
1c5c24e
to
df4e409
Compare
fluent/fluent-bit#8422 Signed-off-by: Richard Treu <richard.treu@sap.com>
@cosmo0920 would you be able to take a look at this? |
df4e409
to
109b0ce
Compare
@agup006 @fujimotos @leonardo-albertovich @patrick-stephens |
f206f30
to
27549b0
Compare
I had to resolve a minor merge-conflicts with the new master, so I needed to update this. No change in logic. @cosmo0920 @braydonk @agup006 |
Tested this in a large cluster where we had frequent oomkill events and can confirm it has helped a lot! Any reason we wouldn't want to default this option to On? |
@brenix Thank you for testing. In the long-term, I also definitely would recommend to turn this option on by default, but for the start I don't want create any regressions. From my side I can also report that in our clusters with many hundreds of Fluent Bit instances running, this fix definitely reduced the OOMkill rate (caused by the high file cache usage with tail plugin) to virtually zero - I am using this for almost 3 months. |
This commit will add option file_cache_advise to the tail plugin. It allows to set the posix_fadvise in POSIX_FADV_DONTNEED mode. This will reduce the usage of the kernel file cache. Feature is enabled by default. Signed-off-by: Richard Treu <richard.treu@sap.com>
8542fac
27549b0
to
8542fac
Compare
fluent/fluent-bit#8422 Signed-off-by: Richard Treu <richard.treu@sap.com>
@edsiper @agup006 @patrick-stephens @fujimotos @leonardo-albertovich @lecaros Docs-PR is here: fluent/fluent-bit-docs#1291 |
fluent/fluent-bit#8422 Signed-off-by: Richard Treu <richard.treu@sap.com>
This PR will add option file_cache_advise to the tail plugin. It allows to set the posix_fadvise in POSIX_FADV_DONTNEED mode. This will drastically reduce the usage of the kernel file cache.
This can be necessary when running in K8s environments, where the file cache is accounted to the total memory usage of the pod (see: kubernetes/kubernetes#43916). I/O intensive loads (like Fluent Bit) might run into OOM-kills because of filling the file cache with reads of the tail plugin.
Enter
[N/A]
in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Documentation
fluent/fluent-bit-docs#1291
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.