Add support for data persistence #351
Conversation
maimaisie
requested review from
samjsong,
rvmiller89 and
vsinghal13
and removed request for
samjsong and
rvmiller89
| @@ -1,5 +1,4 @@ | |||
| compress gzip | |||
| flush_interval "#{ENV['FLUSH_INTERVAL']}" | |||
| flush_thread_count "#{ENV['NUM_THREADS']}" | |||
| chunk_limit_size "#{ENV['CHUNK_LIMIT_SIZE']}" | |||
There was a problem hiding this comment.
should we increase chunk_limit_size to 1mb knowing that the chunks will no longer be compressed?
There was a problem hiding this comment.
we do still do compression for memory buffer and file buffer without persistence, it might be really tricky to have two defaults depending on the buffer type..
There was a problem hiding this comment.
OK I'm fine with keeping this for now. Ideally we figure out how to continue to use gzip since these metrics will compress heavily (repeated dimensions + metadata)
There was a problem hiding this comment.
Could this be the issue we're seeing? fluent/fluentd#2619
Have we tried upgrading our fluentd image from v1.6.3 to v1.8.1 and see if it resolves the issue?
There was a problem hiding this comment.
yep, v1.8.1 fixed the issue! will revert the compression changes
Description
This PR adds support for data persistence to a persistent volume per fluentd pod. After discussing with the team, we decided to switch from using deployments to statefulsets because statefulsets allows easier setup of persistent volume.
Testing performed