Parameterize buffer settings for Fluentd Logs/Metrics #99
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
samjsong
previously approved these changes
Jul 29, 2019
lei-sumo
previously approved these changes
Jul 29, 2019
| add_timestamp "#{ENV['ADD_TIMESTAMP']}" | ||
| timestamp_key "#{ENV['TIMESTAMP_KEY']}" | ||
| proxy_uri "#{ENV['PROXY_URI']}" | ||
| @include buffer.output.conf |
There was a problem hiding this comment.
This is a nice one. Can we also do some similar abstraction for other configurations? e.g.
data_type metrics
metric_data_format prometheus
disable_cookies true
There was a problem hiding this comment.
Sure I can DRY this up more. It's too bad fluentd only supports include and not macro templates that we could parameterize.
There was a problem hiding this comment.
Done, can you approve again ?
| @@ -43,7 +43,9 @@ RUN gem sources --clear-all \ | |||
| # Default settings for log collection | |||
| ENV LOG_FORMAT "fields" | |||
| ENV FLUSH_INTERVAL "5s" | |||
| ENV NUM_THREADS "1" | |||
| ENV NUM_THREADS "4" | |||
There was a problem hiding this comment.
Does it mean every plugin will have 4 threads? Does it harm if we set it to an even higher value?
There was a problem hiding this comment.
Yes it does. I picked 4 because that's what the collector uses. I recommend we try it for a while, work on instrumenting some metrics so we can see queue size, and only increase it once we can observe the effects.
samjsong
approved these changes
Jul 29, 2019
psaia
pushed a commit
to psaia/sumologic-kubernetes-collection
that referenced
this pull request
May 25, 2021
psaia
pushed a commit
to psaia/sumologic-kubernetes-collection
that referenced
this pull request
May 25, 2021
* Instructions for using the adutil * Update security team's filter (SumoLogic#99) * PUBP-8182: provision elections collector (SumoLogic#98) * PUBP-8182: provision elections collector * PUBP-8182: provision elections collector * PUBP-8182: renaming module to match repo name * PUBP-8182: renaming output Co-authored-by: Malcolm Jones <17786500+mlclmj@users.noreply.github.com> * Add logging for AD parameters (SumoLogic#95) * Log the vars subitted * fix wording, enumerate commands directly * drone sign Co-authored-by: Marcin Suterski <msuterski@users.noreply.github.com> * added collector for nytm/dv-drone in TF (SumoLogic#100) * Adding collector for spg-inyt-subscription-api (SumoLogic#101) * remove 'manageIndexes' from samizdat role (SumoLogic#102) * Add ADR regarding AD groups and nesting (SumoLogic#12) * adr 7 * Update and rename 0007-controlling-role-access-through-ad.md to 0008-controlling-role-access-through-ad.md * Feature/add collector int fallback role (SumoLogic#104) * adding new collector to the role * adding new collector to the role * Adutil instructions moved to dv-docs * update link Co-authored-by: Marcin Suterski <msuterski@users.noreply.github.com> Co-authored-by: Shreelekha Tanna <shree.tanna@nytimes.com> Co-authored-by: Malcolm Jones <17786500+mlclmj@users.noreply.github.com> Co-authored-by: Prune Sebastien THOMAS <prune@lecentre.net> Co-authored-by: satyabodharao <47533173+satyabodharao@users.noreply.github.com> Co-authored-by: Shawn Bower <shawn.bower@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
gzipcompression for fluentd to use internally when buffering to maximize amount of buffering we can doTested bringing up Fluentd pods and ensured log and metric collection is working.