filter fluentd container logs #402
Conversation
vsinghal13
requested review from
samjsong,
frankreno,
rvmiller89 and
maimaisie
| @@ -8,51 +8,64 @@ | |||
| time ${[record["log"].split(/[\n\t]+/)[0]].map! {|item| JSON.parse(item)["time"]}.join("")} | |||
| </record> | |||
| </filter> | |||
| # only match fluentd logs based on fluentd container name and label them @FLUENT | |||
There was a problem hiding this comment.
we should try to see if these logs are already tagged by Fluentd itself: https://docs.fluentd.org/deployment/logging#capture-fluentd-logs
There was a problem hiding this comment.
The label @FLUENT_LOG uses <match fluent.**> which will not stop fluentd from outputting to stdout. We will then have duplicate fluentd logs.
Ref: fluent/fluentd#641 (comment)
There was a problem hiding this comment.
Did it not work starting fluentd with -o /dev/null to output logs to file /dev/null instead of stdout? https://docs.fluentd.org/deployment/logging#output-to-log-file
…ub.com/SumoLogic/sumologic-kubernetes-collection into vsinghal-filter-fluentd-container-logs
|
@rvmiller89 have modified the existing pipeline @samjsong Still kept the filter plugin confs in separate files to keep the fluentd pipeline dry and easy to read. |
| @type grep | ||
| <regexp> | ||
| key log | ||
| pattern /\[info\]|\[fatal\]|drop_oldest_chunk/ |
There was a problem hiding this comment.
Wait, we want to ingest info logs here?
There was a problem hiding this comment.
good catch, I had changed that to test locally. Will fix it to be error.
…ub.com/SumoLogic/sumologic-kubernetes-collection into vsinghal-filter-fluentd-container-logs # Conflicts: # deploy/kubernetes/fluentd-sumologic.yaml.tmpl
| # only ingest fluentd logs of levels: {error, fatal} and warning messages if buffer is full | ||
| @type grep | ||
| <regexp> | ||
| key log |
There was a problem hiding this comment.
need to indent this line and the next?
…ub.com/SumoLogic/sumologic-kubernetes-collection into vsinghal-filter-fluentd-container-logs
| <filter **> | ||
| # sumologic kubernetes metadata enrichment filter plugin | ||
| <filter containers.**> |
There was a problem hiding this comment.
@samjsong @rvmiller89 I suspect it is because of this change.
Description
This PR changes the default fluentd log level to be
infoand modifies the fluentd pipeline to ingest only specific fluentd logs into Sumo.Testing performed