Fix Multiline support for kubernetes collection #313
Conversation
…rict matching only to messages beginning with a date.
…/sumologic-kubernetes-collection into vsinghal-fix-multiline
|
Can you merge master as well, I think your branch hasn't picked up the libsonnet stuff |
| enable_ruby | ||
| renew_record true | ||
| <record> | ||
| log ${record["log"].split(/[\n\t]+/).map! {|item| JSON.parse(item)["log"]}.join("")} |
There was a problem hiding this comment.
just confirming: This won't cause issues if somehow the log coming from containers.** doesn't have log/stream/time keys?
There was a problem hiding this comment.
This is only expecting the log key and using that to generate the stream and time key value pairs from that, so it should be fine.
There was a problem hiding this comment.
But if the string contained in log doesn't have stream and time, i.e. if the user is not using docker, then this wouldn't work, correct? Maybe we should document somewhere that we only support docker for now?
There was a problem hiding this comment.
Actually it would work even if the log doesn't have stream and timekeys. Its just that if the key is not present, the JSON.parse value will be empty string for the key.
| regex: | ||
| - name: multi_line | ||
| regex: (?<log>^{"log":"\d{4}-\d{1,2}-\d{1,2} \d{2}:\d{2}:\d{2}.*) | ||
|
|
There was a problem hiding this comment.
@vsinghal13 are we using these values somewhere else? because we have defined same thing in deploy/helm/fluent-bit-overrides.yaml
There was a problem hiding this comment.
The fluent-bit-overrides.yaml is an auto-generated file for all the fluent-bit values that have been overridden in values.yaml . This is done for non-helm users.
Description
In order to fix the multiline support the following config was added:
Multiline log :
Single line log:
Testing performed