Multiline Support and Parser for systemd input

[shahbour]

Below is between feature request and question.

My Aim is to be able to log/save Spring boot Application running inside kubernetes with multi line support specially for Java Error.

I did try setting Docker log-driver to JSON but that raised a problem where multiline is sent in separate JSON lines and we could not combine them.

So now I reverted back to journald to check if can solve the problem but now I got another problem
1- fluentbit.io/parser Annotation is not working any more (not sure if that is a misconfiguration from me or it is a new feature request)
2- Multiline support: I think it is the same as in JSON, we can't combine multiple lines using systemd input.

apiVersion: v1
data:
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               host.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        K8S-Logging.Parser  On
        K8S-Logging.exclude True
        Use_Journal         On
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
  input-kubernetes.conf: |
    [INPUT]
        Name           systemd
        Tag            host.*
        Path              /var/log/journal
        Read_From_Tail  true
  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Retry_Limit     False
  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log
    [PARSER]
        Name        springboot
        Format      regex
        Regex       /^(?<date>[0-9]+-[0-9]+-[0-9]+\s+[0-9]+:[0-9]+:[0-9]+.[0-9]+)\s+\[(?<user_name>.*)\]\s+(?<log_level>[Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)\s+(?<pid>[0-9]+)\s+---\s+\[(?<thread>.*)\]\s+(?<class_name>.*)\s+:\s+(?<message>.*)$/
        Time_Key    date
        Time_Format %Y-%m-%d %H:%M:%S.%L

[dene14]

Same problem... Would like to use custom parsers per app to collect multiline exceptions (nodejs, java, python, etc) in a signle event.
I'm using helm chart "stable/fluent-bit" but don't see how it can work for that purpose... please elaborate

[ReDim42]

Maybe my reply in anohter Issue help u to get further. It is the tail plugin for Inputs.

#821 (comment)

[jlpettersson]

I have opened a PR for #821 if that helps. #852

[kiich]

Super interested in the PR by @jlpettersson as we are also facing same issue.

[tangmin823]

How about Multiline Configuration Parameters， from Input plugin tail?

[edsiper]

(try out the new multiline filter, new tail mode documented below)

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

• Multiline Parsers / Concepts and Configuration
• Tail + New Multiline Core Feature
• New Multiline Filter

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!

[kiich]

Amazing feature! Well done team! 👏

[DrissiReda]

(try out the new multiline filter, new tail mode documented below)

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

* [Multiline Parsers / Concepts and Configuration](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing)

* [Tail + New Multiline Core Feature](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support)

* [New Multiline Filter](https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace)

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!

@edsiper Thanks for this feature, but it doesn't work with annotations, if I use the following annotation:

fluentbit.io/parser: custom-ml

I get the error on fluent-bit:

[2022/02/24 13:56:57] [ warn] [filter:kubernetes:kubernetes.0] annotation parser 'custom-ml' not found

But with normal parsers it works normally. And If I use a normal parser then multiline support won't work.