init: add multiline_parser

[konoui]

Issue #, if available:
#537

Description of changes:
The init process should handle MULTILINE_PARSER in the same way as PARSER.

Currently, a validation error will occur in fluent bit.
https://github.com/fluent/fluent-bit/blob/v2.0.5/src/fluent-bit.c#L862

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[PettitWesley]

@konoui Thank You!

Next step: Someone from our team needs to pull this change and test it before we merge. @matthewfala @Claych

[PettitWesley]

Sorry we have neglected this. I will work on running the integ test soon.

[kakakakakku]

@PettitWesley
Hi, Do you have any plans to release this pull request? I'm also encountering the same error, so it would be really helpful if this could be merged soon😀 Thanks.

[PettitWesley]

@kakakakakku I apologize, this has been on my backlog for a while, but I've been backed up with different high impact issues/bugs/fixes.

I've bumped the priority on my backlog for this... sorry that's all I can do for now :(

[fermani]

Is there any chance to get this merged? We had to build a custom image with the workaround described at #537 (comment). It is not a big deal but we prefer to have this feature in the init tag image by default.

Thank you.

[PettitWesley]

I completed testing with the following. I checked that the files were downloaded and that FLB started up and ran and checked the entrypoint.

            "environment": [
                {
                    "name": "aws_fluent_bit_init_s3_1",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-input.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_2",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/multiline-parser.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_3",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/multiline-filter.conf"
                }
            ],

[MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    flush_timeout 1000
    # Regex rules for multiline parsing
    # ---------------------------------
    #
    # configuration hints:
    #
    #  - first state always has the name: start_state
    #  - every field in the rule must be inside double quotes
    #
    # rules   |   state name   | regex pattern                   | next state name
    # --------|----------------|--------------------------------------------------
    rule         "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule         "cont"          "/^\s+at.*/"                     "cont"

[FILTER]
    name                  multiline
    match                 *
    multiline.key_content message
    multiline.parser      go, multiline-regex-test

Resulting entrypoint:

exec /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /init/fluent-bit-init.conf -R /init/fluent-bit-init-s3-files/multiline-parser.conf

[PettitWesley]

Thanks @konoui and apologies that we took soooo long to merge this!