Dynamically create output folder for each module logs using deployment name in fluentBit

[raghibomar786]

I've several modules deployed in my kubernetes cluster and I use fluentBit to read logs for all modules from /var/log/containers folder and write them to a file on a persistent volume. I'm able to successfully write all log files to a single folder in my output path. Now, I want to write the logs of each module in a separate subfolder in my output path named as deployment/release name of each module. The log file is named in the following format pod-name_namespace-name_deployment-name (i.e, each of the name is separated by an underscore _). How do I extract this information from my log file name and use it in the output path in FluentBit configuration?

My current configuration looks like this:

    [SERVICE]
        Flush 1
        Log_Level info
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Tag kube.*
        Mem_Buf_Limit 50MB
        Skip_Long_Lines Off
        Read_from_Head true
        Inotify_Watcher false

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Labels Off
        Annotations Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name file
        Match kube.*
        Path /output/path
        Mkdir True

And this makes my tag value of the form kube.var.log.containers.pod-name_namespace-name_chart-name, I want to do something like:

[OUTPUT]
        Name file
        Match kube.*
        Path /output/path/${tag.deployment_name}/${tag.pod-name}.log
        Mkdir True

What is the correct way of doing this? From the documentation it seems like Tag_Regex can be used for doing this but how it can be used is not very clear.

[raghibomar786]

@patrick-stephens , you mentioned:
Rewrite tag is what I used previously to handle paths into tail - you can use the tag as the path for output so if you set up your tag correctly then that should work, rather than try to extract bits of the tag just make it what you want.

This creates a separate output file named after the tag for each service but does not create subfolders for each deployment-name. And since new tag can only contain a-z,A-Z, 0-9 and .-, (ref), I cannot put a '/' in my tag that may create a subdirectory.

[patrick-stephens]

I don't think there's a good way to do this slightly bespoke approach at the moment: you could use a LUA filter to create the file that way potentially as a side effect.

Personally I would probably just use a hyphen rather than the slash although it means you lose the subdirectory (but that could be covered by a cron job copy/move as needed later).

[raghibomar786]

Thanks @patrick-stephens , creating file LUA filter may not be very efficient (we're expecting relatively frequent logs from around 20 services). CronJob works but won't be real-time which is a requirement from our customer. I see that Fluentd file output plugin supports record accessor, one way I can think is to forward my records from fluentBit to fluentD which then write to output file. But then that includes additional maintenance of an extra image and its container. I've raised a feature request for the same in FluentBit as well.