[FILTER - parser] JSON parsing of non-JSON causes timestamp to zero out

[zenchild]

Hey all. I'm stuck on a config and cannot figure a way past it and I was hoping the collective would have an idea. I have a fluent-bit instance that is being written to via FORWARD by docker using the fluentd log driver. That part is working great. The issue is, some of the apps have a log that is JSON and some just have strings. I'm trying to parse the log in case it is JSON. That part is working, but the timestamps zero out when the log is a string and I was wondering how to prevent that. Here is an example of the output:

dummy.good: [1683204719.556494379, {"hello":"there"}]
dummy.bad: [0.000000000, {"log":"This is a string message."}]
dummy.good: [1683204720.556250297, {"hello":"there"}]
dummy.bad: [0.000000000, {"log":"This is a string message."}]
dummy.good: [1683204721.555522714, {"hello":"there"}]
dummy.bad: [0.000000000, {"log":"This is a string message."}]

Here is the config that I used to demo the above:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File /fluent-bit/etc/parsers.conf
    HTTP_Server  Off

[INPUT]
    name    dummy
    tag     dummy.good
    dummy   {"log": "{\"hello\": \"there\"}"}
    samples 3

[INPUT]
    name    dummy
    tag     dummy.bad
    dummy   {"log": "This is a string message."}
    samples 3

[FILTER]
    Name parser
    Match dummy.*
    Key_Name log
    Parser docker
    Reserve_Data On

[OUTPUT]
    Name  stdout
    Match *

[OUTPUT]
    Name file
    Match *
    File /applogs/out.log
    Format out_file

I'm am testing this configuration by running:

docker run -it --rm \
    -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \
    -v $(pwd):/applogs \
    --name fluent-test \
    fluent/fluent-bit:latest

Because of the nature of our actual configuration, I am not easily able to match on any particular tag so tag matching would not work in the real-world scenario.

Any insight in how I can get my timestamps to stick around would be greatly appreciated.

[leonardo-albertovich]

Hi @zenchild , I think PR #7323 fixes this issue, do you think you would be able to give the nightly build a try?

[zenchild]

Thanks @leonardo-albertovich! I tested this with both the dummy data and then a real-world dataset we have and it worked great in both scenarios. Much appreciated.