Removing log field from tail

[antonionappi88]

Hello,
i'm trying to pushing some logs in JSON format reading from a file to OpenSearch. I would like to remove the field log but i'm not able to do so. I read in another github issue that it seems not possible but I found also other sources that they state that actually could work but so far I didn't have luck.

[{"date":1684246716.630026,"log":"\"{\"timestamp\":\"2023-05-16T12:48:35.606Z\",\"sequence\":8467,\"loggerClassName\":\"org.infinispan.client.hotrod.logging.Log_$logger\",\"loggerName\":\"org.infinispan.HOTROD\",\"level\":\"INFO\",\"message\":\"ISPN004014: New server added(MY_IP:11111), adding to the pool.\",\"threadName\":\"Thread-0\",\"threadId\":56,\"mdc\":{},\"ndc\":\"\",\"hostName\":\"keycloak-v20-0\",\"processName\":\"QuarkusEntryPoint\",\"processId\":1}\""}]

Any clue if this is possible and if yes how ?

This is my configuration:

[SERVICE]
        Daemon Off
        Flush 1
        Log_Level trace
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
        Parser_File /etc/fluent-bit/parsers.conf
        Plugins_File plugins.conf
[INPUT]
        Name tail
        Path /keycloak-logs/keycloak.log
        Tag kube-keycloak
        Parser json
[FILTER]
        Name modify
        Match kube-keycloak
        Add producer sso
        Add type keycloak-server-dev
[OUTPUT]
        Name stdout
        Match kube-keycloak
        Format json

Thank you
Antonio

[leonardo-albertovich]

You have a typo in your config file, it should be Parsers_File not Parser_File and then you can add this :

[FILTER]
        Name nest
        match kube-keycloak
        Operation lift
        Nested_under log

And that will lift all of what's inside the log key out of it and remove the empty key.

[leonardo-albertovich]

This minimal configuration file demonstrates exactly what you need :

fluent-bit.conf

[SERVICE]
        Flush 1
        Log_Level info
        Parsers_File parsers.conf

[INPUT]
        Name tail
        Path test_file.json
        Tag kube-keycloak
        Parser json
        read_from_head on

[FILTER]
        Name nest
        match kube-keycloak
        Operation lift
        Nested_under log

[OUTPUT]
        Name stdout
        Match *
        Format json

parsers.conf

[PARSER]
    Name   json
    Format json
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

test_file.json

{"the key name": "this is a log line"}

[antonionappi88]

Hello @leonardo-albertovich ,
thanks a lot for the reply! Among all the attempts I tried also that solution and It didn't work. Indeed the issue was on my side, I was manually adding a new json line to my file but this json was malformed. I've just realized when i added read_from_head on. It gots good logs from the top ignoring my manual insertion.

Thanks again!