How to remove key from fluentbit after enabling Merge_Log and Merge_Log_Key ?

[abhinav5656]

Hello guys,

I have to reduce the kubernetes logs index size by removing some field. I have enabled Merge_Log and given the merge_log_key=test. now i want to remove test.log field but not able to remove. i have tried modifier and remove_key but it did not work.

Please help

[patrick-stephens]

What exactly did you try? There are various options to remove keys that should work, e.g. https://docs.fluentbit.io/manual/pipeline/filters/record-modifier#remove-fields-with-remove_key

Can you provide the actual config with stdout output and the full log? I would encourage always to use a local container or similar with dummy input and stdout output to test things quickly and easily: https://calyptia.com/2022/08/08/fluent-bit-tips-tricks/

[abhinav5656]

I have a log json with various keys, I have enabled merge logs so I am getting those keys as columns like log_processed.timestamp, log_proceesed.id, log_processed.response etc out of these columns I need to remove log_processed.response field as it has lot of data.

Below is my configuration

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag              kube.*
        Exclude_Path      /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*, /var/log/containers/istio-proxy, /var/log/containers/istio-init
        Path              /var/log/containers/prod*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     100MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
  filter-keys.conf: |
    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          kubernetes.  
    [FILTER]
        Name                modify
        Match               kube.*
        Remove              log.response
        Remove              log_processed.response
    [FILTER]
        Name                record_modifier
        Match               kube.*
        Remove_key          kubernetes.labels
        Remove_key          kubernetes.namespace_labels
        Remove_key          kubernetes.docker_id
        Remove_key          kubernetes.pod_id
        Remove_key          kubernetes.container_hash
        Remove_key          kubernetes.annotations.sidecar.istio.io/status
        Remove_key          kubernetes.namespace_name
        Remove_key          log_processed.response

it is not removing the log_proccessed.response key.

[patrick-stephens]

This is weird:

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          kubernetes.  

Why are you lifting but then adding a prefix? Your prefix is not then in the remove filter.
Remove the add_prefix stuff and see what happens.