out_loki: Allow specifiling possibly empty record accessor for labels

[toshipp]

Is your feature request related to a problem? Please describe.

I use fluentbit with loki output plugin, and configure labels with $app like as:

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser cri
        Tag kube.*
        Buffer_Max_Size 1MB
        Mem_Buf_Limit 128MB
        Skip_Long_Lines On
        DB /run/fluent-bit/tail.db
    [FILTER]
        Name kubernetes
        Match kube.*
        # Merge_Log On
        # Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Annotations Off
    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes
    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under labels
        Add_prefix label_
    [FILTER]
        Name modify
        Match kube.*
        Copy label_app.kubernetes.io/name app
        Copy label_app app
        Remove_wildcard label_
    [OUTPUT]
        Name loki
        Match kube.*
        Host loki
        labels namespace=$namespace_name, pod=$pod_name, container=container_name, node_name=$host, $stream, $app

But some applications use neither app.kubernetes.io/name nor app label, so the record accessor $app may be empty.
In this case, fluentbit dumps logs like [ warn] [output:loki:loki.0] empty record accessor key translation for pattern: $app many times.

I already know that $app is possibly empty, so the warning is undesirable.

Describe the solution you'd like

I want to specify possibly empty records accessor for labels like below.

[OUTPUT]
        Name loki
        labels $app?

The notation ? means the field is possibly empty.

Describe alternatives you've considered

• Decreasing log level
  It is not desirable because I would miss some warnings.
• Use rewrite tag filter
  It can split logs between having $app and not having one by rewrite tag filter, and then use dedicated output sections for each log.
  I think this is hard to maintain.
• Use modify and nest filters and auto_kubernetes_labels of loki plugin
  Loki plugin has auto_kubernetes_labels, it appends stream labels under $kubernetes[labels] to loki labels.
  I embed $app label into $kubernetes[labels] by modify and nest filters, and set auto_kubernetes_labels to true,
  the $app label is used as loki label.
  I do not want to use all labels from kubernetes for loki labels, so this solution is not perfect.

Additional context

[patrick-stephens]

Why not use a condition to check if it is empty on the modify filter and only fill it then? https://docs.fluentbit.io/manual/pipeline/filters/modify#example-2-conditionally-add-and-remove

[toshipp]

It is not needed because Copy function checks if COPIED_KEY exists implicitly.
My intent is that if app.kubernetes.io/name field exists, use it to output app label, If not and app field exists, use it, and if both do not exist, DONT output app label.

BTW, the snippet is my entire workflow, but not minimal for the request.
An important point is that I want to use app field to loki label if it exists, and do not want to see warn log if it is empty.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[toshipp]

This is not my blocker, but a feature request. Noone has an interest in this, close is ok.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[PRIHLOP]

I am interested in this too. And also this warning message creates a big amount of logs from Fluentbit and it causes ingestion overload.

Please, reopen this issue.

[PRIHLOP]

unstale

[drbugfinder-work]

Fixed in #8680