Unable to set both a custom Index name and Date (Fluent-bit x OpenSearch)

[rdvansloten]

Bug Report

Describe the bug
When configuring fluent-bit forOpenSearch, I can't seem to get both the Kubernetes namespace and date in the index. I want my Indexes to look like this:

MY_KUBERNETES_CLUSTER-$NAMESPACE-$DATE

So for a cluster called "mycluster" and namespace "development" this would be the result:
mycluster-namespace-2024-11-26

When enabling Logstash, I can set the date, but the clustername+namespace rendering does not work as intended. I can only do clustername+date or namespace+date. It doesn't accept a plain string next to the variable.

To Reproduce

• Open the values.yaml file for fluent-bit Helm chart
• Navigate to config.outputs
• Put in

  outputs: |
    [OUTPUT]
        Name opensearch
        Match kube.*
        Host opensearch-cluster.opensearch.svc.cluster.local
        Port 9200
        HTTP_User my-user
        HTTP_Passwd redacted
        Logstash_Format Off
        Index mycluster-$kubernetes['namespace_name']
        Retry_Limit False
        tls On
        tls.verify Off
        Suppress_Type_Name On 

Expected behavior
To be able to format my indexes the way I see fit and combine all the given puzzle pieces.

Your Environment

• Version used: Fluent Bit v3.2.1
• Configuration: In description
• Environment name and version (e.g. Kubernetes? What version?): AKS 1.30.0
• Server type and version:
• Operating System and version: Linux
• Filters and plugins: Whatever comes with the Helm chart

[rdvansloten]

I used a "hack" but it puts the items in the wrong order.

        Logstash_Format On
        Logstash_Prefix_Key $kubernetes['namespace_name']
        Logstash_Prefix_Separator -mycluster-
        Logstash_DateFormat %Y-%m-%d

resulting in development-mycluster-2024-11-26

But it looks super dirty

[Crazyigor1987]

Have similar problems. My workaround was to create a luascript, which appends the prefix to the namespace.
Filter and Output:

[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/namespace-renamer.lua
    call    add_namespace_prefix
    time_as_table    true

[Output]
    Name    opensearch
    Match_Regex    (?:kube|service)\.(.*)
    Host    xxxxxxxxxxxxxxxxxxxxxx
    Port    443
    Logstash_Format    true
    Logstash_Prefix    fluentbit-
    Logstash_Prefix_Key    $namespaceindex
    Suppress_Type_Name    true

The Lua Script:

function add_namespace_prefix(tag, timestamp, record)
	if record["kubernetes"] ~= nil and record["kubernetes"]["namespace_name"] ~= nil then
    		record["namespaceindex"] = "fluentbit-" .. record["kubernetes"]["namespace_name"]
	end
		return 1, timestamp, record
end

Results to

fluentbit-{NAMESPACE}-YYYY.mm.dd

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.