Add namespace_labels to kubernetes metadata

[peterbosalliandercom]

What is the problem:
We are using fluentbit for our logcollection. The output flows through fluentd to logstash to ES. In ES we find that we only get a subset of kubernetes metadata in the index. We are using the kubernetes.namespace_labels to identify the which tenant the logging comes from. We use the following label which is placed by the Capsule (https://capsule.clastix.io) operator to identify the tenant: capsule.clastix.io/tenant: tenantxxx ). In logstash we want to split indexes per tenant by using this namespace_label.

What do we like:
Fluentbit grabbing namespace labels and output it to fluentd.

Background
We are using the BanzaiCloud logging-operator (https://banzaicloud.com/docs/one-eye/logging-operator) so the configuration fixed and defaults to the use of fluentbit for kubernetes metadata, we cannot change that so there is now way to fix it without breaking the operator functionality. (related issue: kube-logging/logging-operator#704) The only way is by overriding everything the operator does (rbac) and fluentd config, but then the whole point of using the operator is lost.

[bilbof]

Same issue #3865

kubernetes.namespace_labels aren't fetched by the kubernetes filter plugin (https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes) unlike fluentd (https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/blob/f131c26b60999a0902c6219f10d290cccfbd03da/lib/fluent/plugin/kubernetes_metadata_common.rb#L51) so a missing feature for folks upgrading.

[patrick-stephens]

It sounds like a good idea so feel free to submit a PR and ping me if you need approval for CI.

[genofire]

I would really like this feature.

[zanloy]

This is the only thing holding me back from switching from fluentd to fluent-bit. We have a large multi-tenant k8s environment and use namespace labels to help process logs and forward them to the correct OpenSearch backend depending on the tenant label. I would imagine this is a fairly common use-case where you might have different logging pipelines based on namespace variables.

[sa9226]

We are looking for this feature too. Any ETA for this feature, please?

[lanrekkeeg]

I was able to resolve this issue by implementing a Lua filter. In the plugin's Lua filter, I extracted the namespace name from the Kubernetes logs using the field record["kubernetes"]["namespace_name"]. With this namespace name, I made an API call to Kubernetes to retrieve the corresponding namespace information since the service account was already mounted.After obtaining the namespace information, I added an additional field called namespace_meta to the record. This field contained the fetched metadata from the namespace.

[sa9226]

@lanrekkeeg , Thank you so much for your suggestion. Would u mind sharing piece of code as I'm totalluy new to Lua. It will be really helpful.

[sa9226]

@lanrekkeeg, we came with lua script to make api call and running into /fluent-bit/config/containerd.lua:14: module 'ssl' not found: trying to see if you built any custom image adding these missing modules. Please suggest. Appreciate your help

[gopikishanm]

Hi @lanrekkeeg @sa9226 can you share the pseudo code to read namespace labels, I am facing the same issue

[loggspark]

@genofire Here is the blog with sample code with logging. Hope it will help.
https://techandtutor.wordpress.com/2023/09/14/how-to-fetch-namespace-metadata-of-pod-via-fluentbit/

[patrick-stephens]

Ta for that @loggspark , we could also look to cross-post that to the Fluent blog if you're up for it? @agup006

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[ryanohnemus]

@patrick-stephens Can you add the exempt-stale label and re-open this one when you're back in the new year? I have a PR open and ready for review for this.