[Question] How to properly use in_kubernetes_events plugin

[jfusterm]

Hi guys,

We recently started testing and using the plugin in_kubernetes_events, which is awesome by the way, and given that we were already using fluent-bit in DaemonSet mode to collect the logs from all our Kubernetes nodes, we updated the configuration file to include something like with its own output.

    [INPUT]
        Name            kubernetes_events
        Tag             kubernetes_events
        kube_url        https://kubernetes.default.svc
        DB              /var/log/fluentbit-kubernetes-events.db
        interval_sec    5

It worked out-of-the-box, just seen a slightly Pod CPU increase usage, which got us thinking if maybe we should deploy a separate fluent-bit in Deployment mode just for this plugin. Given that we just need one fluent-bit collecting Kubernetes events, it made sense.

The funny think is that we haven't seen event log duplication when using it as a DaemonSet, but we haven't seen either any recommendation in the documentation on how to run this specific plugin, or if there might be side effects if we have many fluent-bits using the in_kubernetes_events plugin.

Any general recommendation on this topic would be highly appreciate it.

Thanks!

[patrick-stephens]

I think this was discussed in the Slack channel briefly as well and yeah I believe the intention would be to use it as a separate deployment. @pwhelan what do you think?

It may be fine but it may then hit some critical number of events that then triggers some kind of cascade/runaway. I would say it's always better to split concerns if you can, a failure in one place does not then bring everything down - e.g. you would get logs saying your events deployment is failing in this case rather than just no logs or anything.

EDIT: https://fluent-all.slack.com/archives/C0CTQGHKJ/p1692311229788449

[jfusterm]

Thanks @patrick-stephens for your feedback.

[robertvanloenhout]

We are seeing duplicate kubernetes events. Each pod in the fluent bit daemon set will send the event to opensearch.
Is there any documentation on how to use this plugin? Is the only way to make a separate deployment fluent bit installation with its own configuration?

[patrick-stephens]

At the moment yes, there are some ideas about doing leader election or similar.
cc @stevehipwell

[albertoCrego]

Any updates on the ideas? If I deploy a different Deployment with Fluentbit only for the k8s_events I can't have HA because I can run only one pod.

What is the issue with a single deployment?

1. One Pod is enough to get data from a Control plane
2. When Pod is restarted it will grab all missed data

We have a configuration for records tracking

db      - Set a database file to keep track of recorded Kubernetes events
db.sync - Set a database sync method. values: extra, full, normal and off

And I have another question related to the missed (?) events

interval_sec  - Set the polling interval for each channel.                           | 0
interval_nsec - Set the polling interval for each channel (sub seconds: nanoseconds) | 500000000

with default values it will poll every 0.5 seconds?

interval_sec=0
interval_nsec=0,500,000,000

And command kubectl get events -A give much more events that we get by the Fluent Bit.