Namespaced FluentBit CRs, generic custom plugin field for Fluentd CRs

[adiforluls]

Is your feature request related to a problem? Please describe.

Hi, was looking at fluent-operator. It is a promising project and a flexible product! There are a few enhancements I think are important though:

• FluentBit:

  1. Namespaced CRs for FluentBit: I really like the namespace level CRs for Fluentd to process logs coming out of a namespace. I'd like to see something similar for FluentBit as well, namely a namespace level FluentBitConfig, Filter and Output (Input as well if it makes sense). The reason here is that with FluentBit only mode, it feels like I'm missing out on an important feature and can't process logs at namespace level. Unless there's a very good reason to have only cluster-level configurations for FluentBit, I think this should be done.
  2. Match ClusterFluentBitConfig by label instead of name.

• Fluentd:

  Generic custom plugin type for Fluentd: Just like for FluentBit Input/Filter/Output resources, a custom plugin field where one can specify raw configurations of Fluentd plugins that have not been implemented yet. It's also important in my opinion.

Describe the solution you'd like

• Namespaced FluentBitConfig, Input/Filter/Output (or Filter/Output) for FluentBit Daemonset. The FluentBitConfig can match to namespace and cluster-level resources (behaviour identical to FluentdConfig).
• Match FluentBitConfig via labels for multiple log routing paths.
• Generic custom plugin field for Fluentd Filter and Output.

Additional context

Note: Open to discussion on the above listed enhancements. If there's an agreement on at-least one of these enhancements, would you guys be willing to accept contributions? I'd like to know the contribution guidelines as well.

[benjaminhuo]

1. Namespaced CRs for FluentBit: I really like the namespace level CRs for Fluentd to process logs coming out of a namespace. I'd like to see something similar for FluentBit as well, namely a namespace level FluentBitConfig, Filter and Output (Input as well if it makes sense). The reason here is that with FluentBit only mode, it feels like I'm missing out on an important feature and can't process logs at namespace level. Unless there's a very good reason to have only cluster-level configurations for FluentBit, I think this should be done.

@adiforluls It's nice to discuss the clustered and namespaced CRD of FluentBit.
I have an explanation in #516 (comment):
FluentBit is namespaced CRD which controls the namespaced Fluent Bit Daemonset. ClusterInput, ClusterParser, ClusterFilter, and ClusterOutput are all cluster-wide CRDs, this is because FluentBit is acting as a global agent to collect logs on each K8s node which requires cluster-wide privileges.

Fluentd can receive logs from the global fluentbit daemonset and thus it can handle logs globally or locally in a namespace.
We use an excellent fluentd plugin https://github.com/banzaicloud/fluent-plugin-label-router to route logs to different namespaces and this plugin also makes it possible to route logs for multiple namespaces(each fluentdConfig or clusterFluentdConfig is a separate pipeline), but fluentbit doesn't have such a plugin.

We're adding a new mode called Collector for fluentbit to receive logs from network by deploying a fluentbit statefulset (not from the k8s node itself like the fluentbit daemonset) in #484 (comment) and maybe it makes sense to change the collector CRDs to namespaced CRDs @wanjunlei @wenchajun

[benjaminhuo]

• Generic custom plugin type for Fluentd: Just like for FluentBit Input/Filter/Output resources, a custom plugin field where one can specify raw configurations of Fluentd plugins that have not been implemented yet. It's also important in my opinion

Adding custom plugins for fluentd might be a good idea and @adiforluls you're welcome to work on this!
You can refer to the design of fluentbit customPlugin: https://github.com/fluent/fluent-operator/blob/master/docs/best-practice/custom-plugin.md

The reason we didn't add fluentd custom plugin is that unlike fluentbit, you'll have to customize the fluentd image to add the plugin before you can use it via custom fluentd plugin.

[adiforluls]

1. Namespaced CRs for FluentBit: I really like the namespace level CRs for Fluentd to process logs coming out of a namespace. I'd like to see something similar for FluentBit as well, namely a namespace level FluentBitConfig, Filter and Output (Input as well if it makes sense). The reason here is that with FluentBit only mode, it feels like I'm missing out on an important feature and can't process logs at namespace level. Unless there's a very good reason to have only cluster-level configurations for FluentBit, I think this should be done.

@adiforluls It's nice to discuss the clustered and namespaced CRD of FluentBit. I have an explanation in #516 (comment): FluentBit is namespaced CRD which controls the namespaced Fluent Bit Daemonset. ClusterInput, ClusterParser, ClusterFilter, and ClusterOutput are all cluster-wide CRDs, this is because FluentBit is acting as a global agent to collect logs on each K8s node which requires cluster-wide privileges.

Fluentd can receive logs from the global fluentbit daemonset and thus it can handle logs globally or locally in a namespace. We use an excellent fluentd plugin https://github.com/banzaicloud/fluent-plugin-label-router to route logs to different namespaces and this plugin also makes it possible to route logs for multiple namespaces(each fluentdConfig or clusterFluentdConfig is a separate pipeline), but fluentbit doesn't have such a plugin.

We're adding a new mode called Collector for fluentbit to receive logs from network by deploying a fluentbit statefulset (not from the k8s node itself like the fluentbit daemonset) in #484 (comment) and maybe it makes sense to change the collector CRDs to namespaced CRDs @wanjunlei @wenchajun

@benjaminhuo At some point in the future, do you think it's possible to have namespaced CRs/multiple log pipelines for FluentBit Daemonset if there is a plugin, or not if we follow your explanation here
If the new FluentBit StatefulSet will have namespaced config CRs, can that idea be extended to the Daemonset also (i.e. without requiring a FluentBit plugin for namespace logging/multiple pipelines) in order to receive container logs from k8s node?

[adiforluls]

• Generic custom plugin type for Fluentd: Just like for FluentBit Input/Filter/Output resources, a custom plugin field where one can specify raw configurations of Fluentd plugins that have not been implemented yet. It's also important in my opinion

Adding custom plugins for fluentd might be a good idea and @adiforluls you're welcome to work on this! You can refer to the design of fluentbit customPlugin: https://github.com/fluent/fluent-operator/blob/master/docs/best-practice/custom-plugin.md

The reason we didn't add fluentd custom plugin is that unlike fluentbit, you'll have to customize the fluentd image to add the plugin before you can use it via custom fluentd plugin.

@benjaminhuo Here's the PR for custom fluentd plugin #555, please review.

[benjaminhuo]

@benjaminhuo Here's the PR for custom fluentd plugin #555, please review

@adiforluls Great work, thank you!