Fluentbit metrics service and monitor is not created

[vipinjn24]

Describe the bug:
I tried to install the operator with the logging enabled.

logging:
  enabled: true
  controlNamespace: logging
  fluentd:
    metrics:
      serviceMonitor: true
  fluentbit:
    metrics:
      serviceMonitor: true

Expected behaviour:
It should create the metrics service and service monitor for fluentbit also.

Steps to reproduce the bug:
Use the above configuration and it does create Logging resource and FluentBit Agent resource but FluentBit Agent despite having the service monitor it doesnt create the service and monitor.

Additional context:
The generated manifest for Fluentbit

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  annotations:
    meta.helm.sh/release-name: logging-operator
    meta.helm.sh/release-namespace: logging-operator
  labels:
    app.kubernetes.io/instance: logging-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: logging-operator
    app.kubernetes.io/version: 6.1.0
    helm.sh/chart: logging-operator-6.1.0
  name: logging-operator
spec:
  metrics:
    serviceMonitor: true

Environment details:

• Kubernetes version (e.g. v1.15.2): v1.31.13
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): Baremetal
• logging-operator version (e.g. 2.1.1): 6.1.0
• Install method (e.g. helm or static manifests): Operator using helm chart
• Logs from the misbehaving component (and any other relevant logs): Provided in additional context
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data: Provided in additional context

/kind bug

[vipinjn24]

Also look at: Logging Dashboard do not have the cluster variable populated in Baremetal Kube cluster.
https://grafana.com/grafana/dashboards/7752-logging-dashboard/

do i need to raise a new bug for this?

[bovy89]

6.1.0 introduced an undocumented backwards-incompatible change (?). You should also specify metrics.enabled: true

     metrics:
+      enabled: true
       serviceMonitor: true

also, be aware that serviceMonitor (job name on prometheus) has changed from fluentbit-agent-fluentbit-monitor to fluentbit-agent-fluentbit-metrics

[vipinjn24]

6.1.0 introduced an undocumented backwards-incompatible change (?). You should also specify metrics.enabled: true

     metrics:
+      enabled: true
       serviceMonitor: true

also, be aware that serviceMonitor (job name on prometheus) has changed from fluentbit-agent-fluentbit-monitor to fluentbit-agent-fluentbit-metrics

so it created the monitor but it doesnt add the interval 15s for scraping, due to this some visualizations with rate 1m is not working.
right now i did that manually using a command after adding the logging crd objects and the resources are created by logging operator.

Also look at: Logging Dashboard do not have the cluster variable populated in Baremetal Kube cluster. https://grafana.com/grafana/dashboards/7752-logging-dashboard/

do i need to raise a new bug for this?

I also would like to discuss about this, i cannot see any cluster dimension in the metric "node_cpu_seconds_total" and related metrics.

another observation is that "node_filesystem_size_bytes" and "node_filesystem_free_bytes" does not populate for {pod=~"$fluentd",mountpoint="/buffers",endpoint="buffer-metrics"}, is there something i am missing?

[bovy89]

Without metrics.enabled: true, the fluentbit serviceMonitor is not going to be created

[vipinjn24]

Without metrics.enabled: true, the fluentbit serviceMonitor is not going to be created

this is done, please see my previous comment