Skip to content

Nil pointer dereference caused in 6.2.0 when bufferVolumeMetrics is unset #2145

@am9zZWY

Description

@am9zZWY

Describe the bug:
The error was encountered when upgrading from 6.1.0 to 6.2.0. In our testsuite we wait for the fluentbit DaemonSet which never starts. When investigating the Fluentd Pod, we got a nil pointer dereference in the logs.

Expected behaviour:
The DaemonSet should start without any errors.

Steps to reproduce the bug:
Don't set bufferVolumeMetrics in the FluentdConfig.

Environment details:

  • kind: kind v0.30.0 go1.24.6 linux/amd64
  • logging-operator version: 6.2.0
  • Install method: helm with terraform module kostomization which lets us use helm charts with terraform
  • Logs from the misbehaving component (and any other relevant logs):
{"level":"error","ts":"2025-11-26T15:16:00Z","msg":"Observed a panic","controller":"logging","controllerGroup":"logging.banzaicloud.io","controllerKind":"Logging","Logging":{"name":"cluster-logging"},"namespace":"","name":"cluster-logging","reconcileID":"2298ccc7-bf43-4e42-89a2-8ec44affc937","panic":"runtime error: invalid memory address or nil pointer dereference","panicGoValue":"\"invalid memory address or nil pointer dereference\"","stacktrace":"goroutine 785 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x248fb18, 0xc001660fc0}, {0x1e0f780, 0x35fea80})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.2/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:198 +0x10e\npanic({0x1e0f780?, 0x35fea80?})\n\t/usr/local/go/src/runtime/panic.go:783 +0x132\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).monitorBufferServiceMetrics(0xc0018076b0)\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/service.go:184 +0x4c\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).Reconcile(0xc0018076b0, {0x248fb18, 0xc001661f50})\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/fluentd.go:258 +0x116e\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile(0xc00080b140, {0x248fb18, 0xc001660fc0}, {{{0x0?, 0xc000c64008?}, {0xc000f80f20?, 0x0?}}})\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:293 +0x1d8a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc001660f30?, {0x248fb18?, 0xc001660fc0}, {{{0x0, 0x0?}, {0xc000f80f20?, 0x0?}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216 +0x165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x24b19e0, {0x248fb50, 0xc0004255e0}, {{{0x0, 0x0}, {0xc000f80f20, 0xf}}}, 0x0)\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461 +0x3ad\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x24b19e0, {0x248fb50, 0xc0004255e0})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421 +0x21b\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1 in goroutine 78\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:292 +0x26b\n","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logPanic\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.2/pkg/util/runtime/runtime.go:142\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:198\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:783\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:262\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:925\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).monitorBufferServiceMetrics\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/service.go:184\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/fluentd.go:258\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:293\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2025-11-26T15:16:00Z","msg":"Reconciler error","controller":"logging","controllerGroup":"logging.banzaicloud.io","controllerKind":"Logging","Logging":{"name":"cluster-logging"},"namespace":"","name":"cluster-logging","reconcileID":"2298ccc7-bf43-4e42-89a2-8ec44affc937","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"info","ts":"2025-11-26T15:16:00Z","logger":"logging.validation","msg":"found detached fluentd aggregator, making association","logging":"cluster-logging","name":"cluster-logging"}
  • Resource definition (possibly in YAML format) that caused the issue, without sensitive data:
apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentdConfig
metadata:
  generation: 1
  name: cluster-logging
  namespace: logging
  resourceVersion: "1110"
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            scheduling.breuninger.com/placement-selector: cluster-logging
        matchLabelKeys:
        - pod-template-hash
        topologyKey: kubernetes.io/hostname
  bufferStorageVolume:
    pvc:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 30Gi
        storageClassName: standard
  configReloaderResources:
    limits:
      memory: 10Mi
    requests:
      cpu: 10m
      memory: 10Mi
  labels:
    scheduling.breuninger.com/placement-selector: cluster-logging
  metrics:
    prometheusRules: true
    prometheusRulesOverride:
    - alert: FluentdRecordsCountsHigh
      labels:
        severity: warning
    - alert: FluentdNodeDown
      labels:
        severity: warning
    - alert: FluentdQueueLength
      labels:
        severity: warning
    serviceMonitor: true
  pdb:
    minAvailable: 1
  resources:
    limits:
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 512Mi
  scaling:
    replicas: 2
  security:
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      seccompProfile:
        type: RuntimeDefault
  topologySpreadConstraints:
  - labelSelector:
      matchLabels:
        scheduling.breuninger.com/placement-selector: cluster-logging
    matchLabelKeys:
    - pod-template-hash
    maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule

Setting the bufferVolumeMetrics.enabled in the FluentdConfig explicitly to either true or false, solves this problem.

/kind bug

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions