-
Notifications
You must be signed in to change notification settings - Fork 354
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug:
The error was encountered when upgrading from 6.1.0 to 6.2.0. In our testsuite we wait for the fluentbit DaemonSet which never starts. When investigating the Fluentd Pod, we got a nil pointer dereference in the logs.
Expected behaviour:
The DaemonSet should start without any errors.
Steps to reproduce the bug:
Don't set bufferVolumeMetrics in the FluentdConfig.
Environment details:
- kind: kind v0.30.0 go1.24.6 linux/amd64
- logging-operator version: 6.2.0
- Install method: helm with terraform module kostomization which lets us use helm charts with terraform
- Logs from the misbehaving component (and any other relevant logs):
{"level":"error","ts":"2025-11-26T15:16:00Z","msg":"Observed a panic","controller":"logging","controllerGroup":"logging.banzaicloud.io","controllerKind":"Logging","Logging":{"name":"cluster-logging"},"namespace":"","name":"cluster-logging","reconcileID":"2298ccc7-bf43-4e42-89a2-8ec44affc937","panic":"runtime error: invalid memory address or nil pointer dereference","panicGoValue":"\"invalid memory address or nil pointer dereference\"","stacktrace":"goroutine 785 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x248fb18, 0xc001660fc0}, {0x1e0f780, 0x35fea80})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.2/pkg/util/runtime/runtime.go:132 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:198 +0x10e\npanic({0x1e0f780?, 0x35fea80?})\n\t/usr/local/go/src/runtime/panic.go:783 +0x132\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).monitorBufferServiceMetrics(0xc0018076b0)\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/service.go:184 +0x4c\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).Reconcile(0xc0018076b0, {0x248fb18, 0xc001661f50})\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/fluentd.go:258 +0x116e\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile(0xc00080b140, {0x248fb18, 0xc001660fc0}, {{{0x0?, 0xc000c64008?}, {0xc000f80f20?, 0x0?}}})\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:293 +0x1d8a\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc001660f30?, {0x248fb18?, 0xc001660fc0}, {{{0x0, 0x0?}, {0xc000f80f20?, 0x0?}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216 +0x165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x24b19e0, {0x248fb50, 0xc0004255e0}, {{{0x0, 0x0}, {0xc000f80f20, 0xf}}}, 0x0)\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461 +0x3ad\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x24b19e0, {0x248fb50, 0xc0004255e0})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421 +0x21b\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1 in goroutine 78\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:292 +0x26b\n","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logPanic\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.2/pkg/util/runtime/runtime.go:142\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:198\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:783\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:262\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:925\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).monitorBufferServiceMetrics\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/service.go:184\ngithub.com/kube-logging/logging-operator/pkg/resources/fluentd.(*Reconciler).Reconcile\n\t/usr/local/src/logging-operator/pkg/resources/fluentd/fluentd.go:258\ngithub.com/kube-logging/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/usr/local/src/logging-operator/controllers/logging/logging_controller.go:293\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2025-11-26T15:16:00Z","msg":"Reconciler error","controller":"logging","controllerGroup":"logging.banzaicloud.io","controllerKind":"Logging","Logging":{"name":"cluster-logging"},"namespace":"","name":"cluster-logging","reconcileID":"2298ccc7-bf43-4e42-89a2-8ec44affc937","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"info","ts":"2025-11-26T15:16:00Z","logger":"logging.validation","msg":"found detached fluentd aggregator, making association","logging":"cluster-logging","name":"cluster-logging"}
- Resource definition (possibly in YAML format) that caused the issue, without sensitive data:
apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentdConfig
metadata:
generation: 1
name: cluster-logging
namespace: logging
resourceVersion: "1110"
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
scheduling.breuninger.com/placement-selector: cluster-logging
matchLabelKeys:
- pod-template-hash
topologyKey: kubernetes.io/hostname
bufferStorageVolume:
pvc:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: standard
configReloaderResources:
limits:
memory: 10Mi
requests:
cpu: 10m
memory: 10Mi
labels:
scheduling.breuninger.com/placement-selector: cluster-logging
metrics:
prometheusRules: true
prometheusRulesOverride:
- alert: FluentdRecordsCountsHigh
labels:
severity: warning
- alert: FluentdNodeDown
labels:
severity: warning
- alert: FluentdQueueLength
labels:
severity: warning
serviceMonitor: true
pdb:
minAvailable: 1
resources:
limits:
memory: 512Mi
requests:
cpu: 50m
memory: 512Mi
scaling:
replicas: 2
security:
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
topologySpreadConstraints:
- labelSelector:
matchLabels:
scheduling.breuninger.com/placement-selector: cluster-logging
matchLabelKeys:
- pod-template-hash
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotScheduleSetting the bufferVolumeMetrics.enabled in the FluentdConfig explicitly to either true or false, solves this problem.
/kind bug
fbreckle and torvitas
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working