Skip to content

FluentBit does not start on AKS after upgrade to logging-operator:6.1.0 #2123

@husira

Description

@husira

Describe the bug:
After upgrading to logging-operator 6.1.0 (from 6.0.3) the FluentBit pods do not start anymore.
We are using AKS v1.33.3 and the deployment is done via Helm-Chart. With the old logging-operator version 6.0.3 FluentBit just works fine.

Expected behaviour:
FluentBit should work

Steps to reproduce the bug:
Deployment of the latest logging-operator version (6.1.0) on K8s 1.33.3

Additional context:

Logging k8s resource:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  labels:
    app.kubernetes.io/instance: logging-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: logging-operator
    app.kubernetes.io/version: 6.1.0
    helm.sh/chart: logging-operator-6.1.0
  name: logging-operator
  namespace: logging-operator
spec:
  clusterDomain: cluster.local.
  controlNamespace: logging-operator
  enableRecreateWorkloadOnImmutableFieldChange: true
  fluentd:
    bufferStorageVolume:
      pvc:
        spec: {}
    bufferVolumeImage:
      pullPolicy: Always
    bufferVolumeMetrics:
      prometheusRules: true
      serviceMonitor: true
    bufferVolumeResources:
      limits:
        memory: 50M
      requests:
        cpu: 1m
        memory: 10M
    configCheckResources:
      limits:
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 128Mi
    configReloaderImage:
      pullPolicy: Always
    configReloaderResources:
      limits:
        memory: 128Mi
      requests:
        memory: 32Mi
    fluentOutLogrotate:
      age: "10"
      enabled: true
      path: /fluentd/log/out
      size: "10485760"
    image:
      pullPolicy: Always
    livenessProbe:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; if [ ! -e /buffers ]; then
              exit 1;
            fi; touch -d date -d "@$(($(date +%s) - $LIVENESS_THRESHOLD_SECONDS))" /tmp/marker-liveness; if [ -z "$(find /buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
              exit 1;
            fi;
      initialDelaySeconds: 600
      periodSeconds: 60
    metrics:
      prometheusRules: true
      serviceMonitor: true
    readinessDefaultCheck:
      bufferFileNumber: false
      bufferFreeSpace: true
      bufferFreeSpaceThreshold: 90
      failureThreshold: 1
      initialDelaySeconds: 5
      periodSeconds: 30
      successThreshold: 3
      timeoutSeconds: 3
    resources:
      limits:
        memory: 1024Mi
      requests:
        cpu: 50m
        memory: 50Mi
    scaling: {}
    security:
      podSecurityContext:
        fsGroup: 101
        runAsGroup: 101
        runAsNonRoot: true
        runAsUser: 100
        seccompProfile:
          type: RuntimeDefault
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        readOnlyRootFilesystem: false
status:
  configCheckResults:
    de68d16c: true
  watchNamespaces:
    - '*'

FluentbitAgent k8s resource

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  labels:
    app.kubernetes.io/instance: logging-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: logging-operator
    app.kubernetes.io/version: 6.1.0
    helm.sh/chart: logging-operator-6.1.0
  name: logging-operator
  namespace: logging-operator
spec:
  image:
    pullPolicy: Always
  livenessProbe:
    httpGet:
      path: /
      port: 2020
  metrics:
    prometheusRules: true
    serviceMonitor: true
  readinessProbe:
    httpGet:
      path: /api/v1/metrics/prometheus
      port: 2020
  resources:
    limits:
      memory: 400Mi
    requests:
      cpu: 50m
      memory: 50Mi
  security:
    podSecurityContext:
      runAsUser: 0
      seccompProfile:
        type: RuntimeDefault
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - FOWNER
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
  tolerations:
  - effect: NoSchedule
    operator: Exists

Pods

kubectl get pod -n logging-operator

NAME                               READY   STATUS             RESTARTS       AGE
logging-operator-95744c9d7-4frv2   1/1     Running            0              15m
logging-operator-fluentbit-bphm7   0/1     CrashLoopBackOff   9 (11s ago)    15m
logging-operator-fluentbit-hb2vg   0/1     CrashLoopBackOff   9 (31s ago)    15m
logging-operator-fluentbit-mchdg   0/1     CrashLoopBackOff   9 (11s ago)    15m
logging-operator-fluentbit-mqwtk   0/1     CrashLoopBackOff   9 (11s ago)    15m
logging-operator-fluentd-0         2/2     Running            1 (101s ago)   15m

Logs
After some time the FluentBit pod gets a SIGTERM. No further information is visible in the logs of the FluentBit or the logging-operator pods.

logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [engine] caught signal (SIGTERM)
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ warn] [engine] service will shutdown in max 5 seconds
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input] pausing tail.0
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input] pausing storage_backlog.1
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [engine] service has stopped (0 pending tasks)
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input] pausing tail.0
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input] pausing storage_backlog.1
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [output:forward:forward.0] thread worker #0 stopping...
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [output:forward:forward.0] thread worker #0 stopped
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [output:forward:forward.0] thread worker #1 stopping...
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [output:forward:forward.0] thread worker #1 stopped
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322521 watch_fd=1
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322527 watch_fd=2
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322507 watch_fd=3
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=775460 watch_fd=4
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=778229 watch_fd=5
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774495 watch_fd=6
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=796821 watch_fd=7
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=803814 watch_fd=8
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774602 watch_fd=9
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=779982 watch_fd=10
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774676 watch_fd=11
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774574 watch_fd=12
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774625 watch_fd=13
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=776182 watch_fd=14
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774526 watch_fd=15
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=776127 watch_fd=16
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=776161 watch_fd=17
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3626750 watch_fd=18
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627886 watch_fd=19
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3628575 watch_fd=20
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3625497 watch_fd=21
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627861 watch_fd=22
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3629079 watch_fd=23
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627880 watch_fd=24
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3625564 watch_fd=25
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3619522 watch_fd=26
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627252 watch_fd=27
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627813 watch_fd=28
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322449 watch_fd=29
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322451 watch_fd=30
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322450 watch_fd=31
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3661476 watch_fd=32
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3628793 watch_fd=33
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627330 watch_fd=34
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=776130 watch_fd=35
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3619585 watch_fd=36
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627720 watch_fd=37
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322463 watch_fd=38
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=780078 watch_fd=39
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774478 watch_fd=40
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=774729 watch_fd=41
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3640264 watch_fd=42
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627183 watch_fd=43
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627255 watch_fd=44
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627297 watch_fd=45
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=803811 watch_fd=46
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=780038 watch_fd=47
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3614489 watch_fd=48
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=780044 watch_fd=49
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2323274 watch_fd=50
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=800088 watch_fd=51
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627712 watch_fd=52
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3626934 watch_fd=53
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322548 watch_fd=54
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322508 watch_fd=55
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3617458 watch_fd=56
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3616607 watch_fd=57
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3618840 watch_fd=58
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3619198 watch_fd=59
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=779984 watch_fd=60
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=780017 watch_fd=61
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3629054 watch_fd=62
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3627716 watch_fd=63
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322493 watch_fd=64
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=796747 watch_fd=65
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=803037 watch_fd=66
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3624437 watch_fd=67
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322555 watch_fd=68
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322542 watch_fd=69
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3618905 watch_fd=70
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3619605 watch_fd=71
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3621124 watch_fd=72
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322531 watch_fd=73
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3628253 watch_fd=74
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3628975 watch_fd=75
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=3639679 watch_fd=76
logging-operator-fluentbit-5z5l5 fluent-bit [2025/10/15 10:07:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=2322547 watch_fd=77
kubectl describe pod -n logging-operator logging-operator-fluentbit-bphm7

Name:                 logging-operator-fluentbit-bphm7
Namespace:            logging-operator
Priority:             0
Priority Class Name:  medium
Service Account:      logging-operator-fluentbit
Node:                 aks-nodepool-19102626-vmss00004y/10.1.0.4
Start Time:           Wed, 15 Oct 2025 12:56:51 +0200
Labels:               app.kubernetes.io/instance=logging-operator
                      app.kubernetes.io/managed-by=logging-operator
                      app.kubernetes.io/name=fluentbit
                      controller-revision-hash=5b5c6876bd
                      pod-template-generation=5
Annotations:          checksum/fluent-bit.conf: 14de2df863185dc709be8b1f106a8cad8e63c3677eb1618e3d0c4354cf049bbe
Status:               Running
IP:                   10.244.3.66
IPs:
  IP:           10.244.3.66
Controlled By:  DaemonSet/logging-operator-fluentbit
Containers:
  fluent-bit:
    Container ID:  containerd://039a56b641382e2a195f23acf6f8e257403b8054e5c9cfa949572f188fb5df24
    Image:         ghcr.io/fluent/fluent-bit:4.0.3
    Image ID:      ghcr.io/fluent/fluent-bit@sha256:d6cc0a0a45a4fc61acde53939ed91672ed6fada7bb87710976309c0831232a79
    Port:          <none>
    Host Port:     <none>
    Command:
      /fluent-bit/bin/fluent-bit
      -c
      /fluent-bit/etc-operator/fluent-bit.conf
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 15 Oct 2025 13:17:33 +0200
      Finished:     Wed, 15 Oct 2025 13:18:02 +0200
    Ready:          False
    Restart Count:  11
    Limits:
      memory:  400Mi
    Requests:
      cpu:        50m
      memory:     50Mi
    Liveness:     http-get http://:2020/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:2020/api/v1/metrics/prometheus delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /buffers from buffers (rw)
      /fluent-bit/etc-operator from config (rw)
      /tail-db from positiondb (rw)
      /var/lib/docker/containers from varlibcontainers (ro)
      /var/log/ from varlogs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-drb9d (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  varlogs:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
  varlibcontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  logging-operator-fluentbit
    Optional:    false
  positiondb:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  buffers:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-drb9d:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 :NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  21m                   default-scheduler  Successfully assigned logging-operator/logging-operator-fluentbit-bphm7 to aks-nodepool-19102626-vmss00004y
  Normal   Pulled     21m                   kubelet            Successfully pulled image "ghcr.io/fluent/fluent-bit:4.0.3" in 503ms (503ms including waiting). Image size: 45295166 bytes.
  Normal   Pulled     20m                   kubelet            Successfully pulled image "ghcr.io/fluent/fluent-bit:4.0.3" in 652ms (652ms including waiting). Image size: 45295166 bytes.
  Normal   Pulled     19m                   kubelet            Successfully pulled image "ghcr.io/fluent/fluent-bit:4.0.3" in 965ms (965ms including waiting). Image size: 45295166 bytes.
  Warning  Unhealthy  19m (x8 over 21m)     kubelet            Liveness probe failed: Get "http://10.244.3.66:2020/": dial tcp 10.244.3.66:2020: connect: connection refused
  Normal   Pulled     19m                   kubelet            Successfully pulled image "ghcr.io/fluent/fluent-bit:4.0.3" in 471ms (471ms including waiting). Image size: 45295166 bytes.
  Normal   Started    18m (x5 over 21m)     kubelet            Started container fluent-bit
  Normal   Created    18m (x5 over 21m)     kubelet            Created container: fluent-bit
  Normal   Pulled     18m                   kubelet            Successfully pulled image "ghcr.io/fluent/fluent-bit:4.0.3" in 470ms (470ms including waiting). Image size: 45295166 bytes.
  Warning  BackOff    11m (x31 over 18m)    kubelet            Back-off restarting failed container fluent-bit in pod logging-operator-fluentbit-bphm7_logging-operator(8ef01935-0c7f-4ce9-945b-784045d11c2b)
  Normal   Killing    6m19s (x10 over 20m)  kubelet            Container fluent-bit failed liveness probe, will be restarted
  Normal   Pulling    77s (x11 over 21m)    kubelet            Pulling image "ghcr.io/fluent/fluent-bit:4.0.3"
  Warning  Unhealthy  76s (x56 over 21m)    kubelet            Readiness probe failed: Get "http://10.244.3.66:2020/api/v1/metrics/prometheus": dial tcp 10.244.3.66:2020: connect: connection refused

Environment details:

  • Kubernetes version: v1.33.3
  • Cloud-provider/provisioner: AKS
  • logging-operator version: 6.1.0
  • Install method: Helm
  • Logs from the misbehaving component (and any other relevant logs):

/kind bug

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions