Skip to content

[BUG]CacheRuntime: Dataset stays Failed after the runtime recovers #6160

Description

@btxu-db

What is your environment(Kubernetes version, Fluid version, etc.)

Kubernetes v1.30.0 (kind v0.23.0), Fluid chart fluid-1.1.0 built from 89af5dc9, installed with
--set runtime.cacheruntime.enabled=true.

Describe the bug

Any brief worker outage sets the Dataset to Failed, and it never goes back to Bound. Deleting a
worker pod is enough to reproduce it.

The end state contradicts itself:

Pod:          repro-demo-worker-0   1/1 Running
CacheRuntime: status.worker.phase=Ready, WorkersReady=True
Dataset:      status.phase=Failed, "The ddc runtime is not ready."

lastUpdateTime freezes at the moment of the outage. Forcing a reconcile does not help; only
deleting and recreating the CacheRuntime restores Bound.

The transition is one-way in pkg/ddc/cache/engine/sync.go:87-99:

if !runtimeReady {
    err = e.UpdateDatasetStatus(datav1alpha1.FailedDatasetPhase, runtime, runtimeClass)
    ...
} else if permitSyncEngineStatus {
    err = e.syncDatasetCacheStates(ctx, runtime, runtimeClass)
    ...
}

The ready branch only syncs cache states. BoundDatasetPhase is set solely by the mount flow
(syncDatasetMountsutils.UpdateMountStatus), which does not run on a normal reconcile, so the
phase is never restored.

What you expect to happen:

When runtimeReady is true again, the Dataset returns to Bound without recreating the CacheRuntime.

How to reproduce it

Needs only busybox.

apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntimeClass
metadata:
  name: repro-demo
fileSystemType: reprofs
topology:
  master:
    service: { headless: {} }
    template:
      spec:
        restartPolicy: Always
        containers:
          - name: master
            image: busybox:1.36
            command: ["sh", "-c", "while true; do nc -l -p 50051; done"]
            readinessProbe:
              tcpSocket: { port: 50051 }
              initialDelaySeconds: 3
            ports: [{ containerPort: 50051, name: rpc }]
  worker:
    service: { headless: {} }
    template:
      spec:
        restartPolicy: Always
        containers:
          - name: worker
            image: busybox:1.36
            command: ["sh", "-c", "while true; do nc -l -p 50052; done"]
            readinessProbe:
              tcpSocket: { port: 50052 }
              initialDelaySeconds: 3
            ports: [{ containerPort: 50052, name: data }]
---
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata: { name: repro-demo, namespace: default }
spec:
  placement: Shared
  accessModes: ["ReadWriteMany"]
  mounts: [{ name: r, mountPoint: "reprofs:///" }]
---
apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntime
metadata: { name: repro-demo, namespace: default }
spec:
  runtimeClassName: repro-demo
  master: { replicas: 1 }
  worker:
    replicas: 1
    tieredStore:
      levels:
        - emptyDir: { quota: 1Gi }
          high: "0.8"
          low: "0.5"

Apply it, wait for Bound, then:

kubectl delete pod repro-demo-worker-0 --wait=false

for i in $(seq 1 20); do
  printf "%3ds  dataset=%-9s worker.phase=%s\n" "$((i*6))" \
    "$(kubectl get dataset repro-demo -ojsonpath='{.status.phase}')" \
    "$(kubectl get cacheruntime repro-demo -ojsonpath='{.status.worker.phase}')"
  sleep 6
done
  6s  dataset=Bound     worker.phase=Ready
 12s  dataset=Bound     worker.phase=Ready
 18s  dataset=Bound     worker.phase=Ready
 24s  dataset=Bound     worker.phase=Ready
 30s  dataset=Failed    worker.phase=NotReady
 36s  dataset=Failed    worker.phase=Ready
 42s  dataset=Failed    worker.phase=Ready
 48s  dataset=Failed    worker.phase=Ready
 54s  dataset=Failed    worker.phase=Ready
 60s  dataset=Failed    worker.phase=Ready
 66s  dataset=Failed    worker.phase=Ready
 72s  dataset=Failed    worker.phase=Ready
 78s  dataset=Failed    worker.phase=Ready
 84s  dataset=Failed    worker.phase=Ready
 90s  dataset=Failed    worker.phase=Ready

Additional Information

Not specific to any cache system — it is on the generic ready path, so anything that briefly restarts
a worker (eviction, node reboot, rolling update) should hit it.

A separate bug also triggers this one: resources declared in a CacheRuntimeClass template get wiped,
and the resulting rollout strands the Dataset here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions