What is your environment(Kubernetes version, Fluid version, etc.)
Kubernetes v1.30.0 (kind v0.23.0), Fluid chart fluid-1.1.0 built from 89af5dc9, installed with
--set runtime.cacheruntime.enabled=true.
Describe the bug
Any brief worker outage sets the Dataset to Failed, and it never goes back to Bound. Deleting a
worker pod is enough to reproduce it.
The end state contradicts itself:
Pod: repro-demo-worker-0 1/1 Running
CacheRuntime: status.worker.phase=Ready, WorkersReady=True
Dataset: status.phase=Failed, "The ddc runtime is not ready."
lastUpdateTime freezes at the moment of the outage. Forcing a reconcile does not help; only
deleting and recreating the CacheRuntime restores Bound.
The transition is one-way in pkg/ddc/cache/engine/sync.go:87-99:
if !runtimeReady {
err = e.UpdateDatasetStatus(datav1alpha1.FailedDatasetPhase, runtime, runtimeClass)
...
} else if permitSyncEngineStatus {
err = e.syncDatasetCacheStates(ctx, runtime, runtimeClass)
...
}
The ready branch only syncs cache states. BoundDatasetPhase is set solely by the mount flow
(syncDatasetMounts → utils.UpdateMountStatus), which does not run on a normal reconcile, so the
phase is never restored.
What you expect to happen:
When runtimeReady is true again, the Dataset returns to Bound without recreating the CacheRuntime.
How to reproduce it
Needs only busybox.
apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntimeClass
metadata:
name: repro-demo
fileSystemType: reprofs
topology:
master:
service: { headless: {} }
template:
spec:
restartPolicy: Always
containers:
- name: master
image: busybox:1.36
command: ["sh", "-c", "while true; do nc -l -p 50051; done"]
readinessProbe:
tcpSocket: { port: 50051 }
initialDelaySeconds: 3
ports: [{ containerPort: 50051, name: rpc }]
worker:
service: { headless: {} }
template:
spec:
restartPolicy: Always
containers:
- name: worker
image: busybox:1.36
command: ["sh", "-c", "while true; do nc -l -p 50052; done"]
readinessProbe:
tcpSocket: { port: 50052 }
initialDelaySeconds: 3
ports: [{ containerPort: 50052, name: data }]
---
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata: { name: repro-demo, namespace: default }
spec:
placement: Shared
accessModes: ["ReadWriteMany"]
mounts: [{ name: r, mountPoint: "reprofs:///" }]
---
apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntime
metadata: { name: repro-demo, namespace: default }
spec:
runtimeClassName: repro-demo
master: { replicas: 1 }
worker:
replicas: 1
tieredStore:
levels:
- emptyDir: { quota: 1Gi }
high: "0.8"
low: "0.5"
Apply it, wait for Bound, then:
kubectl delete pod repro-demo-worker-0 --wait=false
for i in $(seq 1 20); do
printf "%3ds dataset=%-9s worker.phase=%s\n" "$((i*6))" \
"$(kubectl get dataset repro-demo -ojsonpath='{.status.phase}')" \
"$(kubectl get cacheruntime repro-demo -ojsonpath='{.status.worker.phase}')"
sleep 6
done
6s dataset=Bound worker.phase=Ready
12s dataset=Bound worker.phase=Ready
18s dataset=Bound worker.phase=Ready
24s dataset=Bound worker.phase=Ready
30s dataset=Failed worker.phase=NotReady
36s dataset=Failed worker.phase=Ready
42s dataset=Failed worker.phase=Ready
48s dataset=Failed worker.phase=Ready
54s dataset=Failed worker.phase=Ready
60s dataset=Failed worker.phase=Ready
66s dataset=Failed worker.phase=Ready
72s dataset=Failed worker.phase=Ready
78s dataset=Failed worker.phase=Ready
84s dataset=Failed worker.phase=Ready
90s dataset=Failed worker.phase=Ready
Additional Information
Not specific to any cache system — it is on the generic ready path, so anything that briefly restarts
a worker (eviction, node reboot, rolling update) should hit it.
A separate bug also triggers this one: resources declared in a CacheRuntimeClass template get wiped,
and the resulting rollout strands the Dataset here.
What is your environment(Kubernetes version, Fluid version, etc.)
Kubernetes v1.30.0 (kind v0.23.0), Fluid chart
fluid-1.1.0built from89af5dc9, installed with--set runtime.cacheruntime.enabled=true.Describe the bug
Any brief worker outage sets the Dataset to
Failed, and it never goes back toBound. Deleting aworker pod is enough to reproduce it.
The end state contradicts itself:
lastUpdateTimefreezes at the moment of the outage. Forcing a reconcile does not help; onlydeleting and recreating the CacheRuntime restores
Bound.The transition is one-way in
pkg/ddc/cache/engine/sync.go:87-99:The ready branch only syncs cache states.
BoundDatasetPhaseis set solely by the mount flow(
syncDatasetMounts→utils.UpdateMountStatus), which does not run on a normal reconcile, so thephase is never restored.
What you expect to happen:
When
runtimeReadyis true again, the Dataset returns toBoundwithout recreating the CacheRuntime.How to reproduce it
Needs only busybox.
Apply it, wait for
Bound, then:Additional Information
Not specific to any cache system — it is on the generic ready path, so anything that briefly restarts
a worker (eviction, node reboot, rolling update) should hit it.
A separate bug also triggers this one:
resourcesdeclared in a CacheRuntimeClass template get wiped,and the resulting rollout strands the Dataset here.