What should Texera do when a computing unit dies?

[eugenegujing]

I noticed that a computing unit that is no longer alive is indistinguishable, in the UI, from one that is still booting. Both render as (Connecting).

Caption: CU 2 is healthy (green, selected, workflow running). CU 1 shows the gold dot and the literal (Connecting) suffix. That label is all the UI will ever say about it — at two seconds old and at two hours dead alike. The tooltip reads "Computing unit is starting up".

This isn't a rendering bug. Texera has no representation of an unhealthy computing unit anywhere in the stack, so there is nothing for the UI to render:

• ComputingUnitState (ComputingUnitState.scala:23) defines exactly Running and Pending.
• ComputingUnitHelpers (:86) resolves status as if (phase == "Running") Running else Pending.
• The frontend DTO type (workflow-computing-unit.ts:49) is status: "Running" | "Pending" — failure isn't expressible.
• local CUs are hard-coded to Running (:74-75) with no liveness check at all, so a dead local process shows green.

The frontend already has red/error branches (computeStatus(), getComputingUnitBadgeColor()). They're dead code, because only Running and Pending ever reach them. Someone anticipated failure states; the backend never grew them.

I've filed the mechanical part as #7669. Three questions I'd rather settle here than inside a PR review.

1. What should the status vocabulary be?

• Mirror Kubernetes — Running / Pending / Failed / Unknown / Terminating. Trivial to map, but it leaks the deployment substrate into a user-facing API and means nothing for local CUs.
• Texera-level abstraction — something like Starting / Ready / Unhealthy / Gone, named for what the user can do with the unit rather than for what k8s calls it. More mapping work, but it stays honest across local, k8s, and whatever comes next.

I lean toward the second, but it needs someone to define what "unhealthy" means for a local CU.

2. How much cluster detail should reach the end user?

"Your unit was OOM-killed" is actionable — raise the memory limit. "ImagePullBackOff: manifest unknown" is not something a data scientist can act on and arguably shouldn't be in their face.

There's also a sharing dimension: CU access can be shared (ComputingUnitAccessResource), so a failure reason reaches READ-privilege users who aren't the owner. Are we comfortable showing pod-level error text to everyone a unit is shared with, or should detail be owner-only with a generic "This unit is unavailable" for others?

3. Should a failed CU recover by itself, and is the current answer the one we want?

We already have a recovery policy: restart the container forever, never replace the pod. It falls out of KubernetesClient.createPod (:117-214), which builds a bare Pod — no owning Deployment or other controller — and never sets restartPolicy, so it defaults to Always. That splits by failure level:

• Container-level failure → retried forever, invisibly. Crash or OOM-kill, the kubelet restarts it in place, indefinitely. Texera isn't involved and doesn't report it.
• Pod-level loss → never recovered. Evicted, node drained, node dead: a bare pod has no controller to recreate it. Gone permanently.

There's no server-side reaper (no cleanup or reconcile job in computing-unit-managing-service), so a dead pod's DB row just persists. Keep delegating to k8s defaults, make it explicit with an owning controller and a restart cap, or move the policy into Texera?

The proposal in #7669, and whether there's a better one

1. Add terminal/unhealthy values to ComputingUnitState
2. Resolve status from real pod state and carry the reason as a statusReason string so the UI can eventually say why.

[eugenegujing]

@kunwp1 Do you have any thought?

[kunwp1]

I think we need to limit our scope and focus on showing user-friendly message that your CU has an issue with a reason so that user can avoid the same issue. There can be many reasons why CU dies and from my experience of managing the dknet deployment, two of the main reasons:

1. The user running workflow causes CU to be OOM-killed.
2. The user stores a huge amount of data to CU's local file system and causes out-of-storage issue.

So I suggest to focus on these two use cases.

To answer your questions,

1. I prefer mirroring Kubernetes because it's simple.
2. Show actionable error message to increase the memory size or reduce storing data to the local file system. And limit the scope to the owner of the CU.
3. No need to automatically recover the CU but instead give users a chance to create with a different configuration.

[chenlica]

@yrenat Do you want to chime in as you are looking at a related issue? If so, please add the corresponding URL.

[yrenat]

Hi @eugenegujing, I agree with @kunwp1 on the possible directions to address your questions, and I would like to add some of my thoughts to your observation.

I think you are right that there is no liveness check for a CU, but there does exist such checks for workflows. This PR aims to clean garbage workflow execution results after a configured TTL. It does not directly solve your problem, but I think it shows the possibility to monitor CU status from a workflow perspective. That may be easier to work with because you are technically utilizing something pre-built.

Anyway, this PR introduces the last_active variable which records the last active time of each workflow inside the CUs. This variable might be helpful for your question because

1. a CU can be considered avtive if there are more than one workflow being active
2. a CU can be considered dead if all the workflows have been idle for a long time
3. ...

Thanks @chenlica for letting me join this discussion. Hope that helps.

[eugenegujing]

Thanks @kunwp1 and @yrenat. I think this settles all three questions. Here's the direction I'll take in #7669, so the PR review can point back to this thread:

1. Status vocabulary: mirror Kubernetes. I'll add Failed / Unknown (and Terminating) to ComputingUnitState and drop the idea of a Texera-level abstraction.
2. Failure detail: owner-only, actionable. Per @kunwp1's two production scenarios, the statusReason will be a human-actionable message rather than raw k8s output — e.g. OOMKilled → "The unit ran out of memory; consider recreating it with a higher memory limit", disk pressure/eviction → "The unit ran out of local storage; consider storing less data on the local file system". Shared (READ) users will only see a generic "This unit is unavailable".
3. No auto-recovery. Failed stays failed; the user recreates with a different configuration. I verified the existing terminate flow already handles this: the delete action is available to the owner regardless of status, and KubernetesClient.deletePod is a no-op when the pod is already gone — so what's missing is only the signal telling the user the unit is dead, which is exactly what this change adds.

@yrenat Thanks for the pointer to #2123, and I really like the idea of inferring liveness with zero new infrastructure. But I'm afraid it might not work as the primary signal for this particular issue, if I'm reading the code right, last_update_time is stamped on state transitions rather than as a heartbeat, so a long-running healthy execution could look the same as a dead CU, and a CU stuck at "(Connecting)" usually has no executions yet, so the table may have no signal for the exact case this issue targets. It also doesn't seem able to tell apart the failure causes (OOM vs. out-of-storage) we want to surface, which I believe only exist in pod state.

I will start working on this for a PR in our current design.