Job never becomes ready when re-attaching to exiting env when pods for the job are deleted

[dnephin]

Background

I believe this is the same issue as described in #5822. That issue was closed over a year ago, so creating a new one.

We have a system that pre-creates tilt environments for tilt ci, so when tilt ci runs it resumes from an existing environment. We've noticed that occasionally those resumes will fail with a timeout. Using the new output from #6262 we noticed that when it happens it's always a job that never goes ready. Looking at logs we notice that these jobs had deleted pods.

#5859 added special handling for "completed jobs without pods" to kubernetesapply/reconciler.go, but that does not appear to work for processExitCondition.

It appears that session/conv.go:Reconciler.k8sRuntimeTarget sees a job with status == v1alpha1.RuntimeStatusOK, but there are no pods, so v1.PodSucceeded == phase is false, and the resource is marked as Active.Ready = true. In processExitCondition

tilt/internal/controllers/core/session/status.go

Line 101 in 38238ec

} else if res.State.Active != nil && (!res.State.Active.Ready || res.Type == v1alpha1.TargetTypeJob) {

means that the ready == True is ignored for jobs, so they never appear ready when the pods are gone.

Expected Behavior

Re-attaching to an environment with tilt ci recognizes a job is completed, even when the pods for that job are deleted.

Current Behavior

Today it will timeout with something like:

Error: Timeout after 30m0s: 1 resources not ready (db-init:runtime)

Steps to Reproduce

1. Using the example Tiltfile from job never becomes ready when re-attaching to existing env #5822
2. tilt ci and wait for it to complete
3. kubectl delete the pod/db-init- pod
4. tilt ci --timeout 1m - this will timeout instead of going ready

Implementation Questions

I think we understand the problem well enough, but I'm not entirely sure how to fix it yet.

I think in k8sRuntimeTarget we can handle when no pod is found from krs.MostRecentPod, and produce a state of Terminated. It seems like the condition is available at that time.