TFJob controller cannot terminate job

[bleachzk]

I test kubeflow/tf-controller-examples/tf-cnn/tf_job_gpu.yaml by running kubectl:

kubectl apply -f tf_job_gpu.yaml
kubectl get job
NAME DESIRED SUCCESSFUL AGE
inception-171202-163257-gpu-1-ps-272u-0 1 0 7m
inception-171202-163257-gpu-1-worker-272u-0 1 0 7m

The TFJob controller cannot terminate job when the WORKER is done based on the TerminationPolicy。

WORKER log：

logs-from-tensorflow-in-inception-171202-163257-gpu-1-worker-272u-0-fdx5x.txt

[gaocegege]

@bleachzk Thanks for your issue, and I am not sure if we support TerminationPolicy now, maybe jlewi@ could give us more info

[jlewi]

@bleachzk Can you please provide the spec/status for your TFJob? Can you also clarify what your expected behavior is and what the observed behavior is?

Its possible you're hitting kubeflow/training-operator#128

[bleachzk]

@jlewi
Worker ran successfully，job should be terminated，but it is still running...

Pod status：

worker logs：

Job status：

[jlewi]

That's kubeflow/training-operator#128

We originally let the jobs continue to run until the TFJob is deleted to make the logs accessible after the job terminated.

We are working on fixing that.

[jlewi]

Duplicate of kubeflow/training-operator#128