TFJob pods deleted on completion/failure impairing debugging

[cwbeitel]

It looks like there's been a change with the current version of TFJob from the earlier versions where replica pods are being deleted on job completion or failure. This makes dev/debugging difficult (i.e. needing to catch the logs as they happen but before the pod is deleted). Indeed solutions like StackDriver are a partial alternative to this (i.e. where some central service collects logs, avoiding the need to fetch them from pods directly) but still missing from that solution is the ability to kubectl describe pod .... Furthermore the norm established by Job's seems to be to retain Pod's after their completion or failure allowing the aforementioned.

[jlewi]

Duplicate of kubeflow/training-operator#536