PS failed but tfjob status is running

[deepanshg]

Tfjob consists of 1 PS and 2 Workers. PS is failed, workers are running and status of tfjob is running, whereas it should be failed according to PR#690

Status:

  Conditions:
    Last Transition Time:  2018-12-02T12:52:38Z
    Last Update Time:      2018-12-02T12:52:38Z
    Message:               TFJob dtf-103553-223 is running.
    Reason:                TFJobRunning
    Status:                True
    Type:                  Running
  Start Time:              2018-12-02T13:06:55Z
  Tf Replica Statuses:
    PS:
      Failed:  1
    Worker:
      Active:  2

The status remains same even after hours.

Some parts of Yaml:

Namespace:    gpuexptazuregpu
Labels:       app=dtf-103553-223
Annotations:  <none>
API Version:  kubeflow.org/v1alpha2
Kind:         TFJob
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-12-02T12:52:35Z
  Generation:          0
  Resource Version:    64383446
  Self Link:           /apis/kubeflow.org/v1alpha2/namespaces/gpuexptazuregpu/tfjobs/dtf-103553-223
  UID:                 2436acaf-f631-11e8-8519-000d3a02fab7
Spec:
  Tf Replica Specs:
    PS:
      Replicas:        1
      Restart Policy:  Never
      Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec: ...
    Worker:
      Replicas:        2
      Restart Policy:  Never
      Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec: ...

tf-opertor-image: gcr.io/kubeflow-images-public/tf_operator:v0.2.0
kubeflow version: v0.2.4

[richardsliu]

This should be fixed in v0.3.0 and later.