Terminate job if fail too much #206

typhoonzero · 2017-07-07T12:52:06Z

Yancey1989 · 2017-07-11T07:32:59Z

docker/paddle_k8s

@@ -25,6 +25,10 @@ check_trainer_ret() {
    echo "Program Abort" > /dev/termination-log
  fi
  echo "termination log wroted..."
+  FAILED_COUNT=$(python /root/k8s_tools.py fetch_pserver_ips)


The arguments should be fetch_job_fail_count

gongweibao · 2017-07-24T23:34:27Z

docker/k8s_tools.py

+        if j.metadata.name == PADDLE_JOB_NAME:
+            failed_count = j.status.failed
+            break
+    if not failed_count:


failed_count没有定义？

gongweibao

只有一个comment

terminate job if fail too much

ae832f2

Yancey1989 reviewed Jul 11, 2017

View reviewed changes

pineking mentioned this pull request Jul 14, 2017

submit task as job #208

Open

gongweibao reviewed Jul 24, 2017

View reviewed changes

Yancey1989 mentioned this pull request Aug 16, 2017

Max failed count #320

Merged

typhoonzero closed this Aug 16, 2017

typhoonzero deleted the fix_max_retry branch November 2, 2017 10:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terminate job if fail too much #206

Terminate job if fail too much #206

typhoonzero commented Jul 7, 2017 •

edited

Loading

Yancey1989 Jul 11, 2017

gongweibao Jul 24, 2017

gongweibao left a comment

Terminate job if fail too much #206

Terminate job if fail too much #206

Conversation

typhoonzero commented Jul 7, 2017 • edited Loading

Yancey1989 Jul 11, 2017

Choose a reason for hiding this comment

gongweibao Jul 24, 2017

Choose a reason for hiding this comment

gongweibao left a comment

Choose a reason for hiding this comment

typhoonzero commented Jul 7, 2017 •

edited

Loading