Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backoff policy and failed pod limit #583

Merged
merged 7 commits into from Aug 28, 2017

Conversation

@soltysh
Copy link
Contributor

soltysh commented Apr 26, 2017

This update addresses problems raised in kubernetes/kubernetes#30243 and kubernetes/kubernetes#43964.

@erictune I've mentioned this to you during last sig-apps, ptal
@kubernetes/sig-apps-feature-requests ptal
@lukasheinrich @nickschuch @Yancey1989 @sstarcher @maximz fyi since you all were interested in this

soltysh added some commits Apr 26, 2017

limits, described above, are set.

All of the above fields will be optional and will apply no matter which `restartPolicy`
is set on a `PodTemplate`.

This comment has been minimized.

@soltysh

soltysh Apr 26, 2017

Author Contributor

I was struggling with this idea, but I think we have necessary information when failure happens in both cases. When Never we count failures ourselves, with OnFailure we can rely on no of restarts. Alternatively, I was thinking to address that only for Never, but I want to hear from others.

This comment has been minimized.

@diegodelemos

diegodelemos Apr 26, 2017

For our use case it would be useful to rely on OnFailure policy and then counting number of restarts (this is in fact what we are doing from outside, watching status over stream connections). Since we aim to submit large amounts of jobs, it would be better to have only one container retrying rather that having a new pod per failure spawned by the job controller. Unless you think this is not going to end up creating performance issues.

This comment has been minimized.

@seh

seh Apr 26, 2017

I interpret the Job restart limit to be predicated on the number of pod failures. If a pod has a restart policy of OnFailure, by my reading of the documentation, a pod in which its sole container terminates in failure means the whole pod transitions to the "Failed" state, perhaps only momentarily, even if the container gets restarted and the pod transitions back to the "Running" state. However, the "Example states" section at the end of the same document never mentions such a transition.

That means that to count the number of attempts for a pod template with restart policy of Never, we'd have to count the number of pods created for the Job. For a policy of OnFailure, that count is the number of pods created (assuming it's possible for a Job to lose a pod and have to schedule a replacement) plus the number of container restarts. (Each pod creation implies each container starting once.)

However, for a pod with multiple containers, we're then treating each failure of a container to imply one failure of the containing pod. That's confusing to reason about. The alternative is to say that a pod with a restart policy of OnFailure only participates in this accounting when the entire pod fails, such as due to the disk failing or the hosting node is removed from the cluster.

This comment has been minimized.

@soltysh

soltysh Apr 26, 2017

Author Contributor

The reason I went with both actually is that I don't want to connect pod restart policy with job backoff policy, in the first place. Additionally, if pod's restart policy is never the kubelet will keep on creating new pods along the old ones (that's the job contract), which in turns the FailedPodsLimit can address.
Currently Never is the only one that returns the pod failure back to job controller which creates a new pod to satisfy completions. With OnFailure we're passing that responsibility to kubelet. After talking to @kargakis I'm starting to think we should've restricted jobs to only use Never, but it's too late now because it would break to many folks. That's why I'd go with the backoff policy not being connected with pod's restart policy to limit the amount of confusion we already caused allowing both restart policies. Does that make sense?

This comment has been minimized.

@seh

seh Apr 26, 2017

It does make sense to me. It's important that users understand, though, that with OnFailure a container failure counts against the Job retry accounting.

If a Job uses pods with three containers, and two of the containers fail "at the same time" and get restarted by the kubelet, does that count as two retries of the Job?

This comment has been minimized.

@soltysh

soltysh Apr 27, 2017

Author Contributor

I'll add that explanation, in that case.

@nickschuch
Copy link

nickschuch left a comment

All this makes perfect sense to me. Would you like me to start demonstrating this in my PR #41451?

// Optional number of retries, before marking this job failed.
BackoffLimit *int
// Optional time (in seconds), how log a job should be retried before marking it failed.

This comment has been minimized.

@nickschuch
By design, Jobs do not have any notion of failure, other than Pod's `restartPolicy`
which is mistakenly taken as Job's restart policy ([#30243](https://github.com/kubernetes/kubernetes/issues/30243),
[#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are
situation where one wants to fail a Job after some amount of retries over certain

This comment has been minimized.

@seh

seh Apr 26, 2017

  • Did you mean "over a certain period" or "over multiple periods?"

This comment has been minimized.

@soltysh

soltysh Apr 26, 2017

Author Contributor

a certain period, will fix.

[#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are
situation where one wants to fail a Job after some amount of retries over certain
period of time, due to a logical error in configuration etc. To do so we are going
following fields will be introduced, which will control the exponential backoff

This comment has been minimized.

@seh

seh Apr 26, 2017

  • Is there a sentence fragment missing in between "we are going" and "following fields?"
    Maybe, "we are going to introduce the following fields?"

This comment has been minimized.

@soltysh

soltysh Apr 26, 2017

Author Contributor

Yup ;)

situation where one wants to fail a Job after some amount of retries over certain
period of time, due to a logical error in configuration etc. To do so we are going
following fields will be introduced, which will control the exponential backoff
when retrying Job: number of retries and time to retry. The two fields will allow

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/retrying Job/retrying a Job/
    or maybe s/retying Job/retrying Jobs/
period of time, due to a logical error in configuration etc. To do so we are going
following fields will be introduced, which will control the exponential backoff
when retrying Job: number of retries and time to retry. The two fields will allow
creating a fine grain control over the backoff policy, limiting the number of retries

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/a fine grain/fine-grained/
following fields will be introduced, which will control the exponential backoff
when retrying Job: number of retries and time to retry. The two fields will allow
creating a fine grain control over the backoff policy, limiting the number of retries
over specified period of time. In the case when only one of them is specified

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/over specified/over a specified/

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/is specified/is specified,/
// before the system tries to terminate it; value must be positive integer
ActiveDeadlineSeconds *int
// Optional number of retries, before marking this job failed.

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s /retries,/retries/
// Optional number of retries, before marking this job failed.
BackoffLimit *int
// Optional time (in seconds), how log a job should be retried before marking it failed.

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/), how/) specifying how/
  • Did you consider using type *uint instead?
    That could encourage unintentional overflow when used with time.
@@ -26,6 +27,31 @@ Jobs are needed for executing multi-pod computation to completion; a good exampl
here would be the ability to implement any type of batch oriented tasks.


## Backoff policy and failed pod limit

By design, Jobs do not have any notion of failure, other than Pod's `restartPolicy`

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/than Pod's/than a pod's/
@@ -18,6 +18,7 @@ Several existing issues and PRs were already created regarding that particular s
1. Be able to get the job status.
1. Be able to specify the number of instances performing a job at any one time.
1. Be able to specify the number of successfully finished instances required to finish a job.
1. Be able to specify backoff policy, when job is continuously failing.

This comment has been minimized.

@seh

seh Apr 26, 2017

  • s/specify backoff/specify a backoff/
limits, described above, are set.

All of the above fields will be optional and will apply no matter which `restartPolicy`
is set on a `PodTemplate`.

This comment has been minimized.

@seh

seh Apr 26, 2017

I interpret the Job restart limit to be predicated on the number of pod failures. If a pod has a restart policy of OnFailure, by my reading of the documentation, a pod in which its sole container terminates in failure means the whole pod transitions to the "Failed" state, perhaps only momentarily, even if the container gets restarted and the pod transitions back to the "Running" state. However, the "Example states" section at the end of the same document never mentions such a transition.

That means that to count the number of attempts for a pod template with restart policy of Never, we'd have to count the number of pods created for the Job. For a policy of OnFailure, that count is the number of pods created (assuming it's possible for a Job to lose a pod and have to schedule a replacement) plus the number of container restarts. (Each pod creation implies each container starting once.)

However, for a pod with multiple containers, we're then treating each failure of a container to imply one failure of the containing pod. That's confusing to reason about. The alternative is to say that a pod with a restart policy of OnFailure only participates in this accounting when the entire pod fails, such as due to the disk failing or the hosting node is removed from the cluster.

@soltysh

This comment has been minimized.

Copy link
Contributor Author

soltysh commented Apr 26, 2017

All this makes perfect sense to me. Would you like me to start demonstrating this in my PR #41451?

@nickschuch let's wait for this to get to a stable shape (more or less) and then you can go ahead an open a fresh one for it. I'll close the old one.

@soltysh soltysh referenced this pull request Apr 26, 2017

Closed

Job: Failure Threshold #41451

@soltysh

This comment has been minimized.

Copy link
Contributor Author

soltysh commented Apr 27, 2017

I've addressed first batch of comments.

@seh

seh approved these changes Apr 27, 2017

Copy link

seh left a comment

I cited one nit that we could fix and asked for clarification, but I won't hold up a merge for them.

is set on a `PodTemplate`. The only difference applies to how failures are counted.
For restart policy `Never` we count actual pod failures (reflected in `.status.failed`
field). With restart policy `OnFailure` we look at pod restarts (calculated from
`.status.containerStatuses[*].restartCount`).

This comment has been minimized.

@seh
too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243))
to introduce following fields will be introduced, which will control the exponential
backoff when retrying a Job: number of retries and time to retry. The two fields
will allow creating a fine-grained control over the backoff policy, limiting the

This comment has been minimized.

@seh

seh Apr 27, 2017

  • Reading this again, I think this would read better as

    will allow fine-grained control over

BackoffLimit *int
// Optional time (in seconds), how log a job should be retried before marking it failed.
// Optional time (in seconds) specifying how long a job should be retried before marking it failed.

This comment has been minimized.

@seh

seh Apr 27, 2017

  • Obviously this duration shouldn't be negative (in which case I assume we'd ignore it, or reject it), but what about zero?
    Since it's a pointer, the pointer being nil means that it's not set, but what if it's present but zero? Is that the same as it not being set, or is that a valid duration meaning that we won't wait any amount of time for a job complete (that it must complete immediately)?

This comment has been minimized.

@soltysh

soltysh May 2, 2017

Author Contributor

We usually require those being >= 0, it'll be checked in validation.

@agonzalezro
Copy link

agonzalezro left a comment

Nice approach, I think those changes are going to be really useful!

which is mistakenly taken as Job's restart policy ([#30243](https://github.com/kubernetes/kubernetes/issues/30243),
[#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are
situation where one wants to fail a Job after some amount of retries over a certain
period of time, due to a logical error in configuration etc. To do so we are going

This comment has been minimized.

@agonzalezro

agonzalezro Apr 28, 2017

  • I think you are repeating "introduce{,d}". It should be something like: "To do so we are going to introduce the following fields"
@@ -79,8 +109,21 @@ type JobSpec struct {
// job should be run with. Defaults to 1.
Completions *int
// Optional duration in seconds relative to the startTime that the job may be active
// before the system tries to terminate it; value must be a positive integer.
ActiveDeadlineSeconds *int

This comment has been minimized.

@agonzalezro

agonzalezro Apr 28, 2017

  • Can uint be used, also in the other fields?

This comment has been minimized.

@soltysh

soltysh May 2, 2017

Author Contributor

We used ints, for consistency and then do appropriate validation so this is non-negative and bigger than 0.

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

*int32

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

*int32

Actually *int64

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

Can you also update the rest of the fields to their canonical types?

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

I think this is the deadline for the entire job to complete, regardless of whether there are restarts, right? Document that.

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

Correct.

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

It clearly states it's relative to job's startTime.

@@ -79,8 +109,21 @@ type JobSpec struct {
// job should be run with. Defaults to 1.
Completions *int
// Optional duration in seconds relative to the startTime that the job may be active

This comment has been minimized.

@agonzalezro

agonzalezro Apr 28, 2017

  • I think the comments for exported fields should start with the name of the field: "ActiveDeadlineSeconds is an optional..."

This comment has been minimized.

@soltysh

soltysh May 2, 2017

Author Contributor

Not anymore, the def will have those in place.

@soltysh

This comment has been minimized.

Copy link
Contributor Author

soltysh commented May 2, 2017

@erictune / @janetkuo any objections for merging this in?

@kargakis kargakis referenced this pull request May 5, 2017

Closed

Failure policy for Jobs #298

BackoffLimit *int
// Optional time (in seconds) specifying how long a job should be retried before marking it failed.
BackoffDeadlineSeconds *int

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

ProgressDeadlineSeconds to match the field in Deployments? Seems like it's the same concept although in Deployment we merely add a condition in the DeploymentStatus.

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

How would a user typically use ActiveDeadlineSeconds with BackoffDeadlineSeconds?

This comment has been minimized.

@sdminonne

sdminonne Jun 12, 2017

Contributor

@erictune @soltysh if I understand correctly the only check system may run is: ActiveDeadlineSeconds must be strictly bigger than BackoffDeadlineSeconds, missing something?

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

@kargakis this is different than ProgressDeadlineSecods, in deployments world PDS is how long you are waiting for the job to make progress and than fail. With Job BackoffDeadlineSeconds applies at any point in time. This means that if a Job expects to reach 5 completions, 4 of them can succeed and only the 5th Pod might be having troubles and BDS should still apply. That's why I'd rather not to use the same name, it's intentional.

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

@erictune @sdminonne ADS is the overall time for a Job (max life-time). Whereas BDS is counted from the moment the first failure happens. But yes, usually BDS will be smaller than ADS, otherwise you'll never hit BDS in case of a failure.

This comment has been minimized.

@kargakis

kargakis Jun 22, 2017

Member

Well, PDS is not that different. We start counting from the last time we saw progress and once the deadline is exceeded we merely switch the Progressing condition to False. "Failing the deployment" should be something that a higher-level orchestrator decides based on the condition.

BackoffDeadlineSeconds *int
// Optional number of failed pods to retain.
FailedPodsLimit *int

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

When are pods left around? I guess when pods have restartPolicy=Never? It feels weird to add a feature that is not related directly to the failurePolicy and also is useful for a specific case in Jobs. Can we leave it out of this proposal?

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

This is for debugging purposes. Leaving pods should allow you to figure out what went wrong. Without it and with restart policy Never you might end up with many failed pods, if you set the limits high. That was one of the original requests wrt to the backoff policy, so I think it's crucial to address it here.

to introduce following fields, which will control the exponential backoff when
retrying a Job: number of retries and time to retry. The two fields will allow
fine-grained control over the backoff policy, limiting the number of retries over
a specified period of time. If only one of the two fields is supplied, an exponential

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

It's not clear what happens if both fields are specified. Is only the number of retries limited or both of the fields limit each other?

This comment has been minimized.

@kargakis

kargakis May 5, 2017

Member

Ok, it's clearer after rereading the bullet points below.

retrying a Job: number of retries and time to retry. The two fields will allow
fine-grained control over the backoff policy, limiting the number of retries over
a specified period of time. If only one of the two fields is supplied, an exponential
backoff with an intervening duration of ten seconds and a factor of two will be

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

I don't think we should promise our users exponential backoff -- we should give ourselves the freedom to do other things in the future. Just say that we will retry at a time that the system choses, and that in any case the number of retries won't exceed the number specified by the user.

too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)),
we are going to introduce a field which will allow specifying the maximum number
of failed pods to keep around. This number will also take effect if none of the
limits described above are set.

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

Is the intent that users will typically set this to 0 unless they expect to need to debug? Or do we typically want to keep around at least 1 failed pod so the user can get the logs?

not save all pods without

This comment has been minimized.

@sdminonne

sdminonne Jun 12, 2017

Contributor

I vote to have this defaulted to 1

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

1 seems reasonable default.

is set on a `PodTemplate`. The only difference applies to how failures are counted.
For restart policy `Never` we count actual pod failures (reflected in `.status.failed`
field). With restart policy `OnFailure` we look at pod restarts (calculated from
`.status.containerStatuses[*].restartCount`).

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

This cannot be synchronous, though, so the limit has to be approximate. Document that it is precise with Never, and approximate with OnFailure?

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

Yup, makes sense.

@@ -79,8 +109,21 @@ type JobSpec struct {
// job should be run with. Defaults to 1.
Completions *int
// Optional duration in seconds relative to the startTime that the job may be active
// before the system tries to terminate it; value must be a positive integer.
ActiveDeadlineSeconds *int

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

I think this is the deadline for the entire job to complete, regardless of whether there are restarts, right? Document that.

ActiveDeadlineSeconds *int
// Optional number of retries before marking this job failed.
BackoffLimit *int

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

Why BackoffLimit instead of RetryLimit or TriesLimit?

This comment has been minimized.

@soltysh

soltysh Jun 22, 2017

Author Contributor

I went with Backoff as a prefix for every field related to that, since we're talking about Backoff policy.

BackoffLimit *int
// Optional time (in seconds) specifying how long a job should be retried before marking it failed.
BackoffDeadlineSeconds *int

This comment has been minimized.

@erictune

erictune May 5, 2017

Member

How would a user typically use ActiveDeadlineSeconds with BackoffDeadlineSeconds?

[#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are
situation where one wants to fail a Job after some amount of retries over a certain
period of time, due to a logical error in configuration etc. To do so we are going
to introduce following fields will be introduced, which will control the exponential

This comment has been minimized.

@sdminonne

sdminonne Jun 12, 2017

Contributor

will be introduced

@soltysh

This comment has been minimized.

Copy link
Contributor Author

soltysh commented Jun 22, 2017

@erictune @kargakis I've addressed your comments, ptal once more. @sdminonne mind pointing your guy who will be implementing this to have a look at it, as well.

@sdminonne

This comment has been minimized.

Copy link
Contributor

sdminonne commented Jun 22, 2017

@soltysh Cedric's GH account is @clamoriniere1A

@sdminonne

This comment has been minimized.

Copy link
Contributor

sdminonne commented Jun 22, 2017

@clamoriniere1A mind have a look at this and eventually comment?

@clamoriniere1A

This comment has been minimized.

Copy link

clamoriniere1A commented Jun 23, 2017

@sdminonne sure, I have started to look at it.

@clamoriniere1A

This comment has been minimized.

Copy link

clamoriniere1A commented Jun 27, 2017

I started to work on the Job Backoff policy and failed pod limit: kubernetes/kubernetes#48075

@k8s-ci-robot k8s-ci-robot added the lgtm label Aug 28, 2017

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Aug 28, 2017

Automatic merge from submit-queue

@k8s-github-robot k8s-github-robot merged commit 0672a5f into kubernetes:master Aug 28, 2017

2 checks passed

Submit Queue Queued to run github e2e tests a second time.
cla/linuxfoundation soltysh authorized
Details

@soltysh soltysh deleted the soltysh:job_failure_policy branch Aug 29, 2017

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Aug 29, 2017

Job failure policy support in JobController
Fix: kubernetes/community#583

Job failure policy integration in JobController. From the
JobSpec.BackoffLimit and JobSpec.BackoffSeconds the JobController will
define the backoff duration between Job retry.

Currently the number of retry for each job is store in the
DynamicBackoff struct that is local to the JobController. It means that
if the JobController restarts the number of retries will be lost and the
backoff duration will be reset to 0.

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Aug 29, 2017

Job failure policy support in JobController
Fix: kubernetes/community#583

Job failure policy integration in JobController. From the
JobSpec.BackoffLimit and JobSpec.BackoffSeconds the JobController will
define the backoff duration between Job retry.

Currently the number of retry for each job is store in the
DynamicBackoff struct that is local to the JobController. It means that
if the JobController restarts the number of retries will be lost and the
backoff duration will be reset to 0.

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Aug 30, 2017

Job failure policy support in JobController
Fix: kubernetes/community#583

Job failure policy integration in JobController. From the
JobSpec.BackoffLimit and JobSpec.BackoffSeconds the JobController will
define the backoff duration between Job retry.

Currently the number of retry for each job is store in the
DynamicBackoff struct that is local to the JobController. It means that
if the JobController restarts the number of retries will be lost and the
backoff duration will be reset to 0.

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Aug 30, 2017

flowcontrol.Backoff add init Duration and retry nb
For the "Backoff policy and failed pod limit" implementation
(kubernetes/community#583), we need to retrieve
the number of backoff retry and also be able to set a different
initial backoff duration for each entry.

This Pull request adds two new methods to ```Backoff```:
- ```NextWithInitDuration```: as ```Next``` the purpose of this method
is to increment the backoff delay exponentially. In addition, you need to
provide the initial delay duration if the entry is not already present.

- ```GetWithRetryNumber```: returns the backoff delay associated to an
entry and also the retry number corresponding to the number of time that
the ```Next|GetWithRetryNumber``` has been call.

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Aug 30, 2017

flowcontrol.Backoff add init Duration and retry nb
For the "Backoff policy and failed pod limit" implementation
(kubernetes/community#583), we need to retrieve
the number of backoff retry and also be able to set a different
initial backoff duration for each entry.

This Pull request adds two new methods to ```Backoff```:
- ```NextWithInitDuration```: as ```Next``` the purpose of this method
is to increment the backoff delay exponentially. In addition, you need to
provide the initial delay duration if the entry is not already present.

- ```GetWithRetryNumber```: returns the backoff delay associated to an
entry and also the retry number corresponding to the number of time that
the ```Next|GetWithRetryNumber``` has been call.

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Aug 31, 2017

Job failure policy support in JobController
Fix: kubernetes/community#583

Job failure policy integration in JobController. From the
JobSpec.BackoffLimit and JobSpec.BackoffSeconds the JobController will
define the backoff duration between Job retry.

Currently the number of retry for each job is store in the
DynamicBackoff struct that is local to the JobController. It means that
if the JobController restarts the number of retries will be lost and the
backoff duration will be reset to 0.

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Sep 1, 2017

update API v1 Job object
Add new fields in api v1.JobSpec object for backoff policy
- BackoffLimit
- FailedPodsLimit

fixes: kubernetes/community#583

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Sep 1, 2017

update API v1 Job object
Add new fields in api v1.JobSpec object for backoff policy
- BackoffLimit
- FailedPodsLimit

fixes: kubernetes/community#583

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Sep 1, 2017

update API v1 Job object
Add new fields in api v1.JobSpec object for backoff policy
- BackoffLimit
- FailedPodsLimit

fixes: kubernetes/community#583

clamoriniere1A added a commit to clamoriniere1A/kubernetes that referenced this pull request Sep 1, 2017

update API v1 Job object
Add new fields in api v1.JobSpec object for backoff policy
- BackoffLimit
- FailedPodsLimit

fixes: kubernetes/community#583

k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this pull request Sep 3, 2017

Kubernetes Submit Queue
Merge pull request #48075 from clamoriniere1A/feature/job_failure_policy
Automatic merge from submit-queue (batch tested with PRs 51335, 51364, 51130, 48075, 50920)

[API] Feature/job failure policy

**What this PR does / why we need it**: Implements the Backoff policy and failed pod limit defined in kubernetes/community#583

**Which issue this PR fixes**: 
fixes #27997, fixes #30243

**Special notes for your reviewer**:
This is a WIP PR, I updated the api batchv1.JobSpec in order to prepare the backoff policy implementation in the JobController.

**Release note**:
```release-note
Add backoff policy and failed pod limit for a job
```

k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this pull request Sep 3, 2017

Kubernetes Submit Queue
Merge pull request #51153 from clamoriniere1A/feature/job_failure_pol…
…icy_controller

Automatic merge from submit-queue

Job failure policy controller support

**What this PR does / why we need it**:
Start implementing the support of the "Backoff policy and failed pod limit" in the ```JobController```  defined in kubernetes/community#583.
This PR depends on a previous PR #48075  that updates the K8s API types.

TODO: 
* [X] Implement ```JobSpec.BackoffLimit``` support
* [x] Rebase when #48075 has been merged.
* [X] Implement end2end tests



implements kubernetes/community#583

**Special notes for your reviewer**:

**Release note**:
```release-note
Add backoff policy and failed pod limit for a job
```
@paulwalker

This comment has been minimized.

Copy link

paulwalker commented Sep 7, 2017

This recent development looks good, but If I may ask, what is the current best config/pattern for a try-once job in 1.7.3? TIA

@soltysh

This comment has been minimized.

Copy link
Contributor Author

soltysh commented Sep 8, 2017

@paulwalker there's nothing unfortunately. You'd need to implement similar mechanism inside of your job manually, I'm afraid 😞

@kargakis

This comment has been minimized.

Copy link
Member

kargakis commented Sep 8, 2017

Or run a pod with RestartPolicy=Never.

@chirag04

This comment has been minimized.

Copy link

chirag04 commented Sep 20, 2017

@erictune can you elaborate on Job terminated after (approximately) first Pod Failure -- must use with Never to get close to Run-Once behavior?

why is that a close to Run-once behavior?

@erictune

This comment has been minimized.

Copy link
Member

erictune commented Sep 20, 2017

@chirag04

This comment has been minimized.

Copy link

chirag04 commented Sep 20, 2017

@erictune thanks for the explanation and that makes perfect sense. 👍

This will result in the following retry sequence: 10s, 20s, 40s, 1m20s, 2m40s,
5m20s. After which the job will be considered failed.

Additionally, to help debug the issue with a Job, and limit the impact of having

This comment has been minimized.

@chenchun

chenchun Dec 12, 2017

Why not also limit the number of successful pods which may takes a lot of resource either ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.