If a cronjob fails, the corresponding pod isn't deleted. It repeats for every new pod.

[vineetchawla]

/kind bug
/sig bugs

What happened:
I created a cronjob using an example to echo "Hello world" every minute. Works perfectly.
If instead I replace the function to be called to wget with an unreachable website, the new pods keep being created and the older ones aren't deleted.

What you expected to happen:
I expected that once the pod for the job fails it should be deleted.

How to reproduce it (as minimally and precisely as possible):
kubectl run hello --schedule="*/1 * * * *" --restart OnFailure --image busybox -- /bin/wget www.asdf.asdf

Anything else we need to know?:
I just used the wrong website as an example to create cronjob failure. Any other command that fails results in the same issue

Environment:

• Kubernetes version (use kubectl version): v1.8.6
• Cloud provider or hardware configuration: minikube 1.9
• OS (e.g. from /etc/os-release): Mac high sierra
• Kernel (e.g. uname -a): darwin kernel 17.3.0
• Install tools:
• Others:

[vineetchawla]

/sig bugs

[CaoShuFeng]

FYI:

https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-termination-and-cleanup

The doc said:

When a Job completes, no more Pods are created, but the Pods are not deleted either. Keeping them around allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output.

[jhorwit2]

/sig apps
/kind bug

@vineetchawla can you please paste the output of kubectl get cronjob hello -oyaml. I'm going to take a guess you're not using the version of cronjobs that defaults the history limit, thus it's keeping an infinite amount.

[vineetchawla]

@jhorwit2 Here is the output
@caesarxuchao I tried the restart policy onFailure as well as never. Behaviour stays the same.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[caesarxuchao]

I see the failedJobsHistoryLimit is set to 1. It seems to be wrong that the failed jobs were not cleaned up. Not sure if it's a known bug in 1.8.6. @janetkuo might have more context.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[janetkuo]

A Job can only become Failed when it has .spec.backoffLimit set and has hit the limit.
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy

My theory is that you probably don't have this field set in your Job, so it never actually failed.

To further confirm, you can do something like this:

$ kubectl get jobs <job-name> -o json | jq '.status.conditions'

A failed job should return something like this:

[
  {
    "lastProbeTime": "2018-10-10T18:35:58Z",
    "lastTransitionTime": "2018-10-10T18:35:58Z",
    "message": "Job has reached the specified backoff limit",
    "reason": "BackoffLimitExceeded",
    "status": "True",
    "type": "Failed"
  }
]