CronJob won't run again after being suspended long enough

[jhorwit2]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"0480917b552be33e2dba47386e51decb1a211df6", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Baremetal
• OS (e.g. from /etc/os-release): OL7
• Kernel (e.g. uname -a): 4.1.12-94.2.1.el7uek.x86_64
• Install tools: custom semi self-hosted
• Others:

What happened:

I tried to reenable a cronjob that I suspended this weekend; however, it won't schedule again. I see this in the controller manager logs a lot.

Cannot determine if <namespace>/<cronjob> needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

.spec.startingDeadlineSeconds is not set and there is no clock skew.

What you expected to happen:

I expect that when I remove the suspension that it would resume scheduling.

How to reproduce it (as minimally and precisely as possible):

I haven't tried reproducing it but I would assume if you suspend a cron for some amount of time then try to enable it again that it won't be scheduled.

Anything else we need to know:

[jhorwit2]

My quick guess at what is happening is that getRecentUnmetScheduleTimes does not take into account whether or not the job was recently suspended, so it appears that the cronjob missed too many scheduled starts, which is true but they are intentional misses.

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cronjob/cronjob_controller.go#L264

edit: That's seems to be the issue. The job was suspended long enough that it exceeded the magic 100 number.

[erictune]

/sig apps

[erictune]

Thanks for the detailed diagnosis, @jhorwit2, sounds right.

[0xmichalis]

Dupe of #42649?

[jhorwit2]

Yeah, it does appear to be the same underlying issue.

[jurgenweber]

ok, I have this in 1.7.10.... How do I get the cronjob to schedule again?

[surbas]

@jurgenweber been facing same issue. My work around is to delete the cron and create again.

[jurgenweber]

Yeah, that is what I had to resort to as well. Just annoying really. ;)

[dharmeshkakadia]

Can we please get some love on this bug :)
There is no good way to suspend a cron job and resume it. Are there alternatives ?

[0xmichalis]

This bug is a dupe of #42649