Cailey/docs/cronjobs #548
Cailey/docs/cronjobs #548
Conversation
I would like to outright ban crons that run every minute, but "every 5 minutes" and less frequent is up to the discretion of the app team. |
|
I would suggest that a "bad" cronjob is one where the overhead is greater than the job workload (and you can make that a "per hour" calculation.) eg: your job is running and it takes 1 second, and runs every 5 minutes. This is a "Bad" cron - 20 seconds of application workload for 30 minutes of cluster effort. Recommend you stick this in a long running pod and scale the resources super low. backup eg: backup cron runs 1 every 4 hours and takes 10 minutes to complete once running. woot, this is a "Good" cron - 25 seconds of cluster work for 6x the app work... |
|
Regarding long running "Cron" pods: The current https://github.com/BCDevOps/backup-container has built in support for running as a long running pod or a kube cronjob. It's using go-crond as it's cron timer. I'd check it out that reference (since it's a BCGov community implementation) before looking at spending time evaluating another option like Jobber. (not that Jobber isn't interesting, but is there a reason we need to invent a new method instead of referencing Wade's work and calling it done?) |
|
👍 |
|
|
||
| ### Cronjobs Scheduled Too Often | ||
|
|
||
| We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. |
There was a problem hiding this comment.
| We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. | |
| We have a handy rule of thumb for deciding how often is too often - when your application workload duration is more than the cluster overhead duration for starting the job(s). (See below for an example ratio calculation) |
|
|
||
| We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. | ||
|
|
||
| If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object. |
There was a problem hiding this comment.
| If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object. | |
| If you find that you need to run a cronjob every minute, then you are far away from the "application workload > cluster overhead to start the workload" rule of thumb, and your job could be having a negative impact on cluster health. In these cases, your cronjob may be turned off my the Platform team and informed that you need to adjust your scheduled job design before re-enabling. |
| We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. | ||
|
|
||
| If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object. | ||
| If every minute is necessary for your app, you *must* use a persistent pod. |
There was a problem hiding this comment.
| If every minute is necessary for your app, you *must* use a persistent pod. | |
| If every minute is necessary for your app, this sounds like a great opportunity for a persistent cron deployment. |
| If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object. | ||
| If every minute is necessary for your app, you *must* use a persistent pod. | ||
|
|
||
| Outside of this limitation, however, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part. |
There was a problem hiding this comment.
| Outside of this limitation, however, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part. | |
| Outside of this rule of thumb, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part. |
|
|
||
| This version of the command will only pull up those cronjobs running once per minute - if you want to find cronjobs with other schedules, just adjust the grep filter accordingly. | ||
|
|
||
| ### Cronjobs with a Bad Ratio of Work to Overhead |
There was a problem hiding this comment.
| ### Cronjobs with a Bad Ratio of Work to Overhead | |
| ### Example calcuating Cronjob Ratio of Work to Overhead |
|
|
||
| There are a number of ways to approach creating this kind of persistent pod. | ||
|
|
||
| A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. |
There was a problem hiding this comment.
| A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. | |
| An existing example (with great documentation) of a long-running pod that implements a cronjob is the [Backup Container](https://github.com/BCDevops/backup-container) | |
| Another option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. |
| A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. | ||
| If your team finds a good solution that works well, we invite you to include a link to your deployment config on this document! | ||
|
|
||
| There is also an example of a long-running pod that implements a cronjob in the [backup container](https://github.com/BCDevOps/backup-container). |
There was a problem hiding this comment.
If above change to jobber line is accepted, then simply delete this line.
| There is also an example of a long-running pod that implements a cronjob in the [backup container](https://github.com/BCDevOps/backup-container). |
No description provided.