Repository navigation
Monitoring Cronicle jobs with Watchgoose (start, success, failure signals) #997
bartekrutkowski
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi, I'm the founder of Watchgoose (https://watchgoose.com), a heartbeat monitor for cron jobs, backups and all kinds of recurring workloads.
I think a lot of people run backups and other scheduled jobs from Cronicle, and while Cronicle can tell you when a job fails, nothing can tell you when a job didn't run at all, for example because the server was down or the event got disabled. A heartbeat/dead-man-switch monitor can detect exactly that: the job pings a URL on every run and you get an alert when the ping stops arriving or reports a failure. There exosts request for exactly this in #873.
I tested two ways of doing it with Cronicle 0.9.142 and I think neither needs any cod changes in Cronicle:
Option 1: wrap the command in the Shell Plugin (recommended)
Save this as
/usr/local/bin/watchgoose-runon each worker andchmod +xit:Then the event's Shell Plugin script becomes:
This sends a start signal, runs the job then sends the exit code to
/<code>(0 is success, anything else is a failure) along with the last 10000 bytes of output. The output still shows in the Cronicle job log and the exit code is passed through so Cronicle marks the job failed as usual.Option 2: the Event Web Hook field (no script changes)
Put the check ping URL in the event (or category) Web Hook field:
Cronicle POSTs JSON to that one URL at job start and job end. Success and failure arrive at the same URL so turn on body keyword filtering for the check and use the text Cronicle puts in the
textfield:Job completed successfully onJob failed on,Failed to launch scheduled eventJob started onThere are two things to remember here:
textfield ends up past that point and the ping won't match. Keep the inline script short (call a script file) or use option 1web_hook_text_templatesthen use your own wording as the keywordsWith either of these options, run the event once by hand (Run Now) and confirm that the check shows as up. A newly created check only starts working and watchgin for missed runs after its first success ping.
In my tests a job that exited 3 and a job killed by its timeout, both came through as failures, normal runs came through as start + success. The full payload format is in the Event Web Hook docs.
Watchgoose has a support plan for Open Source maintainers (Open Source Program - Watchgoose).
I am happy to answer any questions.
Cheers!
All reactions