Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring execution time of script #23

Closed
ZUNbado opened this issue Dec 22, 2015 · 15 comments

Comments

Projects
None yet
9 participants
@ZUNbado
Copy link

commented Dec 22, 2015

It would be interesting to monitor the execution time of script, for example

curl https://hchk.io/UUID/start when script starts
curl https://hchk.io/UUID/end when script ends

And a option to alert on a maximum execution time user-defined, in ranges as same in checks

Kind regards

@cuu508 cuu508 added the enhancement label Dec 30, 2015

@diwu1989

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2015

why not just time the execution of the command yourself?

@ZUNbado

This comment has been minimized.

Copy link
Author

commented Dec 31, 2015

I think it's easier to send two requests (start and end).

I want to get an alert if execution time is longer than X

@danemacmillan

This comment has been minimized.

Copy link

commented Apr 28, 2017

For reference, cronitor.io provides this functionality and is very useful. For example, not all scripts or services properly return exit codes, which means chaining commands with && will run the healthchecks.io request regardless of the previous command's exit code, which is misleading, because a service or script that fails should return non-zero, so chained commands fail to execute, but not all do. In instances like this, knowing the average time a service or script runs will allow sending alerts based on a process that finished very fast, but did not provide a non-zero exit code signifying that it failed.

@audiolion

This comment has been minimized.

Copy link

commented Nov 6, 2017

I would like this feature as well, and am willing to create a PR for it (likely multiple as this touches database, services, frontend, additional rulesets/validations) if the maintainers would accept it.

This will require a refactor of the service as we need to identify which routes we are monitoring, as stated earlier the simplest way to link up which cron routes we want to monitor start/end times for, we use the same hchks.io address hash, but have /start /end appended to the URL to signify we want to track the timing.

After that refactor we can update the frontend templates to include the elapsed time of each checks start/end and provide an average for that endpoint.

Finally we could implement validation rules for alerts when healthchecks start/end time is above/below a certain time or standard deviation or whatever. E.g. if it took 3 standard deviations longer to run sends a notification, or if it runs 3 standard deviations faster it sends notification, or if it takes more than 60 seconds send a notification, or if it takes less than .1 seconds send a notification.

@cuu508

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2017

Hello @audiolion, I'm happy to review and merge PR(s) implementing this feature.

I like your suggestion about doing this in stages: 1) add API support 2) document the API support 3) expose the measured times in UI 4) add configurable alerting based on measured times.

@pladen

This comment has been minimized.

Copy link

commented Jun 1, 2018

cronalarm.com has this feature too
Between "start" and "end" pings the check is displayed in "running" state in the UI.
This is a great way to monitor what is running at the moment ...

@colinfrei

This comment has been minimized.

Copy link

commented Nov 16, 2018

Interested in this as well.

@JeremyWeir

This comment has been minimized.

Copy link

commented Nov 21, 2018

This feature would make choosing this project over cronitor/cronhub/deadmans snitch/etc a no brainer. Not having it is the reason I didn't just sign up at https://healthchecks.io

@cuu508

This comment has been minimized.

Copy link
Contributor

commented Nov 21, 2018

This would be an useful feature – agreed!
I'm planning on implementing it. It touches core parts of the system, so will take time to do properly.

For anybody reading this: are you mainly interested in just logging the execution time, or getting alerted when a job takes longer than expected to execute (even though it manages to finish before going "down")?

@pladen

This comment has been minimized.

Copy link

commented Nov 21, 2018

Logging the execution time is a must-have. Displaying in UI a job as running (ie between the start and end events) is also a great (and simple) feature.

However triggering an alert when getting higher than a threshold seems redundant to me. Periodic check + grace time is sufficient. If my cron job is not finished in time (grace time exhausted), i just want ONE alert that tell me "didnt run" or "didnt finish in time". I don't need another setting with running time, which will mean TWO redundant alerts.

@JeremyWeir

This comment has been minimized.

Copy link

commented Nov 22, 2018

I'm interested in being alerted when the job takes longer than expected. With the interval/grace only, you might never notice a job growing slowly in execution time over time.

@cuu508

This comment has been minimized.

Copy link
Contributor

commented Dec 21, 2018

I've now added a /start endpoint. For now I'm treating it as an experimental feature and so I haven't updated the documentation yet.

Here's how it works for now. Let's say your ping URL is

https://hc-ping.com/bddfa063-d3ca-4fc6-8220-8ff6c712d4f7

Append "/start" at the end and that's the /start endpoint:

https://hc-ping.com/bddfa063-d3ca-4fc6-8220-8ff6c712d4f7/start

When the /start endpoint receives a request, a few things happen:

  • Healthchecks logs a [Started] event. You will see it in the log, along with [OK] and [Failure] events
  • In the web UI, check's status changes to "Started" ("▶" icon)
  • It kicks off a timer: the check now must receive a ping within the grace time, or it will go down

In other words, if a job signals a "start", we expect it to then signal "success" within its grace time. If it does not, we send an alert.

There are still edge cases that I'm still thinking about how to handle best. I also have UI ideas on how to present the new functionality better. But this seems like a logical point to deploy the current implementation and give a status update.

If you're interested in this functionality and get a chance, you're welcome to try it out. If you notice any bugs, inconsistencies, or have other ideas on how to improve it, I'm all ears.

@cuu508

This comment has been minimized.

Copy link
Contributor

commented Dec 25, 2018

Released in v1.4.0

@cuu508 cuu508 closed this Dec 25, 2018

@pladen

This comment has been minimized.

Copy link

commented Jan 7, 2019

in test for 1 week now, it is working great
thank you

@tophee

This comment has been minimized.

Copy link

commented Apr 30, 2019

Just started testing this and have a quick suggestions:

  1. Regarding UI: don't use green (or at least not the same green as for the all-OK check-mark:
    image
    Perhaps orange would be appropriate for a started timer?

  2. Regarding syntax (or whatever category this falls under). I understand that the intended usage is that you start the timer with an appended /start and stop it with an ordinary ping (without any endpoint). It would be nice, though, if it would also accept a ping with a /stop endpoint for stopping the timer.

Why this redundancy? Not a big deal, but in my mind it just makes sense that a start command is followed by a stop (I intuitively appended a /stop before properly reading the instructions). But intuition and aesthetics aside, I think it also increases the readability of scripts if I can see that this is where the timer starts and this is where it stops.

Notably, though, while stopping a running timer should work both with and without /stop, a /stop should not be accepted as an ordinary ping (i.e. it will do nothing if it wasn't preceded by a /start)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.