Add hc_check_started prometheus metrics

[ajgon]

This is simple fix adds another metric to prometheus endpoint - hc_check_in_progress. Its purpose is to return 1 for checks which are currently in progress (marked as started, and not finished or failed).

The main purpose are prometheus alerts. For example, in my case some of the cron tasks use 100% CPU, and it's normal - which means, I don't want to receive alerts from prometheus/alertmanager about CPU usage during this time. Thanks to this simple metric, I can add simple rule like node_load5 > 8 and on() hc_check_in_progress{name="my-heavy-cron-job"} == 0 and never be bothered unnecessarily again :)

[cuu508]

Thanks for the PR!

How about calling this metric hc_check_started instead of hc_check_in_progress?

It's shorter, and "started" is already used elsewhere in the codebase, "in_progress" isn't. Also I think it would describe the check's state better: the check was started, but no idea if it is still running or AWOL.

[ajgon]

Hah! The funny thing is, I originally thought about naming it hc_check_started, but eventually settled on _in_progress for some reason 😄

I fixed the PR 👍