Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitor latest task runs per CLI #146

Merged
merged 10 commits into from Jun 19, 2021
Merged

Monitor latest task runs per CLI #146

merged 10 commits into from Jun 19, 2021

Conversation

nhoening
Copy link
Contributor

@nhoening nhoening commented Jun 17, 2021

New CLI function to monitor if tasks have run successfully and their latest run time is acceptable.

Alerts are sent to sentry and configurable email addresses.

@nhoening nhoening requested a review from Flix6x Jun 17, 2021
@nhoening
Copy link
Contributor Author

@nhoening nhoening commented Jun 17, 2021

I decided to have one CLI task for multiple task checks. We also know how often we call these tasks in the cron environment, so it's not a bad choice to also pass this info (how long ago is acceptable) in the same environment.

To test:

  • set your email address so you get the alert: MAIL_MONITORING_RECIPIENTS = "you@something.com"
  • check if your table latest_task_run contains any entries you might want to test, add one if you want
  • Call the CLI task, e.g.: flexmeasures monitor tasks --task get_historic_meter_data 5 --task run_forecasting_jobs 15 (where 5 and 15 are the amounts in minutes that are acceptable)

…un info as extra context. Also alert if no latest run exist.
@nhoening nhoening linked an issue Jun 17, 2021 that may be closed by this pull request
@nhoening nhoening changed the title Montitor tasks per CI Montitor latest task runs per CLI Jun 17, 2021
Flix6x
Flix6x approved these changes Jun 18, 2021
Copy link
Contributor

@Flix6x Flix6x left a comment

Missing changelog entry.

if (
now - acceptable_interval
<= latest_run.datetime
<= now + acceptable_interval
Copy link
Contributor

@Flix6x Flix6x Jun 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the now + acceptable_interval supposed to do? Doesn't seem documented.

Copy link
Contributor Author

@nhoening nhoening Jun 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I got this from Bobby's code. It checks if the latest run time isn't too far from the monitor's now perspective.

Well, re-thinking it, I am actually not sure this makes sense. This date can practically happen to be in the future from the monitor's perspective, but if the clocks of monitoring server and task-executing server are out of sync, that should become a different kind of warning I guess. Then the allowed interval isn't a good measurement, as we don't know how much we are out of bounds.

AAAnyway ― in our case the same server is executing and monitoring so I'll not do that extra warning. I'll simply remove the future check.

@nhoening nhoening changed the title Montitor latest task runs per CLI Monitor latest task runs per CLI Jun 19, 2021
@nhoening nhoening changed the base branch from issue-119-Start_using_Sentry_in_our_general_stack to main Jun 19, 2021
@nhoening nhoening merged commit e4a26dc into main Jun 19, 2021
2 checks passed
@nhoening nhoening deleted the montitor-tasks-per-CLI branch Jun 19, 2021
@Flix6x Flix6x added this to the 0.6.0 milestone Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants