Monitoring Framework

[irskep]

Tron needs a framework for doing monitoring.

Some potential monitoring systems:

• Email notification of failures
• Nagios notifications
• IRC updates
• Logging

Most we need a flexible framework of recording tron events and deciding what to pass onto configured monitors. Some of these may be integrated with how we do the web service framework. As in a nagios monitor could fairly easily use the web service interface to find the status of jobs. However sending an email on a failure would not work well with this system.

We now have support for crash emails, which is part of solving this guy.

Copied from rhettg#2

[irskep]

I'm thinking some of this would be simpler and more flexible with an event hook system. Example config:

hooks:
  succeed:
    - "echo 'succeeded: %(action_name)s'"
  fail:
    - "email user@domain.com 'failed: %(action_name)s'"

[irskep]

There has been no traffic on this. Still worth doing/keeping around as a ticket?

[dnephin]

I still like this idea, but it may be a few released out.

[justincinmd]

Are there any plans to do this? It would be extremely useful to have this for jobs with many downstream dependencies.

[dnephin]

I don't think we have any plans to work on this any time soon.

We have to figure out how to fix the underlying architectural issues with Tron before we make any additions to it.

[solarkennedy]

I've started a prototype for monitoring in #335.

[solarkennedy]

@irskep I'm going to close this ticket, as we have the check_tron_jobs command now. It isn't as flexible as running arbitrary commands for hooks, but it is useful for yelp developers as it uses the same pysensu-yelp interface available to them in other tools (like paasta).

[irskep]

OK! I haven't worked on Tron since 2012, so you know best. :-)