Send summary messages of your Luigi jobs to Slack
Python
Latest commit 4688f8a Sep 8, 2016 @alexdebrie alexdebrie committed on GitHub Version bump for bugfix
Permalink
Failed to load latest commit information.
luigi_monitor Failure and Failures Sep 8, 2016
.gitignore A few more pypi changes Sep 29, 2015
README.md Run luigi-monitor from cmdline. Sep 5, 2016
message.png Initial commit Sep 28, 2015
setup.py Version bump for bugfix Sep 8, 2016

README.md

Luigi Monitor

message

Send summary messages of your Luigi jobs to Slack.

Overview

Luigi is a great tool for making job pipelines, but it's hard to know the status of a run. Luigi has support for error emails, but this requires configuring your machine to send email, which is a hassle for short-lived EMR clusters. Further, it sends an email for every failure, which can quickly swamp your inbox.

By contrast, this tool gathers all your failures and missing dependencies and sends a summary Slack message when the job is finished.

Usage

import luigi
from luigi_monitor import monitor

...

if __name__ == "__main__":
    with monitor(slack_url=<your_slack_url>, max_print=10):
        luigi.run(main_task_cls=MainClass)

Alternatively:

luigi-monitor --module path.to.module TaskName

NB: if you plan to use luigi-monitor from the command line, set options using luigi.cfg:

[luigi-monitor]
slack_url=<slack_hook>
max_print=<int>

This is a work in progress. Particularly, note that:

  • It only sends notifications for FAILURE and DEPENDENCY_MISSING events.
  • It only sends notifications via Slack
  • If you have more than 5 notifications in a category (FAILURE or DEPENDENCY_MISSING), it will notify you of that rather than posting a long list of errors.