Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerting / Notifications #419

Closed
jantman opened this issue Aug 28, 2019 · 4 comments

Comments

@jantman
Copy link
Owner

commented Aug 28, 2019

My employer would like built-in support in awslimitchecker for notifying/alerting someone when there are warning or critical thresholds crossed (or, in theory, if the run fails). We use PagerDuty but we'd like to make it easy for others to contribute integrations for other notification methods. I'm being allowed to work on this during my work time.

My thought is implementing something similar to the plan in #418 for metrics outputs - we'd have a NotificationProvider base class with a notify() method that takes a list of limits with critical thresholds passed and a list of limits with warning thresholds passed, and some configuration information from CLI options. It should also handle sending a notification if the run failed. The first subclass would be PagerDuty, but we'd accept PRs for additional ones.

@jantman

This comment has been minimized.

Copy link
Owner Author

commented Aug 30, 2019

Implementation plan is something similar to Metrics providers...

  • Similar CLI options as notification providers - list, specify class, and config options
  • Initialize before running the limit checks
  • wrap everything after initialize in a try/finally, so we can notify on failures/exceptions
  • send metrics first then notification
  • PagerDuty notifier
    • by default, if service_key is specified, send success/resolve if no issues
    • include formatted text output as well as data dict for warnings/criticals
    • config option for notification title/subject
    • config option for additional information to be included at end of notification
    • config options for an account name/alias
    • config option to specify a severity, instead of using the one from thresholds
jantman added a commit that referenced this issue Aug 30, 2019
jantman added a commit that referenced this issue Sep 3, 2019
@jantman

This comment has been minimized.

Copy link
Owner Author

commented Sep 4, 2019

Docs updates needed for this:

  • cli_usage.rst.template - alerts section like metrics
  • cli_usage.rst regenerate
  • development.rst section on adding alert providers
  • README.rst addition to features list
  • getting_started.rst addition to feature list, and addition to IAM permissions admonition
  • docs/build_generated_docs.py addition to IAM policy admonition and addition to examples
jantman added a commit that referenced this issue Sep 4, 2019
jantman added a commit that referenced this issue Sep 4, 2019
jantman added a commit that referenced this issue Sep 5, 2019
jantman added a commit that referenced this issue Sep 6, 2019
@jantman jantman referenced this issue Sep 6, 2019
14 of 14 tasks complete
jantman added a commit that referenced this issue Sep 6, 2019
jantman added a commit that referenced this issue Sep 6, 2019
@jantman

This comment has been minimized.

Copy link
Owner Author

commented Sep 6, 2019

This has been fixed by #425 and will be in the next release.

@jantman

This comment has been minimized.

Copy link
Owner Author

commented Sep 10, 2019

This was merged to master in #427 and released as 7.1.0. The release is now live on PyPI and the Docker image should be live on Docker Hub shortly.

@jantman jantman closed this Sep 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.