Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(core) data pipeline & stage alerting #55

Merged
merged 12 commits into from
Mar 24, 2022
Merged

(core) data pipeline & stage alerting #55

merged 12 commits into from
Mar 24, 2022

Conversation

malachi-constant
Copy link
Contributor

@malachi-constant malachi-constant commented Mar 15, 2022 •
edited
Loading

Changes

  • Adding add_notifications() method to pipeline class. This will allow setting a sns topic to consume alarms / event rules for the entire data pipeline.
  • Updating add_stage() to add Sns Actions for any alarms defined by the stage as long as a notifications topic is configured for the data pipeline.
  • Updating sqs to lambda stage with cloudwatch alarms property.
  • Updating sqs to lambda stage with cloudwatch alarm parameters to enable an alarm for lambda function errors.
  • Adds test for above stage with alarms configured.'
  • Updates full data pipeline test to check alarm and notification configuration.

Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@malachi-constant malachi-constant self-assigned this Mar 15, 2022
@malachi-constant malachi-constant marked this pull request as draft March 15, 2022 19:00
@malachi-constant malachi-constant changed the title adding 'add_notifications' method to 'DataPipeline()' (core) data pipeline & stage alerting Mar 15, 2022
@malachi-constant malachi-constant added this to the 0.2.0 milestone Mar 15, 2022
@malachi-constant malachi-constant linked an issue Mar 15, 2022 that may be closed by this pull request
(core) data pipeline alerting #35
Closed
@malachi-constant malachi-constant marked this pull request as ready for review March 15, 2022 19:56
@malachi-constant malachi-constant added the enhancement New feature or request label Mar 15, 2022
@jaidisido jaidisido removed this from the 0.2.0 milestone Mar 18, 2022
jaidisido
jaidisido previously requested changes Mar 21, 2022
View reviewed changes
core/aws_ddk_core/pipelines/pipeline.py Outdated Show resolved Hide resolved
core/aws_ddk_core/pipelines/pipeline.py Show resolved Hide resolved
core/aws_ddk_core/pipelines/pipeline.py Outdated Show resolved Hide resolved
core/aws_ddk_core/stages/sqs_lambda.py Outdated Show resolved Hide resolved
core/aws_ddk_core/stages/sqs_lambda.py Outdated Show resolved Hide resolved
core/aws_ddk_core/stages/sqs_lambda.py Outdated Show resolved Hide resolved
jaidisido
jaidisido reviewed Mar 24, 2022
View reviewed changes
Copy link
Contributor

@jaidisido jaidisido left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff, very minor comment.

Another thing to discuss is whether we want to add alarm_threshold & Co as arguments to the SQSToLambda stage?

core/aws_ddk_core/pipelines/stage.py Outdated Show resolved Hide resolved
@jaidisido jaidisido added this to In Progress in Roadmap Mar 24, 2022
@jaidisido jaidisido moved this from In Progress to Review in Roadmap Mar 24, 2022
@malachi-constant malachi-constant added this to the 0.3.0 milestone Mar 24, 2022
@malachi-constant malachi-constant merged commit e56a72d into main Mar 24, 2022
@malachi-constant malachi-constant moved this from Review to Done in Roadmap Mar 24, 2022
@jaidisido jaidisido deleted the feat/data-pipeline-alarting branch March 25, 2022 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kukushking kukushking kukushking approved these changes

@jaidisido jaidisido jaidisido approved these changes

@cnfait cnfait Awaiting requested review from cnfait

Assignees

@malachi-constant malachi-constant

Labels
core enhancement New feature or request
Projects
Roadmap
Done
Milestone
0.3.0
Development

Successfully merging this pull request may close these issues.

(core) data pipeline alerting
3 participants
@malachi-constant @kukushking @jaidisido