Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

dbarrundiag · 2021-11-17T23:30:26Z

This commit adds new exciting features to the Airflow AWS Redshift module:

New Operators to manage the status of a Redshift Cluster

We are implementing two new Operators: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator. These operators will let developers leverage Airflow to resume and pause Redshift clusters before/after loading data to optimize for costs.

…ilar to the aws ec2 hook, this commit implements a wait_for_state function in the redshift hook module. Additionally we are implementing two new Operators: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator. These operators will let developers leverage Airflow to resume and pause Redshift clusters before/after loading data to optimize for costs.

boring-cyborg · 2021-11-17T23:30:29Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

… files, run isort, and add new operators to the amazon provider.yaml

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py

airflow/providers/amazon/aws/hooks/redshift.py

…rings and remove `wait_for_state` method in favor of the sensor at `airflow/providers/amazon/aws/sensors/redshift.py`

josh-fell · 2021-11-22T05:16:04Z

WDYT about information about these new operators to the existing AWS operator docs?

https://github.com/apache/airflow/tree/main/docs/apache-airflow-providers-amazon/operators

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py

…` method instead of doing the cast in all Operators that use the function.

airflow/providers/amazon/aws/sensors/redshift.py

… and change type of `target_status`

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py

airflow/providers/amazon/aws/operators/redshift_resume_cluster.py

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py

airflow/providers/amazon/aws/operators/redshift_resume_cluster.py

tests/providers/amazon/aws/operators/test_redshift_pause_cluster.py

tests/providers/amazon/aws/operators/test_redshift_resume_cluster.py

tests/providers/amazon/aws/sensors/test_redshift.py

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

…erators' into redshift-resume-pause-cluster-operators

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

airflow/providers/amazon/aws/operators/redshift_resume_cluster.py

…unified file.

…erators' into redshift-resume-pause-cluster-operators # Conflicts: # airflow/providers/amazon/aws/sensors/redshift.py # docs/apache-airflow-providers-amazon/operators/redshift.rst

…unified file.

airflow/providers/amazon/aws/operators/redshift.py

tests/providers/amazon/aws/operators/test_redshift.py

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

…erators' into redshift-resume-pause-cluster-operators

tests/providers/amazon/aws/operators/test_redshift.py

dstandish

These two comments should also be applied to the pause cluster case

tests/providers/amazon/aws/operators/test_redshift.py

dstandish

small fixups

tests/providers/amazon/aws/sensors/test_redshift.py

airflow/providers/amazon/aws/sensors/redshift.py

dstandish

nice work, and thanks for bearing with all the changes.

github-actions · 2021-12-13T19:25:28Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

dstandish

need to take another look at module names

boring-cyborg · 2021-12-13T19:53:44Z

Awesome work, congrats on your first merged pull request!

boring-cyborg bot added area:providers provider:amazon-aws AWS/Amazon - related issues labels Nov 17, 2021

dbarrundia3 added 2 commits November 18, 2021 07:02

Add missing requirements for Static checks. Add license to all python…

7a921c5

… files, run isort, and add new operators to the amazon provider.yaml

Adding changes for pre-commit success

0edb98d

o-nikolas reviewed Nov 19, 2021

View reviewed changes

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py Outdated Show resolved Hide resolved

o-nikolas reviewed Nov 19, 2021

View reviewed changes

airflow/providers/amazon/aws/hooks/redshift.py Outdated Show resolved Hide resolved

Adding ClusterStates Enum to redshift hook to avoid having magic st…

9911c72

…rings and remove `wait_for_state` method in favor of the sensor at `airflow/providers/amazon/aws/sensors/redshift.py`

Adding documentation to existing AWS operator docs

a68cbbf

dbarrundiag requested review from mik-laj and vikramkoka as code owners November 22, 2021 14:09

ferruzzi reviewed Nov 22, 2021

View reviewed changes

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py Outdated Show resolved Hide resolved

o-nikolas requested changes Nov 22, 2021

View reviewed changes

airflow/providers/amazon/aws/operators/redshift_pause_cluster.py Outdated Show resolved Hide resolved

Moving the cast to ClusterStates Enum to inside the `cluster_status…

b20a38e

…` method instead of doing the cast in all Operators that use the function.

o-nikolas approved these changes Nov 22, 2021

View reviewed changes

airflow/providers/amazon/aws/sensors/redshift.py Outdated Show resolved Hide resolved

Fixed logging to redshift sensor to show the actual value of the ENUM…

81f2f75

… and change type of `target_status`

o-nikolas approved these changes Nov 22, 2021

View reviewed changes

dbarrundia3 added 2 commits November 29, 2021 06:49

Fixed logging to redshift sensor to show the actual value of the ENUM…

61b2935

… and change type of `target_status`

Adding seealso to Operator documentation

3b3f1e0

dstandish requested changes Nov 30, 2021

View reviewed changes

dbarrundiag and others added 8 commits November 30, 2021 13:23

Update airflow/providers/amazon/aws/operators/redshift_resume_cluster.py

b151337

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Update airflow/providers/amazon/aws/operators/redshift_pause_cluster.py

ec92273

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Removing unused check_interval from operator

3ce29da

Merge remote-tracking branch 'origin/redshift-resume-pause-cluster-op…

456cb73

…erators' into redshift-resume-pause-cluster-operators

Update airflow/providers/amazon/aws/sensors/redshift.py

4d6e402

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Update docs/apache-airflow-providers-amazon/operators/redshift.rst

57c4c35

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Update docs/apache-airflow-providers-amazon/operators/redshift.rst

e03031f

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Update tests/providers/amazon/aws/sensors/test_redshift.py

92cfcf2

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

eladkal reviewed Dec 1, 2021

View reviewed changes

airflow/providers/amazon/aws/operators/redshift_resume_cluster.py Outdated Show resolved Hide resolved

dbarrundia3 added 5 commits December 7, 2021 14:49

Rolling back changes to not use enum and move all Operators into one …

06a7e3d

…unified file.

Merge remote-tracking branch 'origin/redshift-resume-pause-cluster-op…

d20662a

…erators' into redshift-resume-pause-cluster-operators # Conflicts: # airflow/providers/amazon/aws/sensors/redshift.py # docs/apache-airflow-providers-amazon/operators/redshift.rst

Rolling back changes to not use enum and move all Operators into one …

e72bec0

…unified file.

Rolling back changes to not use enum and move all Operators into one …

e659253

…unified file.

Rolling back changes to not use enum and move all Operators into one …

b303dc7

…unified file.

dstandish requested changes Dec 7, 2021

View reviewed changes

dbarrundiag and others added 4 commits December 7, 2021 16:44

Update airflow/providers/amazon/aws/operators/redshift.py

6c63fed

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Update airflow/providers/amazon/aws/operators/redshift.py

3ac8f33

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Removing no longer required subclass

50becc6

Merge remote-tracking branch 'origin/redshift-resume-pause-cluster-op…

653d6b2

…erators' into redshift-resume-pause-cluster-operators

dstandish self-requested a review December 7, 2021 22:56

dstandish requested changes Dec 8, 2021

View reviewed changes

tests/providers/amazon/aws/operators/test_redshift.py Outdated Show resolved Hide resolved

eladkal mentioned this pull request Dec 8, 2021

Restructure Amazon provider operators files #20139

Closed

13 tasks

Add unittests for TestPauseClusterOperator and TestResumeClusterOperator

f012c29

dstandish requested changes Dec 9, 2021

View reviewed changes

tests/providers/amazon/aws/operators/test_redshift.py Outdated Show resolved Hide resolved

tests/providers/amazon/aws/operators/test_redshift.py Outdated Show resolved Hide resolved

Clean unittest for redshift Operators

a7ff6ca

dstandish requested changes Dec 9, 2021

View reviewed changes

tests/providers/amazon/aws/sensors/test_redshift.py Outdated Show resolved Hide resolved

airflow/providers/amazon/aws/sensors/redshift.py Outdated Show resolved Hide resolved

Small fixups

f4ac7a4

dstandish approved these changes Dec 13, 2021

View reviewed changes

github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Dec 13, 2021

dstandish reviewed Dec 13, 2021

View reviewed changes

dstandish self-requested a review December 13, 2021 19:31

dstandish approved these changes Dec 13, 2021

View reviewed changes

dstandish merged commit e77c05f into apache:main Dec 13, 2021

potiuk mentioned this pull request Dec 31, 2021

Status of testing Providers that were prepared on December 31, 2021 #20615

Closed

91 tasks

potiuk mentioned this pull request Feb 5, 2022

Status of testing Providers that were prepared on February 05, 2022 #21348

Closed

54 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

dbarrundiag commented Nov 17, 2021 •

edited

Loading

boring-cyborg bot commented Nov 17, 2021

josh-fell commented Nov 22, 2021 •

edited

Loading

dstandish left a comment

dstandish left a comment

dstandish left a comment

github-actions bot commented Dec 13, 2021

dstandish left a comment

boring-cyborg bot commented Dec 13, 2021

Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

Conversation

dbarrundiag commented Nov 17, 2021 • edited Loading

New Operators to manage the status of a Redshift Cluster

boring-cyborg bot commented Nov 17, 2021

josh-fell commented Nov 22, 2021 • edited Loading

dstandish left a comment

Choose a reason for hiding this comment

dstandish left a comment

Choose a reason for hiding this comment

dstandish left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2021

dstandish left a comment

Choose a reason for hiding this comment

boring-cyborg bot commented Dec 13, 2021

dbarrundiag commented Nov 17, 2021 •

edited

Loading

josh-fell commented Nov 22, 2021 •

edited

Loading