Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NeptuneStopDbClusterOperator - Handle invalid cluster states #38287

Merged
merged 13 commits into from
May 22, 2024

Conversation

ellisms
Copy link
Contributor

@ellisms ellisms commented Mar 19, 2024


closes #38120

Handles case where the stop cluster api returns either InvalidClusterState or InvalidDBInstanceState exceptions. The operator now uses the waiter to wait for the cluster to be available. Also updated the start cluster operator to do the same.

@ellisms
Copy link
Contributor Author

ellisms commented Mar 19, 2024

@ferruzzi This is the PR for the neptune operators.

@eladkal
Copy link
Contributor

eladkal commented Apr 3, 2024

@ellisms is the PR ready for re-review?

@ellisms
Copy link
Contributor Author

ellisms commented Apr 3, 2024

@eladkal Not yet, just getting back into the office. Working on making the suggested changes.

@ellisms
Copy link
Contributor Author

ellisms commented Apr 16, 2024

This is ready for re-review.

Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple nitpicks, but nothing blocking. Feel free to resolve them if you disagree with any of them.

airflow/providers/amazon/aws/hooks/neptune.py Outdated Show resolved Hide resolved
airflow/providers/amazon/aws/operators/neptune.py Outdated Show resolved Hide resolved
airflow/providers/amazon/aws/operators/neptune.py Outdated Show resolved Hide resolved
tests/providers/amazon/aws/operators/test_neptune.py Outdated Show resolved Hide resolved
ellisms and others added 2 commits May 21, 2024 20:20
@eladkal eladkal merged commit a78ee74 into apache:main May 22, 2024
41 checks passed
RNHTTR pushed a commit to RNHTTR/airflow that referenced this pull request Jun 1, 2024
…apache#38287)

* Cache s3 resource to reduce memory usage

* Fixed missing rebase changes and error in resource init

* catch invalid state exceptions and wait if not terminal status

* PR changes

* Update airflow/providers/amazon/aws/triggers/neptune.py

Co-authored-by: Niko Oliveira <onikolas@amazon.com>

* Update airflow/providers/amazon/aws/operators/neptune.py

Co-authored-by: Niko Oliveira <onikolas@amazon.com>

* PR review changes. Created helper function to deal with waitable ClientErrors

* added missing exception code

* Update airflow/providers/amazon/aws/operators/neptune.py

Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>

* fix static checks

* Update airflow/providers/amazon/aws/operators/neptune.py

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

* Update airflow/providers/amazon/aws/operators/neptune.py

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

* minor PR changes

---------

Co-authored-by: Niko Oliveira <onikolas@amazon.com>
Co-authored-by: Vincent <97131062+vincbeck@users.noreply.github.com>
Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Amazon NeptuneStopDbClusterOperator bug
7 participants