Amazon NeptuneStopDbClusterOperator bug

[ferruzzi]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

latest; AWS System test stack pulls from main

Apache Airflow version

latest; AWS System test stack pulls from main

Operating System

linux

Deployment

Docker-Compose

Deployment details

No response

What happened

On occasion the stop_db_cluster() call will return a "backing-up" state which is not handled, causing an exception. Example log message:

ERROR [airflow.task] Task failed with exception
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/taskinstance.py", line 447, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/opt/airflow/airflow/models/taskinstance.py", line 417, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
  File "/opt/airflow/airflow/providers/amazon/aws/operators/neptune.py", line 186, in execute
    resp = self.hook.conn.stop_db_cluster(DBClusterIdentifier=self.cluster_id)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidDBClusterStateFault: An error occurred (InvalidDBClusterStateFault) when calling the StopDBCluster operation: DbCluster env989356b0-cluster is in backing-up state but expected it to be one of available.
INFO  [airflow.models.taskinstance] Marking task as UP_FOR_RETRY. dag_id=example_neptune, task_id=stop_task, execution_date=20210101T000000, start_date=20240313T100059, end_date=20240313T100100

What you think should happen instead

If that is a possible state, it needs to be handled gracefully

How to reproduce

The issue is intermittent so I'm not 100% sure. Force a cluster to start a backup then immediately try to delete it while it is in progress.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ferruzzi]

@ellisms - This was your contribution, do you want to have a look at it?

[ellisms]

Yep, I'll take a look at it

[ferruzzi]

Cool, thanks. Assigned to you. Let me know if you need anything more from me, and feel free to ping me in the PR for a review.