FivetranOperator stuck in deferred state till it times out

[dgrewal-tc]

Hi Team,
We are using the FivetranOperator to trigger a Fivetran connector. The connector completes in 5-10 minutes, but the Airflow task does not finish. It remains in a deferred state until it times out and fails.

Airflow Version : 2.9.2
airflow-provider-fivetran-async==2.1.0

Failed logs

2024-09-15, 10:26:53 UTC] INFO - Pausing task as DEFERRED.
[2024-09-15, 10:26:54 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 11:26:58 UTC] INFO - Resuming after deferral
[2024-09-15, 11:26:59 UTC] INFO - Executing <Task(FivetranOperator)> on 2024-09-14 08:00:00+00:00
[2024-09-15, 11:26:59 UTC] INFO - Started process 9527 to run task
[2024-09-15, 11:26:59 UTC] INFO - Job 918648: Subtask
[2024-09-15, 11:26:59 UTC] ERROR - Task failed with exception
Traceback (most recent call last):
airflow.exceptions.AirflowTaskTimeout

Success logs

[2024-09-15, 12:12:51 UTC] INFO - Pausing task as DEFERRED. 
[2024-09-15, 12:12:52 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 12:12:52 UTC] INFO - Using connection ID 'id' for task execution.
[2024-09-15, 12:12:52 UTC] INFO - sync is still running...
[2024-09-15, 12:12:52 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:14:53 UTC] INFO - sync is still running...
[2024-09-15, 12:14:53 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:16:53 UTC] INFO - sync is still running...
[2024-09-15, 12:16:53 UTC] INFO - sleeping for 120 seconds.

[2024-09-15, 12:19:00 UTC] INFO - Resuming after deferral
[2024-09-15, 12:19:00 UTC] INFO - Executing <Task(FivetranOperator):> on 2024-09-14 08:00:00+00:00
[2024-09-15, 12:19:00 UTC] INFO - Started process 11601 to run task
[2024-09-15, 12:19:00 UTC] INFO - Job 918720: Subtask
[2024-09-15, 12:19:00 UTC] INFO - Fivetran connector finished syncing at 2024-09-15 12:17:07.840000+00:00
[2024-09-15, 12:19:00 UTC] INFO - Marking task as SUCCESS.
[2024-09-15, 12:19:00 UTC] INFO - Task exited with return code 0

[pankajastro]

Hey @dgrewal-tc, Do you know what are delta between working and not working?

[dgrewal-tc]

@pankajastro
Here it is:

Working:

Airflow Version : 2.8.1
airflow-provider-fivetran-async==2.1.0

Not working:

Airflow Version : 2.9.2
airflow-provider-fivetran-async==2.1.0

[pankajastro]

Ok, then you should report an issue in https://github.com/apache/airflow