livy provider can not kill the spark job when the trigger is timeout.

[coffee34]

Apache Airflow Provider(s)

apache-livy

Versions of Apache Airflow Providers

3.5.4

Apache Airflow version

2.7.2

Operating System

Amazon MWAA

Deployment

Amazon (AWS) MWAA

Deployment details

I used LivyOperator (deferrable = True) to create spark job by livy, and execution_timeout is 2h.

What happened

When the spark job run for more than 2h, airflow will detect it's timeout and cancel the trigger.
And then wake up the task to execute on_kill function.
Then livy operator failed with
AttributeError: 'LivyOperator' object has no attribute '_batch_id'
when it execute on_kill function.

What you think should happen instead

The Livy operator should fail due to a timeout, and simultaneously, it should kill the Spark job.

How to reproduce

Utilize the LivyOperator in your workflow.
Configure the operator with deferrable=True and execution_timeout=600 (for a 10-minute timeout).
Execute a Spark job that is expected to run for longer than 10 minutes.

Anything else

Seems like when the livy operator is rewaked up after trigger is time out, the _batch_id here is not initialized, which cause it failed to run on_kill function.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[mateuslatrova]

Hi, I would like to work on this issue!

[coffee34]

Oh thank you so much