[FIX] typo in parameter

[gaurav7261]

fix typo in databricks parameter
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[gaurav7261]

Hi @Lee-W , I see the repair run in deferrable operator is not in 2.9.0 yet, do you know when will the new release of airflow happening

[Lee-W]

Hi @gaurav7261 , it's not related to airflow but apache-airflow-providers-databricks. I think it'll be released in the next wave of RC which should happen sometime this week if no new bug found

[gaurav7261]

Thanks @Lee-W , also @Lee-W one more question, we use deferrable airflow operator, initially we use runnow operator, somehow in deferrable operator, we always get same exception Exception: Job run failed with terminal state: {'life_cycle_state': 'INTERNAL_ERROR', 'result_state': 'FAILED', 'state_message': 'Task FLAT_CREATION_pl_cafe_inventory_balance failed with message: Workload failed, see run output for details. This caused all downstream tasks to get skipped.'} using context.get('exception'), but same was working with exact job error from databricks, is this also fixed or is it expected behaviour in deferrable mode or can I fix it, just want to know whether you have any context for same or not

[Lee-W]

Thanks @Lee-W , also @Lee-W one more question, we use deferrable airflow operator, initially we use runnow operator, somehow in deferrable operator, we always get same exception Exception: Job run failed with terminal state: {'life_cycle_state': 'INTERNAL_ERROR', 'result_state': 'FAILED', 'state_message': 'Task FLAT_CREATION_pl_cafe_inventory_balance failed with message: Workload failed, see run output for details. This caused all downstream tasks to get skipped.'} using context.get('exception'), but same was working with exact job error from databricks, is this also fixed or is it expected behaviour in deferrable mode or can I fix it, just want to know whether you have any context for same or not

Might need more information to understand why it fails 🤔 I guess the triggerer component works fine. Otherwise, you won't be able to see this message. This is how it works under the hook. Maybe we could test it directly and see whether it works.

[gaurav7261]

Ya but using RunNowOperator, somehow airflow operator is able to give us exact databricks error like Column Not Found in query issue etc, we are able to get same via airflow context, but via deferrable operator, we always 100% get same this error Exception: Job run failed with terminal state: {'life_cycle_state': 'INTERNAL_ERROR', 'result_state': 'FAILED', 'state_message': 'Task FLAT_CREATION_pl_cafe_inventory_balance failed with message: Workload failed, see run output for details. This caused all downstream tasks to get skipped.'} only databricks task name change.

[gaurav7261]

I think the main issue is https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L159
VS
https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L71

what you think @Lee-W , in RunNowOperator handler we get the notebook error as well 🤔

[Lee-W]

I think the main issue is https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L159 VS https://github.com/apache/airflow/blob/main/airflow/providers/databricks/operators/databricks.py#L71

what you think @Lee-W , in RunNowOperator handler we get the notebook error as well 🤔

Yep, looks like this is the root cause. Ideally, they should be the same, but sometimes some features might be missing 😞 Are you interested in creating a PR for this fix? 🙂

[gaurav7261]

will do today

[gaurav7261]

FYI: @Lee-W #39110