Fail DbtCloudHook.wait_for_job_run_status on ERROR or CANCELLED dbt Cloud job runs

[SameerMesiah97]

Description

This change updates DbtCloudHook.wait_for_job_run_status to raise an exception when a dbt Cloud job run reaches an unexpected terminal state (ERROR or CANCELLED), instead of returning False.

The method now behaves as a safe synchronization primitive:

• Returns True when the job run reaches an expected status.
• Raises DbtCloudJobRunException on terminal failure before the expected state.
• Raises on timeout if the expected status is not reached in time.

Rationale

wait_for_job_run_status blocks until a job run completes and defaults to waiting for SUCCESS. Returning a boolean for terminal failure states is error-prone in an Airflow context, where task success and failure are exception-driven.

The previous behavior allowed Airflow tasks to complete successfully even when the underlying dbt Cloud job failed, which is surprising given the method name, blocking behavior, and default expectations. Raising on unexpected terminal states aligns the method with Airflow execution semantics and prevents silent task success.

Notes

DbtCloudRunJobOperator.on_kill now treats cancellation confirmation as best-effort and guards against propagated exceptions to avoid masking task termination. Previously, this method relied on a boolean return value from wait_for_job_run_status to confirm cancellation. Since wait_for_job_run_status can now raise DbtCloudJobRunException under normal operation (for example, if the job reaches an unexpected terminal state or the confirmation times out), allowing those exceptions to propagate from on_kill could incorrectly surface as task failures during shutdown.

Tests

Updated unit tests for wait_for_job_run_status to assert exception-driven behavior on terminal job run failures.

Documentation

The docstring for wait_for_job_run_status has been updated to explicitly document its exception-driven behavior, including the conditions under which DbtCloudJobRunException is raised.

Backwards Compatibility

This change does not alter the public API or method signature, but it does change runtime behavior:

• Terminal job run failures now raise instead of returning False.
• Callers relying on boolean inspection must handle exceptions instead.

This behavior is consistent with Airflow’s exception-driven model and matches user expectations for a blocking “wait until complete” helper.

Closes: #61297