[BUG] WorkflowDependencySensor doesn't fails the workflow run if invalid "dependency_job_id" is given

[anshima03]

To do a negative testing we tried using a random dependency_job_id in WorkflowDependencySensor function expecting that the workflow will fail but it continued searching for the job_id.

sensor = WorkflowDependencySensor(
databricks_host="https://......cloud.databricks.com",
databricks_token='' ,
dependency_job_id='some random job id',
poke_interval_seconds=60,
timeout_seconds=3600,
delta=timedelta(days=1)
)

The expectation is that the job fails saying "this job id does not exist" or similar but we see that it keeps searching that random job id and is in loop.

Screenshots

Brickflow 10.3 version