Target health rules not overriding resource definition

[kijewskimateusz]

Describe the issue

Health rules are not overriden by the databricks.yml configuration inside the target block which cause metric conflicht

Configuration

resources.yml file is as follows

resources:
  jobs:
    stand_plan_live_continuous_job:
      name: Live - Continuous load of Stand Plan data
      schedule:
        quartz_cron_expression: "0 */5 * * * ?"
        timezone_id: UTC
      health:
        rules:
          - metric: RUN_DURATION_SECONDS
            op: GREATER_THAN
            value: 300

databricks.yml content is as follows

include:
  - resources/*.yml

targets:
  qa:
    mode: production
    default: false
    workspace:
      host: ***
      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}
    resources:
      jobs:
        stand_plan_live_continuous_job:
          health:
            rules:
              - metric: RUN_DURATION_SECONDS
                op: GREATER_THAN
                value: 900
          schedule:
            quartz_cron_expression:  "0 0 */2 * * ?"
            timezone_id: UTC
    presets:
      trigger_pause_status: UNPAUSED

Steps to reproduce the behavior

1. Eet up pipeline with health rules configuration provided in both resources.yml and databricks.yml
2. Run Databricks bundle deploy

Expected Behavior

Pipeline gets deployed with health rule RUN_DURATION_SECONDS set to 900 seconds

Actual Behavior

Pipeline fails with information that health rule metric has already been found.

❯ databricks bundle deploy -t qa
Uploading bundle files to /Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files...
Deploying resources...
Updating deployment state...
Error: terraform apply: exit status 1

Error: cannot update job: Duplicate (metric,op) pairs: (RUN_DURATION_SECONDS, GREATER_THAN) found in health rules

  with databricks_job.stand_plan_live_continuous_job,
  on bundle.tf.json line 426, in resource.databricks_job.stand_plan_live_continuous_job:
 426:       }

OS and CLI version

Please provide the version of the CLI (eg: v0.1.2) and the operating system (eg: windows). You can run databricks --version to get the version of your Databricks CLI

Is this a regression?

Don't know

Debug Logs

19:20:43 Info: start pid=43254 version=0.254.0 args="databricks, bundle, deploy, -t, qa, --log-level=debug"
19:20:43 Debug: Found bundle root at /Users/mateusz/work/sita/operations-at-airports/TotalOptimizer.data-platform (file /Users/mateusz/work/sita/operations-at-airports/TotalOptimizer.data-platform/databricks.yml) pid=43254
19:20:43 Info: Phase: load pid=43254
19:20:43 Debug: Apply pid=43254 mutator=EntryPoint
19:20:43 Debug: Apply pid=43254 mutator=scripts.preinit
19:20:43 Debug: No script defined for preinit, skipping pid=43254 mutator=scripts.preinit
19:20:43 Debug: Apply pid=43254 mutator=ProcessRootIncludes
19:20:43 Debug: Apply pid=43254 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/historical/operations_daily.yml)
19:20:43 Debug: Apply pid=43254 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/historical/operations_initial.yml)
19:20:43 Debug: Apply pid=43254 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/historical/revenue_initial.yml)
19:20:43 Debug: Apply pid=43254 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/live/stand_plan.yml)
19:20:43 Debug: Apply pid=43254 mutator=ProcessRootIncludes mutator=ProcessInclude(resources/databricks_dashboard/tmodashboard.yml)
19:20:43 Debug: Apply pid=43254 mutator=VerifyCliVersion
19:20:43 Debug: Apply pid=43254 mutator=EnvironmentsToTargets
19:20:43 Debug: Apply pid=43254 mutator=ComputeIdToClusterId
19:20:43 Debug: Apply pid=43254 mutator=InitializeVariables
19:20:43 Debug: Apply pid=43254 mutator=DefineDefaultTarget(default)
19:20:43 Debug: Apply pid=43254 mutator=validate:unique_resource_keys
19:20:43 Debug: Apply pid=43254 mutator=SelectTarget(qa)
19:20:43 Debug: Loading profile dbw-tao-qa-weu-001 because of host match pid=43254
19:20:43 Debug: Failed to configure auth: "pat" pid=43254 sdk=true
19:20:43 Debug: Failed to configure auth: "basic" pid=43254 sdk=true
19:20:44 Debug: GET /oidc/.well-known/oauth-authorization-server
< HTTP/2.0 200 OK
< {
<   "authorization_endpoint": "https://adb-1803404685110667.7.azuredatabricks.net/oidc/v1/authorize",
<   "claims_supported": [
<     "iss",
<     "sub",
<     "aud",
<     "iat",
<     "exp",
<     "jti",
<     "name",
<     "family_name",
<     "given_name",
<     "preferred_username"
<   ],
<   "code_challenge_methods_supported": [
<     "S256"
<   ],
<   "grant_types_supported": [
<     "client_credentials",
<     "authorization_code",
<     "refresh_token"
<   ],
<   "id_token_signing_alg_values_supported": [
<     "RS256"
<   ],
<   "issuer": "https://adb-1803404685110667.7.azuredatabricks.net/oidc",
<   "jwks_uri": "https://westeurope-c2.azuredatabricks.net/oidc/jwks.json",
<   "request_uri_parameter_supported": false,
<   "response_modes_supported": [
<     "query",
<     "fragment",
<     "form_post"
<   ],
<   "response_types_supported": [
<     "code",
<     "id_token"
<   ],
<   "scopes_supported": [
<     "all-apis",
<     "email",
<     "offline_access",
<     "openid",
<     "profile",
<     "sql"
<   ],
<   "subject_types_supported": [
<     "public"
<   ],
<   "token_endpoint": "https://adb-1803404685110667.7.azuredatabricks.net/oidc/v1/token",
<   "token_endpoint_auth_methods_supported": [
<     "client_secret_basic",
<     "client_secret_post",
<     "none"
<   ]
< } pid=43254 sdk=true
19:20:44 Debug: Generating Databricks OAuth token for Service Principal (062101b2-1f0d-494f-8df9-7aa882a5abe2) pid=43254 sdk=true
19:20:44 Debug: Apply pid=43254 mutator=<func>
19:20:44 Debug: Apply pid=43254 mutator=<func>
19:20:44 Info: Phase: initialize pid=43254
19:20:44 Debug: Apply pid=43254 mutator=validate:AllResourcesHaveValues
19:20:44 Debug: Apply pid=43254 mutator=validate:interpolation_in_auth_config
19:20:44 Debug: Apply pid=43254 mutator=RewriteSyncPaths
19:20:44 Debug: Apply pid=43254 mutator=SyncDefaultPath
19:20:44 Debug: Apply pid=43254 mutator=SyncInferRoot
19:20:44 Debug: Apply pid=43254 mutator=PopulateCurrentUser
19:20:44 Debug: POST /oidc/v1/token
> <http.RoundTripper>
< HTTP/2.0 200 OK
< {
<   "access_token": "**REDACTED**",
<   "expires_in": 3600,
<   "scope": "all-apis",
<   "token_type": "Bearer"
< } pid=43254 sdk=true
19:20:44 Debug: GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "displayName": "AZR-SPN-totaloptimizer",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "062101b2-1f0d-494f-8df9-7aa882a5abe2"
<     }
<   ],
<   "groups": [
<     {
<       "$ref": "Groups/395515217574770",
<       "display": "admins",
<       "type": "direct",
<       "value": "395515217574770"
<     }
<   ],
<   "id": "2880969010418190",
<   "name": {
<     "givenName": "AZR-SPN-totaloptimizer"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
<   ],
<   "userName": "062101b2-1f0d-494f-8df9-7aa882a5abe2"
< } pid=43254 mutator=PopulateCurrentUser sdk=true
19:20:44 Debug: Apply pid=43254 mutator=LoadGitDetails
19:20:44 Debug: Apply pid=43254 mutator=ApplySourceLinkedDeploymentPreset
19:20:44 Debug: Apply pid=43254 mutator=DefineDefaultWorkspaceRoot
19:20:44 Debug: Apply pid=43254 mutator=ExpandWorkspaceRoot
19:20:44 Debug: Apply pid=43254 mutator=DefaultWorkspacePaths
19:20:44 Debug: Apply pid=43254 mutator=PrependWorkspacePrefix
19:20:44 Debug: Apply pid=43254 mutator=RewriteWorkspacePrefix
19:20:44 Debug: Apply pid=43254 mutator=SetVariables
19:20:44 Debug: Apply pid=43254 mutator=ResolveVariableReferences
19:20:44 Debug: Apply pid=43254 mutator=ResolveResourceReferences
19:20:44 Debug: Apply pid=43254 mutator=ResolveVariableReferences
19:20:44 Debug: Apply pid=43254 mutator=validate:volume-path
19:20:44 Debug: Apply pid=43254 mutator=ApplyTargetMode
19:20:44 Debug: Apply pid=43254 mutator=ConfigureWSFS
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=ResolveVariableReferences(resources)
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=NormalizePaths
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=ResolveVariableReferences(resources)
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=ExpandPipelineGlobPaths
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=MergeJobClusters
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=MergeJobParameters
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=MergeJobTasks
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=MergePipelineClusters
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=MergeApps
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=CaptureSchemaDependency
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=ConfigureDashboardSerializedDashboard
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=SetRunAs
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=OverrideCompute
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=ApplyPresets
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=validate:SingleNodeCluster
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {dashboards 0} {}], parent_path, /Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/resources)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {dashboards 0} {}], embed_credentials, false)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {volumes 0} {}], volume_type, MANAGED)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {}], name, Untitled)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {}], max_concurrent_runs, 1)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {schedule 0}], pause_status, UNPAUSED)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {trigger 0}], pause_status, UNPAUSED)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {continuous 0}], pause_status, UNPAUSED)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {task 0} {} {dbt_task 0}], schema, default)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {task 0} {} {for_each_task 0} {task 0} {dbt_task 0}], schema, default)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {job_clusters 0} {} {new_cluster 0}], num_workers, 0)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {job_clusters 0} {} {new_cluster 0} {workload_type 0} {clients 0}], notebooks, true)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {job_clusters 0} {} {new_cluster 0} {workload_type 0} {clients 0}], jobs, true)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {pipelines 0} {}], edition, ADVANCED)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {pipelines 0} {}], channel, CURRENT)"
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=DefaultQueueing
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=ApplyBundlePermissions
19:20:44 Debug: Apply pid=43254 mutator=ProcessStaticResources mutator=FilterCurrentUserFromPermissions
19:20:44 Debug: Apply pid=43254 mutator=PythonMutator(load)
19:20:44 Debug: Apply pid=43254 mutator=PythonMutator(init)
19:20:44 Debug: Apply pid=43254 mutator=PythonMutator(load_resources)
19:20:44 Debug: Apply pid=43254 mutator=PythonMutator(apply_mutators)
19:20:44 Debug: Apply pid=43254 mutator=CheckPermissions
19:20:44 Debug: Apply pid=43254 mutator=TranslatePaths
19:20:44 Debug: Apply pid=43254 mutator=PythonWrapperWarning
19:20:44 Debug: Apply pid=43254 mutator=artifacts.Prepare
19:20:44 Info: No local tasks in databricks.yml config, skipping auto detect pid=43254 mutator=artifacts.Prepare
19:20:44 Debug: Apply pid=43254 mutator=apps.Validate
19:20:44 Debug: Apply pid=43254 mutator=ValidateTargetMode
19:20:44 Debug: Apply pid=43254 mutator=ValidateSharedRootPermissions
19:20:44 Debug: Apply pid=43254 mutator=metadata.AnnotateJobs
19:20:44 Debug: Apply pid=43254 mutator=metadata.AnnotatePipelines
19:20:44 Debug: Apply pid=43254 mutator=terraform.Initialize
19:20:44 Debug: Using Terraform from DATABRICKS_TF_EXEC_PATH at /Users/mateusz/.windsurf/extensions/databricks.databricks-2.9.4-darwin-arm64/bin/dependencies/terraform pid=43254 mutator=terraform.Initialize
19:20:44 Debug: DATABRICKS_TF_PROVIDER_VERSION as 1.68.0 does not match the current version 1.81.0, ignoring DATABRICKS_TF_CLI_CONFIG_FILE pid=43254 mutator=terraform.Initialize
19:20:44 Debug: Environment variables for Terraform: DATABRICKS_CLIENT_SECRET, DATABRICKS_AUTH_TYPE, HOME, PATH, TMPDIR, DATABRICKS_USER_AGENT_EXTRA, DATABRICKS_HOST, DATABRICKS_CLIENT_ID pid=43254 mutator=terraform.Initialize
19:20:44 Debug: Apply pid=43254 mutator=scripts.postinit
19:20:44 Debug: No script defined for postinit, skipping pid=43254 mutator=scripts.postinit
19:20:44 Debug: Apply pid=43254 mutator=fast_validate(readonly)
19:20:44 Debug: ApplyParallel pid=43254 mutator=fast_validate(readonly) mutator=validate:job_cluster_key_defined
19:20:44 Debug: ApplyParallel pid=43254 mutator=fast_validate(readonly) mutator=validate:job_task_cluster_spec
19:20:44 Debug: ApplyParallel pid=43254 mutator=fast_validate(readonly) mutator=validate:artifact_paths
19:20:44 Info: Phase: build pid=43254
19:20:44 Debug: Apply pid=43254 mutator=scripts.prebuild
19:20:44 Debug: No script defined for prebuild, skipping pid=43254 mutator=scripts.prebuild
19:20:44 Debug: Apply pid=43254 mutator=artifacts.Build
19:20:44 Debug: Apply pid=43254 mutator=scripts.postbuild
19:20:44 Debug: No script defined for postbuild, skipping pid=43254 mutator=scripts.postbuild
19:20:44 Debug: Apply pid=43254 mutator=ResolveVariableReferences
19:20:44 Debug: Apply pid=43254 mutator=ResolveVariableReferences(resources)
19:20:44 Info: Phase: deploy pid=43254
19:20:44 Debug: Apply pid=43254 mutator=scripts.predeploy
19:20:44 Debug: No script defined for predeploy, skipping pid=43254 mutator=scripts.predeploy
19:20:44 Debug: Apply pid=43254 mutator=lock:acquire
19:20:44 Info: Acquiring deployment lock (force: false) pid=43254 mutator=lock:acquire
19:20:45 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock?overwrite=false
> {
>   "AcquisitionTime": "2025-06-10T19:20:44.7378+02:00",
>   "ID": "31317a4a-73df-4052-9778-f8bcd5153cc7",
>   "IsForced": false,
>   "User": "062101b2-1f0d-494f-8df9-7aa882a5abe2"
> }
< HTTP/2.0 200 OK pid=43254 mutator=lock:acquire sdk=true
19:20:45 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1749576045321,
<   "modified_at": 1749576045321,
<   "object_id": 521378236053684,
<   "object_type": "FILE",
<   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock",
<   "resource_id": "521378236053684"
< } pid=43254 mutator=lock:acquire sdk=true
19:20:45 Debug: GET /api/2.0/workspace/export?direct_download=true&path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock
< HTTP/2.0 200 OK
< {
<   "AcquisitionTime": "2025-06-10T19:20:44.7378+02:00",
<   "ID": "31317a4a-73df-4052-9778-f8bcd5153cc7",
<   "IsForced": false,
<   "User": "062101b2-1f0d-494f-8df9-7aa882a5abe2"
< } pid=43254 mutator=lock:acquire sdk=true
19:20:45 Debug: Apply pid=43254 mutator=terraform:state-pull
19:20:45 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/terraform.tfstate&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1749234954121,
<   "modified_at": 1749575640894,
<   "object_id": 4037883856920373,
<   "object_type": "FILE",
<   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/terraform.tf... (5 more bytes)",
<   "resource_id": "4037883856920373"
< } pid=43254 mutator=terraform:state-pull sdk=true
19:20:46 Debug: GET /api/2.0/workspace-files/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/terraform.tfstate
< HTTP/2.0 200 OK
< <Streaming response> pid=43254 mutator=terraform:state-pull sdk=true
19:20:46 Debug: Apply pid=43254 mutator=CheckDashboardsModifiedRemotely
19:20:46 Debug: Apply pid=43254 mutator=deploy:state-pull
19:20:46 Info: Opening remote deployment state file pid=43254 mutator=deploy:state-pull
19:20:46 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deployment.json&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1749234948561,
<   "modified_at": 1749575622722,
<   "object_id": 4037883856920372,
<   "object_type": "FILE",
<   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deployment.j... (3 more bytes)",
<   "resource_id": "4037883856920372"
< } pid=43254 mutator=deploy:state-pull sdk=true
19:20:47 Debug: GET /api/2.0/workspace-files/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deployment.json
< HTTP/2.0 200 OK
< <Streaming response> pid=43254 mutator=deploy:state-pull sdk=true
19:20:47 Info: Local deployment state is the same or newer, ignoring remote state pid=43254 mutator=deploy:state-pull
19:20:47 Debug: Apply pid=43254 mutator=ValidateGitDetails
19:20:47 Debug: Apply pid=43254 mutator=check-running-resources
19:20:47 Debug: Apply pid=43254 mutator=artifacts.CleanUp
19:20:47 Debug: POST /api/2.0/workspace/delete
> {
>   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/artifacts/.interna... (1 more bytes)",
>   "recursive": true
> }
< HTTP/2.0 200 OK
< {} pid=43254 mutator=artifacts.CleanUp sdk=true
19:20:47 Debug: POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/artifacts/.interna... (1 more bytes)"
> }
< HTTP/2.0 200 OK
< {} pid=43254 mutator=artifacts.CleanUp sdk=true
19:20:47 Debug: Apply pid=43254 mutator=libraries.ExpandGlobReferences
19:20:47 Debug: Apply pid=43254 mutator=CheckForSameNameLibraries
19:20:47 Debug: Apply pid=43254 mutator=SwitchToPatchedWheels
19:20:47 Debug: Apply pid=43254 mutator=libraries.Upload
19:20:47 Debug: Apply pid=43254 mutator=TransformWheelTask
19:20:47 Debug: Apply pid=43254 mutator=files.Upload
Uploading bundle files to /Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files...
19:20:47 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files
< HTTP/2.0 200 OK
< {
<   "object_id": 4037883856919466,
<   "object_type": "DIRECTORY",
<   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files",
<   "resource_id": "4037883856919466"
< } pid=43254 mutator=files.Upload sdk=true
19:20:47 Debug: Path /Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files has type directory (ID: 4037883856919466) pid=43254 mutator=files.Upload
19:20:48 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files/resources/live/stand_plan.yml?overwrite=true
> resources:
>   jobs:
>     initial_stand_plan_job:
>       name: Daily - Generate initial Stand Plans
> ... (4876 more bytes)
< HTTP/2.0 200 OK pid=43254 mutator=files.Upload sdk=true
19:20:48 Info: Uploaded bundle files pid=43254 mutator=files.Upload
19:20:48 Debug: Apply pid=43254 mutator=deploy:state-update
19:20:48 Info: Loading deployment state from /Users/mateusz/work/sita/operations-at-airports/TotalOptimizer.data-platform/.databricks/bundle/qa/deployment.json pid=43254 mutator=deploy:state-update
19:20:48 Debug: Apply pid=43254 mutator=deploy:state-push
19:20:48 Info: Writing local deployment state file to remote state directory pid=43254 mutator=deploy:state-push
19:20:49 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deployment.json?overwrite=true
> {
>   "cli_version": "0.254.0",
>   "files": [
>     {
>       "is_notebook": false,
>       "local_path": "scratch/README.md"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "src/Naples/Functional_Unit_Testing/Functional_Unit_Testing_Flights_Data.py"
>     },
>     {
>       "is_notebook": true,
>       "local_path": "src/Naples/Operations/Operations_ETL.py"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "src/dashboard/TMO_Dashboard.lvdash.json"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "taskfile.yml"
>     },
>     {
>       "is_notebook": false,
>       "local_path": ".windsurfrules"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "docs/technical.md"
>     },
>     {
>       "is_notebook": true,
>       "local_path": "src/Naples/GenericFramework_Animesh/Generic Framework for validation check Revenue (1).py"
>     },
>     {
>       "is_notebook": true,
>       "local_path": "src/Naples/Live_code/cnfg_live_baseline_standplan_merging_linked_and_unlinked_d2.py"
>     },
>     {
>       "is_notebook": true,
>       "local_path": "scratch/Clone data between workspaces .py"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "resources/historical/operations_daily.yml"
>     },
>     {
>       "is_notebook": true,
>       "local_path": "scratch/Check Workspace IP address and call AOSNice APIs.py"
>     },
>     {
>       "is_notebook": true,
>       "local_path": "src/Naples/Live_code/cnfg_initial_live_data_load.py"
>     },
>     "... (63 additional elements)"
>   ],
>   "id": "150cd91d-8d8a-4bc6-b033-2857446471b0",
>   "seq": 146,
>   "timestamp": "2025-06-10T17:20:48.55414Z",
>   "version": 1
> }
< HTTP/2.0 200 OK pid=43254 mutator=deploy:state-push sdk=true
19:20:49 Debug: Apply pid=43254 mutator=ApplyWorkspaceRootPermissions
19:20:49 Debug: Apply pid=43254 mutator=terraform.Interpolate
19:20:49 Debug: Apply pid=43254 mutator=terraform.Write
19:20:49 Debug: job normalization diagnostic: unknown field: permissions pid=43254 mutator=terraform.Write
19:20:49 Debug: job normalization diagnostic: unknown field: permissions pid=43254 mutator=terraform.Write
19:20:49 Debug: job normalization diagnostic: unknown field: permissions pid=43254 mutator=terraform.Write
19:20:49 Debug: job normalization diagnostic: unknown field: permissions pid=43254 mutator=terraform.Write
19:20:49 Debug: job normalization diagnostic: unknown field: permissions pid=43254 mutator=terraform.Write
19:20:49 Debug: job normalization diagnostic: unknown field: permissions pid=43254 mutator=terraform.Write
19:20:49 Debug: Apply pid=43254 mutator=terraform.Plan
19:21:26 Debug: Planning complete and persisted at /Users/mateusz/work/sita/operations-at-airports/TotalOptimizer.data-platform/.databricks/bundle/qa/terraform/plan
 pid=43254 mutator=terraform.Plan
19:21:26 Debug: Apply pid=43254 mutator=trackUsedCompute
Deploying resources...
19:21:26 Debug: Apply pid=43254 mutator=terraform.Apply
19:21:28 Debug: Error: terraform apply: exit status 1

Error: cannot update job: Duplicate (metric,op) pairs: (RUN_DURATION_SECONDS, GREATER_THAN) found in health rules

  with databricks_job.stand_plan_live_continuous_job,
  on bundle.tf.json line 426, in resource.databricks_job.stand_plan_live_continuous_job:
 426:       }

 pid=43254 mutator=terraform.Apply
19:21:28 Debug: Apply pid=43254 mutator=terraform:state-push
Updating deployment state...
19:21:28 Info: Writing local state file to remote state directory pid=43254 mutator=terraform:state-push
19:21:28 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/terraform.tfstate?overwrite=true
> {
>   "check_results": null,
>   "lineage": "dc70fdbc-a668-10e9-8b8c-1b4662218ace",
>   "outputs": {},
>   "resources": [
>     {
>       "instances": [
>         {
>           "attributes": {
>             "always_running": false,
>             "budget_policy_id": null,
>             "continuous": null,
>             "control_run_state": false,
>             "dbt_task": null,
>             "deployment": [
>               {
>                 "kind": "BUNDLE",
>                 "metadata_file_path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/metadata.jso... (1 more bytes)"
>               }
>             ],
>             "description": null,
>             "edit_mode": "UI_LOCKED",
>             "email_notifications": [
>               {
>                 "no_alert_for_skipped_runs": false,
>                 "on_duration_warning_threshold_exceeded": null,
>                 "on_failure": null,
>                 "on_start": null,
>                 "on_streaming_backlog_exceeded": null,
>                 "on_success": null
>               }
>             ],
>             "environment": null,
>             "existing_cluster_id": null,
>             "format": "MULTI_TASK",
>             "git_source": null,
>             "health": null,
>             "id": "908143205118192",
>             "job_cluster": [
>               {
>                 "job_cluster_key": "history_table_job_cluster",
>                 "new_cluster": [
>                   {
>                     "apply_policy_default_values": false,
>                     "autoscale": null,
>                     "aws_attributes": null,
>                     "azure_attributes": [
>                       {
>                         "availability": "ON_DEMAND_AZURE",
>                         "first_on_demand": 0,
>                         "log_analytics_info": null,
>                         "spot_bid_max_price": 0
>                       }
>                     ],
>                     "cluster_id": "",
>                     "cluster_log_conf": null,
>                     "cluster_mount_info": null,
>                     "cluster_name": "",
>                     "custom_tags": {},
>                     "data_security_mode": "SINGLE_USER",
>                     "docker_image": null,
>                     "driver_instance_pool_id": "",
>                     "driver_node_type_id": "",
>                     "enable_elastic_disk": true,
>                     "enable_local_disk_encryption": false,
>                     "gcp_attributes": null,
>                     "idempotency_token": "",
>                     "init_scripts": null,
>                     "instance_pool_id": "",
>                     "is_single_node": false,
>                     "kind": "",
>                     "library": null,
>                     "node_type_id": "Standard_DS3_v2",
>                     "num_workers": 1,
>                     "policy_id": "",
>                     "runtime_engine": "",
>                     "single_user_name": "",
>                     "spark_conf": {},
>                     "spark_env_vars": {
>                       "ENV": "qa"
>                     },
>                     "spark_version": "15.4.x-scala2.12",
>                     "ssh_public_keys": null,
>                     "use_ml_runtime": false,
>                     "workload_type": null
>                   }
>                 ]
>               }
>             ],
>             "library": null,
>             "max_concurrent_runs": 1,
>             "max_retries": 0,
>             "min_retry_interval_millis": 0,
>             "name": "Daily - Silver Flights History Table Load",
>             "new_cluster": null,
>             "notebook_task": null,
>             "notification_settings": null,
>             "parameter": null,
>             "performance_target": null,
>             "pipeline_task": null,
>             "python_wheel_task": null,
>             "queue": [
>               {
>                 "enabled": true
>               }
>             ],
>             "retry_on_timeout": false,
>             "run_as": [
>               {
>                 "service_principal_name": "062101b2-1f0d-494f-8df9-7aa882a5abe2",
>                 "user_name": ""
>               }
>             ],
>             "run_job_task": null,
>             "schedule": [
>               {
>                 "pause_status": "UNPAUSED",
>                 "quartz_cron_expression": "0 0 * ? * * *",
>                 "timezone_id": "UTC"
>               }
>             ],
>             "spark_jar_task": null,
>             "spark_python_task": null,
>             "spark_submit_task": null,
>             "tags": {
>               "feature": "flight_info_history",
>               "mode": "History",
>               "run_mode": "daily"
>             },
>             "task": [
>               {
>                 "clean_rooms_notebook_task": null,
>                 "condition_task": null,
>                 "dashboard_task": null,
>                 "dbt_task": null,
>                 "depends_on": null,
>                 "description": "",
>                 "disable_auto_optimization": false,
>                 "email_notifications": [
>                   {
>                     "no_alert_for_skipped_runs": false,
>                     "on_duration_warning_threshold_exceeded": null,
>                     "on_failure": null,
>                     "on_start": null,
>                     "on_streaming_backlog_exceeded": null,
>                     "on_success": null
>                   }
>                 ],
>                 "environment_key": "",
>                 "existing_cluster_id": "",
>                 "for_each_task": null,
>                 "gen_ai_compute_task": null,
>                 "health": null,
>                 "job_cluster_key": "history_table_job_cluster",
>                 "library": null,
>                 "max_retries": 0,
>                 "min_retry_interval_millis": 0,
>                 "new_cluster": null,
>                 "notebook_task": [
>                   {
>                     "base_parameters": {},
>                     "notebook_path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files/src/silver/s... (25 more bytes)",
>                     "source": "WORKSPACE",
>                     "warehouse_id": ""
>                   }
>                 ],
>                 "notification_settings": null,
>                 "pipeline_task": null,
>                 "power_bi_task": null,
>                 "python_wheel_task": null,
>                 "retry_on_timeout": false,
>                 "run_if": "ALL_SUCCESS",
>                 "run_job_task": null,
>                 "spark_jar_task": null,
>                 "spark_python_task": null,
>                 "spark_submit_task": null,
>                 "sql_task": null,
>                 "task_key": "process_flights_info_history",
>                 "timeout_seconds": 0,
>                 "webhook_notifications": null
>               }
>             ],
>             "timeout_seconds": 0,
>             "timeouts": null,
>             "trigger": null,
>             "url": "https://adb-1803404685110667.7.azuredatabricks.net/#job/908143205118192",
>             "webhook_notifications": [
>               {
>                 "on_duration_warning_threshold_exceeded": null,
>                 "on_failure": null,
>                 "on_start": null,
>                 "on_streaming_backlog_exceeded": null,
>                 "on_success": null
>               }
>             ]
>           },
>           "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJ1cGRhdGUi... (52 more bytes)",
>           "schema_version": 2,
>           "sensitive_attributes": null
>         }
>       ],
>       "mode": "managed",
>       "name": "history_table_daily_job",
>       "provider": "provider[\"registry.terraform.io/databricks/databricks\"]",
>       "type": "databricks_job"
>     },
>     "... (5 additional elements)"
>   ],
>   "serial": 272,
>   "terraform_version": "1.5.5",
>   "version": 4
> }
< HTTP/2.0 200 OK pid=43254 mutator=terraform:state-push sdk=true
19:21:28 Debug: Apply pid=43254 mutator=terraform.Load
19:21:29 Debug: Apply pid=43254 mutator=apps.InterpolateVariables
19:21:29 Debug: Apply pid=43254 mutator=apps:UploadConfig
19:21:29 Debug: Apply pid=43254 mutator=metadata.Compute
19:21:29 Debug: Apply pid=43254 mutator=metadata.Upload
19:21:29 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/metadata.json?overwrite=true
> {
>   "config": {
>     "bundle": {
>       "git": {
>         "branch": "mkijewski/1377285/dab-schedule-notification-adjustments",
>         "bundle_root_path": ".",
>         "commit": "84e650d54af3e52a928aabdfac0aec2271041313",
>         "origin_url": "SITA-PSE@vs-ssh.visualstudio.com:v3/SITA-PSE/SITA%20Operations%20at%20Airports%20-%20AMS/TotalOp... (21 more bytes)"
>       }
>     },
>     "resources": {
>       "jobs": {
>         "history_table_daily_job": {
>           "id": "908143205118192",
>           "relative_path": "resources/live/stand_plan.yml"
>         },
>         "initial_stand_plan_job": {
>           "id": "608473532553833",
>           "relative_path": "resources/live/stand_plan.yml"
>         },
>         "operations_historical_daily_job": {
>           "id": "1095869539834236",
>           "relative_path": "resources/historical/operations_daily.yml"
>         },
>         "operations_historical_initial_job": {
>           "id": "53678199707665",
>           "relative_path": "resources/historical/operations_initial.yml"
>         },
>         "revenue_historical_initial_job": {
>           "id": "1067094201749014",
>           "relative_path": "resources/historical/revenue_initial.yml"
>         },
>         "stand_plan_live_continuous_job": {
>           "id": "733516530013482",
>           "relative_path": "resources/live/stand_plan.yml"
>         }
>       }
>     },
>     "workspace": {
>       "file_path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/files"
>     }
>   },
>   "version": 1
> }
< HTTP/2.0 200 OK pid=43254 mutator=metadata.Upload sdk=true
19:21:29 Debug: Apply pid=43254 mutator=lock:release
19:21:29 Info: Releasing deployment lock pid=43254 mutator=lock:release
19:21:29 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1749576045321,
<   "modified_at": 1749576045321,
<   "object_id": 521378236053684,
<   "object_type": "FILE",
<   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock",
<   "resource_id": "521378236053684"
< } pid=43254 mutator=lock:release sdk=true
19:21:29 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1749576045321,
<   "modified_at": 1749576045321,
<   "object_id": 521378236053684,
<   "object_type": "FILE",
<   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock",
<   "resource_id": "521378236053684"
< } pid=43254 mutator=lock:release sdk=true
19:21:30 Debug: GET /api/2.0/workspace/export?direct_download=true&path=/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock
< HTTP/2.0 200 OK
< {
<   "AcquisitionTime": "2025-06-10T19:20:44.7378+02:00",
<   "ID": "31317a4a-73df-4052-9778-f8bcd5153cc7",
<   "IsForced": false,
<   "User": "062101b2-1f0d-494f-8df9-7aa882a5abe2"
< } pid=43254 mutator=lock:release sdk=true
19:21:30 Debug: POST /api/2.0/workspace/delete
> {
>   "path": "/Workspace/Users/062101b2-1f0d-494f-8df9-7aa882a5abe2/.bundle/total_optimizer/state/deploy.lock"
> }
< HTTP/2.0 200 OK
< {} pid=43254 mutator=lock:release sdk=true
Error: terraform apply: exit status 1

Error: cannot update job: Duplicate (metric,op) pairs: (RUN_DURATION_SECONDS, GREATER_THAN) found in health rules

  with databricks_job.stand_plan_live_continuous_job,
  on bundle.tf.json line 426, in resource.databricks_job.stand_plan_live_continuous_job:
 426:       }



19:21:30 Debug: failed execution pid=43254 exit_code=1
19:21:30 Debug: POST /telemetry-ext
> {
>   "items": null,
>   "protoLogs": [
>     "{\"frontend_log_event_id\":\"416369fc-9fc6-4baf-9544-aa8c194e6da0\",\"entry\":{\"databricks_cli_log\":{\"... (583 more bytes)"
>   ],
>   "uploadTime": 1749576090255
> }
< HTTP/2.0 200 OK
< {
<   "errors": null,
<   "numProtoSuccess": 1,
<   "numRealtimeSuccess": 0,
<   "numSuccess": 0
< } pid=43254 sdk=true
19:21:30 Debug: All 1 logs uploaded successfully pid=43254

[pietern]

Thanks for creating the issue.

We do not merge duplicate health rules by key at this moment. You can work around this by introducing a variable (for example, stand_plan_run_duration) and overriding its value for the qa target.

We'll have to double-check with the team working on the jobs API whether or not it semantically makes sense to collapse and override duplicate metrics at the client-side, or whether duplicates will ever have valid semantics.