Pipelines do not use patched wheel when dynamic artifact version is enabled

[liahagan]

Describe the issue

Pipeline environment dependencies do not automatically use the patched wheel when using artifacts.<name>.dynamic_version: true. This works perfectly fine for dependencies in jobs and the documentation does not mention this being a job-specific feature.

Configuration

Prerequisites

• Databricks CLI
• uv

Project structure

.
├── databricks.yml
├── pyproject.toml
└── src
    ├── etl
    │   └── transformations
    │       └── sample_transformation.py
    ├── notebooks
    │   └── sample_notebook.ipynb
    └── sample_package
        └── __init__.py

Above is a basic project structure that is the basis for the configuration below.

sample_transformation.py, sample_notebook.ipynb and __init__.py can be blank files.

databricks.yml

bundle:
  name: dynamic_version_test
  engine: direct

artifacts:
  python_artifact:
    type: whl
    build: uv build --wheel
    dynamic_version: true

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: <workspace-host>

resources:
  jobs:
    sample_job:
      name: sample_job
      tasks:
        - task_key: sample_task
          notebook_task:
            notebook_path: ./src/notebooks/sample_notebook.ipynb
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: "5"
            dependencies:
              - ./dist/*.whl
  pipelines:
    sample_pipeline:
      name: sample_pipeline
      catalog: main
      schema: default
      serverless: true
      root_path: ./src/etl
      libraries:
        - glob:
            include: ./src/etl/transformations/**
      environment:
        dependencies:
          - ./dist/*.whl

pyproject.toml

[project]
name = "sample_package"
version = "0.0.1"
authors = [{ name = "name@example.com" }]
requires-python = ">=3.10,<3.13"
dependencies = []

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Steps to reproduce the behavior

1. Run databricks bundle deploy
2. Run databricks bundle open sample_job or find the job manually in the workspace
3. Confirm that the job's "default" environment is using a patched wheel on the format sample_package-0.0.1+1234567890987654321-py3-none-any.whl
4. Run databricks bundle open sample_pipeline or find the pipeline manually in the workspace
5. Confirm that the pipeline's environment is referencing a non-patched wheel on the format sample_package-0.0.1-py3-none-any.whl

Expected Behavior

Both the job and the pipeline environments should use the same wheel with the patched name.

Actual Behavior

Only the job is using the patched wheel.

OS and CLI version

• OS: macOS
• Databricks CLI version: v0.299.2

Is this a regression?

No, I don't think so.

Debug Logs

17:03:49 Info: start pid=77713 version=0.299.2 args="databricks, bundle, deploy, -t, dev, --debug"
...
17:03:49 Debug: Apply pid=77713 mutator=ApplyArtifactsDynamicVersion
17:03:49 Debug: Apply pid=77713 mutator=artifacts.Prepare
17:03:49 Debug: artifacts block is defined, skipping auto-detecting pid=77713 mutator=artifacts.Prepare
...
17:03:49 Debug: Apply pid=77713 mutator=fast_validate(readonly)
17:03:49 Debug: ApplyParallel pid=77713 mutator=fast_validate(readonly) mutator=validate:job_cluster_key_defined
17:03:49 Debug: ApplyParallel pid=77713 mutator=fast_validate(readonly) mutator=validate:job_task_cluster_spec
17:03:49 Debug: ApplyParallel pid=77713 mutator=fast_validate(readonly) mutator=validate:artifact_paths
17:03:49 Info: Phase: build pid=77713
17:03:49 Debug: Apply pid=77713 mutator=scripts.prebuild
17:03:49 Debug: No script defined for prebuild, skipping pid=77713 mutator=scripts.prebuild
17:03:49 Debug: Apply pid=77713 mutator=artifacts.Build
Building python_artifact...
17:03:50 Debug: build output for python_artifact:
Building wheel...
Successfully built dist/sample_package-0.0.1-py3-none-any.whl
 pid=77713 mutator=artifacts.Build
17:03:50 Debug: Apply pid=77713 mutator=artifacts.Build mutator=expandGlobs
17:03:50 Info: Patched wheel (built) /Users/path/to/project/dynamic_version_test/dist/sample_package-0.0.1-py3-none-any.whl -> /Users/path/to/project/dynamic_version_test/.databricks/bundle/dev/patched_wheels/python_artifact_sample_package/sample_package-0.0.1+1779289430798249639-py3-none-any.whl pid=77713 mutator=artifacts.Build
17:03:50 Debug: Patching ind=0 artifactName=python_artifact Source=/Users/path/to/project/dynamic_version_test/dist/sample_package-0.0.1-py3-none-any.whl patchedWheel=/Users/path/to/project/dynamic_version_test/.databricks/bundle/dev/patched_wheels/python_artifact_sample_package/sample_package-0.0.1+1779289430798249639-py3-none-any.whl pid=77713 mutator=artifacts.Build
17:03:50 Debug: Apply pid=77713 mutator=scripts.postbuild
17:03:50 Debug: No script defined for postbuild, skipping pid=77713 mutator=scripts.postbuild
17:03:50 Debug: Apply pid=77713 mutator=ResolveVariableReferences
17:03:50 Debug: Apply pid=77713 mutator=ResolveVariableReferences(resources)
17:03:50 Debug: Apply pid=77713 mutator=libraries.ExpandGlobReferences
17:03:50 Debug: Apply pid=77713 mutator=CheckForSameNameLibraries
17:03:50 Debug: Apply pid=77713 mutator=SwitchToPatchedWheels
17:03:50 Debug: Updating resources.jobs.sample_job.environments[0].spec.dependencies[0] from dist/sample_package-0.0.1-py3-none-any.whl to .databricks/bundle/dev/patched_wheels/python_artifact_sample_package/sample_package-0.0.1+1779289430798249639-py3-none-any.whl pid=77713 mutator=SwitchToPatchedWheels
17:03:50 Debug: Apply pid=77713 mutator=TransformWheelTask
...
17:03:51 Debug: POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/artifacts/.interna... (1 more bytes)"
> }
< HTTP/2.0 200 OK
< {} pid=77713 mutator=artifacts.CleanUp sdk=true
17:03:51 Debug: Apply pid=77713 mutator=libraries.Upload
Uploading .databricks/bundle/dev/patched_wheels/python_artifact_sample_package/sample_package-0.0.1+1779289430798249639-py3-none-any.whl...
Uploading dist/sample_package-0.0.1-py3-none-any.whl...
17:03:51 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/artifacts/.internal/sample_package-0.0.1-py3-none-any.whl?overwrite=true
> PBPsample_package/__init__.pyPBP`ث�q|'sample_p... (917 more bytes)
< HTTP/2.0 200 OK pid=77713 mutator=libraries.Upload sdk=true
17:03:51 Info: Upload succeeded pid=77713 mutator=libraries.Upload
17:03:51 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/artifacts/.internal/sample_package-0.0.1+1779289430798249639-py3-none-any.whl?overwrite=true
> BP    sample_package/__init__.pyUT6^��P... (1223 more bytes)
< HTTP/2.0 200 OK pid=77713 mutator=libraries.Upload sdk=true
17:03:51 Info: Upload succeeded pid=77713 mutator=libraries.Upload
...
Deploying resources...
17:03:53 Debug: POST /api/2.2/jobs/create
> {
>   "deployment": {
>     "kind": "BUNDLE",
>     "metadata_file_path": "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/state/metadata.jso... (1 more bytes)"
>   },
>   "edit_mode": "UI_LOCKED",
>   "environments": [
>     {
>       "environment_key": "default",
>       "spec": {
>         "dependencies": [
>           "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/artifacts/.interna... (59 more bytes)"
>         ],
>         "environment_version": "5"
>       }
>     }
>   ],
>   "format": "MULTI_TASK",
>   "max_concurrent_runs": 4,
>   "name": "[dev username] sample_job",
>   "queue": {
>     "enabled": true
>   },
>   "tags": {
>     "dev": "username"
>   },
>   "tasks": [
>     {
>       "environment_key": "default",
>       "notebook_task": {
>         "notebook_path": "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/files/src/notebook... (17 more bytes)"
>       },
>       "task_key": "sample_task"
>     }
>   ]
> }
< HTTP/2.0 200 OK
< {
<   "job_id": 1081705303080705
< } pid=77713 sdk=true
17:03:53 Info: Created resources.jobs.sample_job id="1081705303080705" pid=77713
17:03:54 Debug: POST /api/2.0/pipelines
> {
>   "catalog": "main",
>   "channel": "CURRENT",
>   "deployment": {
>     "kind": "BUNDLE",
>     "metadata_file_path": "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/state/metadata.jso... (1 more bytes)"
>   },
>   "development": true,
>   "edition": "ADVANCED",
>   "environment": {
>     "dependencies": [
>       "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/artifacts/.interna... (39 more bytes)"
>     ]
>   },
>   "libraries": [
>     {
>       "glob": {
>         "include": "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/files/src/etl/tran... (14 more bytes)"
>       }
>     }
>   ],
>   "name": "[dev username] sample_pipeline",
>   "root_path": "/Workspace/Users/user@example.com/.bundle/dynamic_version_test/dev/files/src/etl",
>   "schema": "default",
>   "serverless": true,
>   "tags": {
>     "dev": "username"
>   }
> }
< HTTP/2.0 200 OK
< {
<   "pipeline_id": "80358402-fc8a-42a6-bbde-e9686472e4b2"
< } pid=77713 sdk=true
17:03:54 Info: Created resources.pipelines.sample_pipeline id="80358402-fc8a-42a6-bbde-e9686472e4b2" pid=77713
...
Deployment complete!