Translate code_source git and include_paths in convert-to-dabs by vinchenzo-db · Pull Request #6495 · databricks/cli

vinchenzo-db · 2026-09-02T17:59:57Z

Summary

experimental air convert-to-dabs previously rejected a code_source.snapshot that
pinned a git ref or narrowed to include_paths — it errored and told the user to work
around it. Now it translates them into a tgz artifact (added in #6428): DABs builds
the tarball from the git ref / include subset at deploy, and code_source_path points
at the built tarball.

This closes the conversion gaps for the two cases convert used to refuse.

Mapping

For a snapshot that pins a git ref and/or include_paths, convert emits a tgz
artifact and points code_source_path at its output:

snapshot.git.{branch,commit} → artifact git.{branch,commit}
snapshot.include_paths → artifact include

Archive entries must nest under the code directory's basename (the runtime extracts to
/databricks/code_source/<dir>), so convert emits path = the code dir's parent and
include = basename-prefixed subpaths:

root_path: ./src → path: ".", include: ["src"]
root_path: ./src, include_paths: [foo] → path: ".", include: ["src/foo"]

A plain snapshot (no git / include_paths) is unchanged: code_source_path stays the
source directory, packaged at deploy. remote_volume is still rejected — a per-source
volume isn't representable in a bundle (set workspace.artifact_path instead).

Testing

Unit tests assert the emitted artifact (type, path, git/include, files) and
the rewritten code_source_path for both the git-ref and include_paths cases; the
plain-snapshot path is covered by the existing convert acceptance golden.

E2E test:
TEST INCLUDE
Setup

$ rm -rf /tmp/pr2-include && mkdir -p /tmp/pr2-include/src && cd /tmp/pr2-include
printf 'print("include ok")\n' > src/train.py
printf 'python train.py\n'      > src/command.sh
printf 'excluded\n'             > src/extra.txt
cat > train.yaml <<'YAML'
experiment_name: pr2_include_demo
command: python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
code_source:
  type: snapshot
  snapshot:
    root_path: ./src
    include_paths: [train.py, command.sh]
YAML

convert to dabs

$ /tmp/cli-pr2 experimental air convert-to-dabs train.yaml
Wrote a Databricks Asset Bundle to .:
  databricks.yml
  generated_artifacts/training_config.yaml
  generated_artifacts/command.sh

To deploy and run this workload as a bundle:
  1. /tmp/cli-pr2 bundle validate
  2. /tmp/cli-pr2 bundle deploy
  3. /tmp/cli-pr2 bundle run pr2_include_demo --no-wait

bundle deploy uploads the code source and launch scripts automatically.
To see what it deployed and where: /tmp/cli-pr2 bundle summary

Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
persistent job that is not garbage-collected. When you are done, remove the
job and its uploaded files with:
  /tmp/cli-pr2 bundle destroy

investigate artifacts:

$ cat databricks.yml
cat generated_artifacts/command.sh
bundle:
  name: pr2_include_demo
sync:
  paths:
    - generated_artifacts
artifacts:
  code_source:
    type: tgz
    path: .
    include:
      - src/train.py
      - src/command.sh
    files:
      - source: ./dist/code_source.tgz
targets:
  dev:
    mode: development
    default: true
resources:
  jobs:
    pr2_include_demo:
      name: pr2_include_demo
      tasks:
        - task_key: pr2_include_demo
          environment_key: default
          max_retries: 3
          ai_runtime_task:
            experiment: pr2_include_demo
            deployments:
              - command_path: ./generated_artifacts/command.sh
                compute:
                  accelerator_type: GPU_1xA10
                  accelerator_count: 1
            code_source_path: ./dist/code_source.tgz
      environments:
        - environment_key: default
          spec:
            environment_version: "4"
cd /databricks/code_source/src
python train.py%

deploy:

$ /tmp/cli-pr2 bundle validate && /tmp/cli-pr2 bundle deploy
Name: pr2_include_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_include_demo/dev

Validation OK!
Building code_source...
Uploading dist/code_source.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/pr2_include_demo/dev/files...
Files: 2 uploaded, 0 deleted
Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:29]
$ tar tzf dist/code_source.tgz
src/command.sh
src/train.py

run:

# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:34]
$ /tmp/cli-pr2 bundle summary
Name: pr2_include_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_include_demo/dev
Resources:
  Jobs:
    pr2_include_demo:
      Name: [dev v_chen] pr2_include_demo
      URL:  https://e2-dogfood.staging.cloud.databricks.com/jobs/212600874211444?w=6051921418418893
# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:39]
$ /tmp/cli-pr2 bundle run pr2_include_demo --no-wait
Run URL: https://e2-dogfood.staging.cloud.databricks.com/jobs/212600874211444/runs/882449706897243?o=6051921418418893

TEST GIT
Setup dirty git:

# v.chen at ip-10-90-20-219 in /tmp/pr2-include (git:) [19:22:48]
$ rm -rf /tmp/pr2-git && mkdir -p /tmp/pr2-git/src && cd /tmp/pr2-git
git init -q -b main
printf 'print("git v1 committed")\n' > src/train.py
printf 'python train.py\n'           > src/command.sh
git add -A && git commit -qm "v1"
Databricks pre-commit Git Hook V2.5.0
Running secret scanning on changes staged for commit.
secret-scan hook completed in 74 ms
Unknown project name: None, skipping linting.
pre-commit-total hook completed in 90 ms
Databricks commit-msg Git Hook V2.5.0
Running secret scanning on commit message.
secret-scan hook completed in 23 ms
commit-msg-total hook completed in 29 ms
# v.chen at ip-10-90-20-219 in /tmp/pr2-git (git:main) [19:26:30]
$ printf 'print("v2 UNCOMMITTED - should NOT appear")\n' > src/train.py

cat > train.yaml <<'YAML'
experiment_name: pr2_git_demo
command: python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
code_source:
  type: snapshot
  snapshot:
    root_path: ./src
    git: {branch: main}
YAML

convert to dabs and investigate artifact:

$ /tmp/cli-pr2 experimental air convert-to-dabs train.yaml
cat databricks.yml
Wrote a Databricks Asset Bundle to .:
  databricks.yml
  generated_artifacts/training_config.yaml
  generated_artifacts/command.sh

To deploy and run this workload as a bundle:
  1. /tmp/cli-pr2 bundle validate
  2. /tmp/cli-pr2 bundle deploy
  3. /tmp/cli-pr2 bundle run pr2_git_demo --no-wait

bundle deploy uploads the code source and launch scripts automatically.
To see what it deployed and where: /tmp/cli-pr2 bundle summary

Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
persistent job that is not garbage-collected. When you are done, remove the
job and its uploaded files with:
  /tmp/cli-pr2 bundle destroy
bundle:
  name: pr2_git_demo
sync:
  paths:
    - generated_artifacts
artifacts:
  code_source:
    type: tgz
    path: .
    git:
      branch: main
    include:
      - src
    files:
      - source: ./dist/code_source.tgz
targets:
  dev:
    mode: development
    default: true
resources:
  jobs:
    pr2_git_demo:
      name: pr2_git_demo
      tasks:
        - task_key: pr2_git_demo
          environment_key: default
          max_retries: 3
          ai_runtime_task:
            experiment: pr2_git_demo
            deployments:
              - command_path: ./generated_artifacts/command.sh
                compute:
                  accelerator_type: GPU_1xA10
                  accelerator_count: 1
            code_source_path: ./dist/code_source.tgz
      environments:
        - environment_key: default
          spec:
            environment_version: "4"

deploy:

$ /tmp/cli-pr2 bundle validate && /tmp/cli-pr2 bundle deploy
Name: pr2_git_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_git_demo/dev

Validation OK!
Building code_source...
Uploading dist/code_source.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/pr2_git_demo/dev/files...
Created jobs.pr2_git_demo
Files: 2 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

Prove we properly exclude dirty git:

# v.chen at ip-10-90-20-219 in /tmp/pr2-git (git:main) [19:28:02]
$ mkdir -p /tmp/pr2-git-check && tar xzf dist/code_source.tgz -C /tmp/pr2-git-check
grep -R "committed" /tmp/pr2-git-check/src/train.py    # -> "git v1 committed"
grep -R "UNCOMMITTED" /tmp/pr2-git-check/src/train.py || echo "OK: uncommitted change correctly excluded"
print("git v1 committed")
OK: uncommitted change correctly excluded

run:

$ /tmp/cli-pr2 bundle summary
Name: pr2_git_demo
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/pr2_git_demo/dev
Resources:
  Jobs:
    pr2_git_demo:
      Name: [dev v_chen] pr2_git_demo
      URL:  https://e2-dogfood.staging.cloud.databricks.com/jobs/235795448342543?w=6051921418418893
# v.chen at ip-10-90-20-219 in /tmp/pr2-git (git:main) [19:28:47]
$ /tmp/cli-pr2 bundle run pr2_git_demo --no-wait
Run URL: https://e2-dogfood.staging.cloud.databricks.com/jobs/235795448342543/runs/137864060324526?o=6051921418418893

Instead of rejecting a git-pinned or include_paths-narrowed code_source, convert now emits a `tgz` artifact (the DABs artifact snapshotter): DABs builds the tarball from the git ref / include subset at deploy, and code_source_path points at the built tarball. Plain snapshots are unchanged — code_source_path stays the source directory, packaged by the deploy-time aicode mutator. - snapshot.git.{branch,commit} -> artifact git.{branch,commit} - snapshot.include_paths -> artifact include (code-source-root-relative, matching air CLI semantics) - remote_volume is still rejected (not representable per code source). This unblocks full-fidelity conversion of the cases convert previously refused. Stacked on the artifacts tgz/git/include primitive (#6428). Co-authored-by: Isaac <no-reply@databricks.com>

Match the artifact snapshotter's path-relative entry naming: emit the artifact `path` as the code dir's parent and `include` as basename-prefixed subpaths, so archive entries come out as "<basename>/..." — the layout the runtime extracts to /databricks/code_source/<dir>. - root_path ./src, no include_paths -> path ".", include ["src"] - root_path ./src, include_paths [x] -> path ".", include ["src/x"] Co-authored-by: Isaac <no-reply@databricks.com>

AI Runtime training runs are long, so the generated next-steps suggest `bundle run <job> --no-wait` — submit and return the terminal rather than blocking on a streamed run. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

The launcher runs command.sh from the launch dir, but code_source extracts to /databricks/code_source/<dir>, so a bare command like `python train.py` couldn't find its files. Prepend a cd into that dir to the generated command.sh so relative paths resolve against the code — matching how the command reads under `air run`. Convert-only; native `air run` is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>

ben-hansen-db · 2026-09-02T21:56:34Z

 // environments[] spec, which the runtime installs from directly; no requirements.yaml
 // is emitted.
+//
+// When the snapshot pins a git ref or narrows to include_paths, convert instead emits


if there's git: but not branch or commit what is the behavior? Do we error in CLI? Does translation fail?

It would fail when user runs convert-to-dabs before any sort of dab command

throws git: must specify either 'branch' or 'commit'

ben-hansen-db · 2026-09-02T21:57:14Z

 const generatedArtifactsDir = "generated_artifacts"

+// codeSourceArtifactKey names the `tgz` artifact convert emits for a git/include
+// snapshot. codeSourceTgzArtifact is where DABs writes the built tarball — kept out of


can we make sure that include_paths tests directories? That is main use case not individual files

ben-hansen-db · 2026-09-02T22:01:10Z

+
+// cdIntoCodeSource prepends a `cd` into the extracted code dir to the generated
+// command.sh, so the user's command runs against the code rather than the launch dir.
+func cdIntoCodeSource(artifacts []uploadItem, dirName string) {


I am confused by this. Why is it needed?

Not clear to me what this is fixing

The launcher runs command.sh from the launch directory, but code_source is extracted to /databricks/code_source/<dir>. If no cd, a relative command like python train.py runs from the launch dir and can't find the file.

I think running python train.py is what most users would assume and they would be confused why they have to cd into /databricks/code_source/<dir> before running.

^ This was a requirement raised by Pieter in the previous pr because we are making tgz a generic artifact type, we need to follow DABs semantics for what they do with the other artifact types which I agree with. This comment

isn't that what we do currently in CLI? Why is there a difference with DABs

ben-hansen-db · 2026-09-02T22:02:20Z

+	if snap == nil || (snap.Git == nil && len(snap.IncludePaths) == 0) {
+		return nil
+	}
+	codeDirRel := strings.TrimPrefix(codeDirPath, "./")


from claude:

Issue: With root_path: ., codeDirRel is ".", so path.Base → "." and path.Dir → ".", yielding include: ["."] and an injected cd /databricks/code_source/..
Fix: Guard/normalize the root_path == "." case or confirm it's unreachable.

Guard convert against a git/include_paths snapshot whose root_path resolves to the bundle root: there is no basename to nest the archive under, and an include rooted at "." would sweep the bundle's own generated files into the tarball. Point root_path at a subdirectory instead. Also add a test covering directory include_paths (the primary use case). Co-authored-by: Isaac <no-reply@databricks.com>

The AIR runtime exports $CODE_SOURCE_PATH (and symlinks it under $HOME) but never cds into the code source — the user's command owns its own cd, as every air example does. So a working air command already cds via $CODE_SOURCE_PATH, and injecting cd /databricks/code_source/<dir> only rescued commands that would also fail under air run, making convert diverge from it. Copy the command verbatim instead. Verified end-to-end on e2-dogfood: a cd $CODE_SOURCE_PATH command runs to SUCCESS with no injection. Co-authored-by: Isaac <no-reply@databricks.com>

The fixture command now cds via $CODE_SOURCE_PATH (a real air command), and command.sh is emitted verbatim with no injected cd — matching the runtime contract. Regenerates the golden. Co-authored-by: Isaac <no-reply@databricks.com>

vinchenzo-db and others added 5 commits September 2, 2026 16:54

Suggest bundle run --no-wait after convert

e376581

AI Runtime training runs are long, so the generated next-steps suggest `bundle run <job> --no-wait` — submit and return the terminal rather than blocking on a streamed run. Co-authored-by: Isaac <no-reply@databricks.com>

Drop comment on the suggested run step

fc3c493

Co-authored-by: Isaac <no-reply@databricks.com>

vinchenzo-db requested a review from ben-hansen-db September 2, 2026 18:59

ben-hansen-db reviewed Sep 2, 2026

View reviewed changes

ben-hansen-db approved these changes Sep 2, 2026

View reviewed changes

vinchenzo-db and others added 2 commits September 2, 2026 23:37

vinchenzo-db added this pull request to the merge queue Sep 3, 2026

Merged via the queue into main with commit a6a3734 Sep 3, 2026
32 checks passed

vinchenzo-db deleted the vchen/air-convert-git-include branch September 3, 2026 00:54

Navigation Menu

Uh oh!

Translate code_source git and include_paths in convert-to-dabs - #6495

Translate code_source git and include_paths in convert-to-dabs#6495
vinchenzo-db merged 8 commits into
mainfrom
vchen/air-convert-git-include

vinchenzo-db commented Sep 2, 2026 •

edited

Loading

Uh oh!

ben-hansen-db Sep 2, 2026

Uh oh!

vinchenzo-db Sep 2, 2026

Uh oh!

vinchenzo-db Sep 2, 2026

Uh oh!

ben-hansen-db Sep 2, 2026

Uh oh!

ben-hansen-db Sep 2, 2026

Uh oh!

ben-hansen-db Sep 2, 2026

Uh oh!

vinchenzo-db Sep 2, 2026 •

edited

Loading

Uh oh!

ben-hansen-db Sep 2, 2026 •

edited

Loading

Uh oh!

ben-hansen-db Sep 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

vinchenzo-db commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Mapping

Testing

Uh oh!

ben-hansen-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vinchenzo-db commented Sep 2, 2026 •

edited

Loading

vinchenzo-db Sep 2, 2026 •

edited

Loading

ben-hansen-db Sep 2, 2026 •

edited

Loading