Skip to content

v1.5.0 — Asset Library framework

Choose a tag to compare

View all tags
@vmariiechko vmariiechko released this 25 Apr 13:23
· 44 commits to main since this release
v1.5.0
bdb7f80

Highlights

This release introduces the Asset Library, a new way to install individual Databricks artifacts into any bundle on demand, plus the first asset shipping with it: sdp-checkpoint-recovery. Existing users see no change to the core databricks bundle init flow.

What's new

Asset Library framework

Standalone sub-templates under assets/<name>/, installable into any existing Databricks bundle:

databricks bundle init https://github.com/vmariiechko/databricks-bundle-template \
  --template-dir assets/<asset-name>

Each asset is self-contained (own schema, own README, own tests) and composes additively. The CLI errors cleanly on file collisions; assets never modify existing files.

First asset: sdp-checkpoint-recovery

Recovers a Lakeflow Spark Declarative Pipeline from DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE after a source table has been dropped and recreated, without clearing target table data. Two scripts ship together:

  • sdp_reset_checkpoint_local.py — runs from your machine via the Databricks CLI profile chain. Supports --dry-run (mapped to validate_only=True) and validates flow FQN format locally before any API call.
  • sdp_reset_checkpoint_workspace.py — uploads as a Databricks notebook. Parametrized with dbutils.widgets for pipeline_id, flows, and dry_run.

Both scripts try the native SDK signature (databricks-sdk>=0.100) and fall back to a raw REST call via WorkspaceClient.api_client.do on older runtimes (notably Databricks Runtime serverless, which bundles its own SDK).

Documentation and tooling

  • Pipeline terminology corrected to "Lakeflow Spark Declarative Pipelines" / "SDP" across all generated docs and template content. Outdated "LDP" / "Lakeflow Declarative Pipelines" usages replaced; relevant docs.databricks.com URLs updated.
  • PR and issue templates updated to recognize asset-related changes.
  • Ruff configuration added to pyproject.toml with Google docstring convention and a curated rule set.

Upgrade

No action required. The default databricks bundle init <repo> flow is unchanged. To install an asset, use --template-dir assets/<name> against your existing bundle root.

Breaking changes

None.

Pointers

Assets 2
Loading