Runtime and orchestration
- Multi-pod Kubernetes step operator jobs: Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator. Set
pod_countinKubernetesStepOperatorSettingsto launch the step as an indexed job, making it easier to distribute command-style workloads across pods. PR #5104 - Local Docker sandbox: ZenML now includes a local Docker sandbox, plus a unified settings model for containerized sandboxes. Local sandbox workflows also support file upload and download, making it easier to test containerized ZenML behavior locally before moving to remote infrastructure. PR #5102
Deployment, security, and artifact integrity
- Enrollment keys from Kubernetes Secrets: The Helm chart now supports
server.pro.enrollmentKeySecretRef, so ZenML Pro enrollment keys can be injected from an existing Kubernetes Secret instead of being stored inline in Helm release values. The secret reference is applied consistently to the server, migration, and worker containers. PR #5123 - Cloudpickle artifact hash validation: Cloudpickle-materialized artifacts now store a SHA-256 hash when written and validate that hash before loading. This helps detect corrupted or unexpectedly modified artifact files earlier and fail with a clearer integrity signal. PR #5103
CLI and dashboard UX
- Select a stack during login:
zenml loginnow accepts--stack, allowing you to connect to a server and immediately set the active stack in one command. When used together with--project, ZenML applies the project first so the stack is resolved in the intended project context. PR #5125 - Improved secret value display: Secret values in the dashboard no longer expand indefinitely in the UI and are truncated for readability. A direct copy action is now available, making it easier to work with long secret values without disrupting the page layout. PR #1104
- Timeline updates for cancelled runs: The dashboard timeline now shows steps that were not started because a run was cancelled. The timeline filter also supports filtering for
Not Started, making cancelled or partially executed runs easier to inspect. PR #1108
Fixed
- Fewer SQLite lock failures for local stores: Local SQL stores backed by SQLite now wait up to 60 seconds for write locks instead of using Python鈥檚 5-second default. This reduces
sqlite3.OperationalError: database is lockedfailures when concurrent steps, such as mapped dynamic pipeline steps, finish and publish artifacts at the same time. PR #5096 - Artifact deletion on deployed servers: Artifact version deletion now correctly applies project scope when checking whether an artifact is unused. This fixes deletion through the API on remote deployed servers where the default project is disabled. PR #5100
- Complete exception tracebacks: Exception reporting now includes the full traceback lineage instead of only the final exception traceback. This makes chained failures easier to debug because the original cause is preserved alongside the final error. PR #5098
- Smarter Kubernetes dynamic pipeline retries: Kubernetes orchestrator pods are no longer retried when the pipeline run is already in a finished state that cannot be retried. This avoids unnecessary pod restarts that would immediately exit with
Run is already finished.PR #5107 - Correct scoped prefix lookups: Prefix-based lookups now keep scoped and default filters properly constrained when matching by ID or name prefix. This prevents unrelated entities from being returned, for example when looking up a schedule trigger by a prefix that does not match its name or ID. PR #5126
- Generic token provenance and service account auth: Generic and stack-deployment tokens now preserve the original service account or device provenance used to create them. This fixes authentication failures for pipeline workloads using server-issued JWTs for workspace-local service accounts, especially with external authentication enabled. PR #5127
- MLflow tracking with managed runtimes and Databricks: ZenML鈥檚 MLflow experiment tracker now behaves more reliably when managed runtimes inject MLflow environment variables such as
MLFLOW_RUN_ID. This avoids accidentally resuming an inherited run when ZenML needs to create its own run, improving compatibility with Databricks-backed MLflow setups. PR #5122 - API keys after service account adoption: Existing workspace-level API keys remain valid when a workspace service account is adopted by an organization-level service account with the same name. This allows teams to migrate service accounts gradually without interrupting workloads that still use older API keys. PR #5138
What's Changed
- Add version 0.96.1 to legacy docs by @github-actions[bot] in #5093
- Add 0.96.2 to the migration tests by @github-actions[bot] in #5095
- Bump the minor-and-patch group across 1 directory with 9 updates by @dependabot[bot] in #5076
- Raise sqlite busy timeout for local stores by @htahir1 in #5096
- Fix missing project scope when deleting artifact versions by @schustmi in #5100
- Fix exception info cutoff by @schustmi in #5098
- Fix flag syntax in
service-connector registerhelp example by @strickvl in #5101 - Stop format_name_template from mutating the caller's substitutions dict by @chuenchen309 in #5088
- Count an installed prerelease as installed by @chuenchen309 in #5089
- Bump the minor-and-patch group with 6 updates by @dependabot[bot] in #5105
- Docker sandbox by @schustmi in #5102
- Bump the minor-and-patch group with 6 updates by @dependabot[bot] in #5115
- Bump actions/setup-python from 6.3.0 to 7.0.0 by @dependabot[bot] in #5116
- Bump astral-sh/setup-uv from 8.3.2 to 9.0.0 by @dependabot[bot] in #5117
- Examples linting/formatting due to the new ruff version by @bcdurak in #5130
- Support zenml enrollment key as a kubernetes secrets by @amitvikramraj in #5123
- Upgrading the
mypydependency by @bcdurak in #5131 - Validate cloudpickle content hash by @schustmi in #5103
- Fix clean_requirements dropping > and != version specifiers by @chuenchen309 in #5085
- Improved Kubernetes dynamic pipeline retries by @schustmi in #5107
- Ability to select active stack upon login by @bcdurak in #5125
- Add multi-node support for Kubernetes step operator by @schustmi in #5104
- Fix scoped filters in prefix lookups by @bcdurak in #5126
- Scope generic tokens to the original service account or device used to create them by @stefannica in #5127
- MLFlow Experiment Tracker Databricks Improvements by @bcdurak in #5122
- Fix broken trigger filtering by associate ids by @Json-Andriopoulos in #5132
- Fix flaky schedule deletion test by @Json-Andriopoulos in #5135
- Allow old API keys to be used for adopted service accounts by @stefannica in #5138
- Pin JAX to avoid TensorFlow ml-dtypes conflict by @Json-Andriopoulos in #5137
- Fix Haystack agent example for Haystack 3 by @strickvl in #5142
- Prepare release 0.96.3 by @github-actions[bot] in #5143
New Contributors
- @chuenchen309 made their first contribution in #5088
Full Changelog: 0.96.2...0.96.3