Skip to content

v3.0.0

Choose a tag to compare

@erinaldidb erinaldidb released this 22 May 22:19

Pixels 3.0.0

Pixels 3.0.0 is the GA release of the next-generation Databricks medical-imaging
accelerator. It builds on 3.0.0-rc.1 (and the nine release candidates that followed)
and ships the complete repository reorganization, a fully-declarative Databricks
Asset Bundle (DAB) installer
with a 14-task DAG, and hardened Lakebase, DICOMweb
gateway
, OHIF viewer, Vista3D model serving, Genie, and Lakeview
dashboard
deployment, alongside a long list of stability, performance, and
documentation improvements.

For the feature highlights introduced over 2.2.0 (DICOMweb Gateway, Lakebase
integration, Variant data type, DICOM Redactor, Genie automation, OHIF v3.11.1,
MONAI 1.5.2, etc.), see the v3.0.0-rc.1 release notes.

This changelog covers every change since 3.0.0-rc.1 — 208 commits, 9 release
candidates, 14 merged pull requests.


Highlights since 3.0.0-rc.1

  • Repository reorganization — Python library moved to src/dbx/pixels/ (src layout),
    apps consolidated under apps/, model assets under models/monai/, install task
    notebooks under install/, demos under notebooks/, docs under docs/.
  • One-click DAB installer — a single make deploy + databricks bundle run pixels_install
    provisions the entire stack: UC schema/volume/tables, Lakebase Postgres instance with
    Reverse ETL sync, DICOMweb gateway and OHIF viewer apps, Vista3D model serving endpoint,
    Genie space, Lakeview dashboard, STOW processor job, and grants.
  • 14-task install job with explicit dependencies, retry-free execution, and a final
    validate_install task that verifies every surface end-to-end.
  • Cross-workspace, cross-catalog ready — every notebook is parameterized via widgets
    fed by databricks.yml variables. Default catalog is now main, schema is pixels.
  • Idempotent deployment — every install task can be re-run without breaking the stack
    (schemas, endpoints, apps, Lakebase grants, VS index, Genie space).
  • Lakebase robustness — DML grants for app service principals, role-propagation race
    fix, deduped instance_paths_vw to avoid PK violations on Reverse ETL sync, automated
    synced-table creation via API, full connectivity/sync/grants/writes tests in
    validate_install.
  • Model serving robustness — split task 03 into 03a register / 03b deploy /
    03c validate; endpoint creation and update are fully idempotent; predict calls use
    the REST API with explicit timeouts; stale-config detection across schema/model
    renames; champion alias on registration.
  • OHIF mixed-content fix on HTTPS Databricks Apps origins; viewer-side reverse proxy
    for MONAI info / infer calls (cross-app OBO with request-correlated logging).
  • Wheel + OHIF served from UC Volume — deploy_prep uploads databricks_pixels-*.whl
    and ohif.tar.gz to /Volumes/<catalog>/<schema>/<volume>/. Apps pip install the
    wheel from the Volume at startup and extract OHIF in-container, removing large
    .wasm files from DAB sync.
  • STOW-RS hardening — gateway derives stow_operations table dynamically from the
    configured Pixels table, normalizes cache keys to composite (study, series, sop)
    instance UIDs, and ships unit tests for the handler.
  • Performance — mapInPandas-based parallel unzip for large archives, concurrent
    I/O in DicomMetaExtractor, configurable worker counts.
  • Documentation — docs/INSTALL.md rewritten for the 14-task DAG, prerequisites,
    and scale_to_zero_enabled. New CLAUDE.md capturing project structure, build
    commands, conventions, and the mandatory post-install integration-test suite.

Repository reorganization (the reorg branch)

The repository was restructured in seven phases to clearly separate library code,
install notebooks, apps, models, and demo material:

  • Phase 1 — consolidate install tasks into install/ (7d7f4d6)
  • Phase 2 — extract apps to apps/, move shared app code to dbx/pixels/common/ (3dcef06)
  • Phase 3 — move the library to src/dbx/pixels/ (src layout); update setup.py (0c29e33)
  • Phase 5 — move the model to models/vista3d/, demos to notebooks/, docs to docs/ (4e10863)
  • Phase 6 + 7 — style fixes, bundle validation, refresh README.md and CLAUDE.md links (c0080b0)
  • Rename models/vista3d/ → models/monai/ to reflect broader scope (MONAI Label + Vista3D bundle) (13e21bd)
  • Update CLAUDE.md for the reorg; fix pre-commit paths for the src layout (b50130a)
  • Restore OHIF in package resources for wheel distribution (a647b7f)
  • Decouple Vista3D model from DAB sync; move plan docs to plans/ (80ba431)

make dev, make build, make deploy, make test, make style, make check, and
make clean continue to work with the new layout. The Asset Bundle is configured to
sync the right subset of the new tree and to include dist/*.whl.

DAB installer & install job

A complete DAB installer replaces ad-hoc deployment notebooks. The install job is a
14-task DAG that provisions everything top-to-bottom.

Structure

  • Add resources/install-job.yml, resources/dashboard.yml, resources/unity-catalog.yml DAB resources (9b7e40e)
  • Split task 03 into 03a register-model, 03b deploy-endpoint, 03c validate-model (09d46b6)
  • Split deploy-apps into 5 install tasks: deploy_prep, deploy_lakebase,
    deploy_gateway, deploy_viewer, deploy_grants (1a0965b)
  • Drop numeric task prefixes from task names (1a0965b)
  • Add 10_validate install task to verify every Pixels surface (73ea589)
  • Add 09-post-install-update task for dashboard params + app thumbnail (3685dec)
  • Add stow_processor task — job creation + perms (65d2f76)
  • Move init-schema into the DAB; dedup Genie space; fix 03a restart (65d2f76)
  • Add install/uninstall.py to tear down a Pixels deployment (2c62bf9)
  • Retry validation up to 12× at 3-min intervals until all checks pass (3e39948)
  • Disable per-task retries (max_retries: 0) so failures surface immediately (bf645fc)
  • Increase serverless compute environment from 1 to 4 (d4f2860)
  • Wire lakebase_instance_name through DAB → job → notebook widgets (6e35338)
  • Add scale_to_zero_enabled variable; pin deps; reorder job tasks (d1650c0)
  • Detect stale serving-endpoint config across schema/model name changes (4bf62ef)
  • Skip trashed Pixels dashboards in post_install_update (51ed6cf)
  • Split app-shell creation from code deploy to fix fresh-install grants race (9a48438)
  • Template dashboard params at make deploy time, not via post-install patch (ce29876)
  • Broaden pre-commit E2E test rule to apply to every commit (f770544)
  • Address PR #211 review feedback for 3.0.0 release prep (ecbbbd7)
  • Update Makefile and requirements.txt for improved environment setup (7cc7041)

Configuration & parameterization

  • Parameterize catalog references for cross-workspace installs (22968df)
  • Parameterize VLM/LLM names (f9d8e77)
  • Parameterize dashboard table, inline app deps, and upgrade CUDA/PyTorch stack (7e90cbb)
  • Change default catalog from dmoore to main; update INSTALL.md (a030a9a)
  • Init DDL objects; move init_tables out of Catalog.init() (7b77ff5, 247d8fd)
  • Fix init_schema notebook to be self-contained on serverless (69a02f3)
  • Convert 00-init-schema to pure SQL; fix 10_validate failures (6ecd2f0)
  • Fix catalog creation on Default Storage workspaces (9709d6a)
  • Handle ABAC policy error in init_env table existence check (e429de0)

App packaging & deployment

  • Decouple OHIF from the wheel; serve from the UC Volume at app startup (ae079ac)
  • Upload the wheel to the UC Volume for app pip install (8434113)
  • Restore OHIF in package resources for wheel distribution (a647b7f)
  • Clean up leftover wheel files from app dirs before deploy (0698194)
  • Replace git+https dep with the UC Volume wheel in the stow-processor job (3fa1765)
  • Use the local wheel for app deps instead of git+https (6857e16)
  • Switch to git+https@fix/install during fix-up, then back to the local wheel (633fbe7)
  • Fix BUILD cache: version comment in requirements.txt; Makefile ohif.tar.gz (38ea38c)
  • Add app-ping.sh smoke test for the DICOMweb viewer app (aa85375)
  • Add tests/deployment/pixels_inference.sh model-serving smoke test (e6d3c4c)
  • Self-host landing page CSS; fix thumbnail API; improve app idempotency (b1c0727)

Lakebase

  • Add Lakebase tests to validate-install: connectivity, sync, grants, writes (510ffd0)
  • Fix Lakebase grants: add UPDATE + DELETE on all tables, default privileges (d07eecb)
  • Grant Lakebase DML privileges to the app service principal (fd509e8)
  • Fix Lakebase role-propagation race in get_or_create_sp_role (2d12f92)
  • Deduplicate instance_paths_vw to prevent PK violation on Reverse ETL sync (b59d068)
  • Automate Reverse ETL synced-table creation via API (92e5ef7)
  • Add synced-table verification, retry for deploy_apps, clarify Reverse ETL docs (6e416e6)
  • Refactor endpoint retrieval logic in lakebase.py (6c6ee2a, plus style passes)
  • Update hypercorn dependency to include HTTP/2 (h2) support (2181742, 1a309f1)
  • Enable Managed File Events config for Lakebase pipelines (f2060ad)

Model serving (Vista3D / MONAI Label)

  • Default DICOMWEB_USE_USER_AUTH to true (OBO auth) (4d5047d)
  • Fix cross-app OBO and add request-correlated debug logging (610b264)
  • Align dcm_ingest databricks-sdk pin with install pipeline (0.60.0 → 0.88.0) (8d9e80c)
  • Vista3D conda env: drop cuda-toolkit, bump pydicom, pin pylibjpeg-openjpeg (45c3c33)
  • Add champion alias, model description, and fix MONAI GET proxy (f1a80fe)
  • Remove vista3d_bundle/vista3d sync exclude to fix model packaging (cb7f6de)
  • Pin torch 2.5.1+cu124; align Python 3.12.3; fix 03a import; remove dabs/dbdemos (3dd25fe)
  • Rework task 03 to use VISTA3D.ipynb with serverless guards (626660b)
  • Fix VISTA3D.ipynb — guard unconditional imports; broaden exception catches (a4bb479)
  • Make serving-endpoint creation idempotent (a7a5285)
  • Make 03b skip update when endpoint already has correct model version (a8b4ae5)
  • Add volume-based error logging to 03c for serverless debugging (7b45561)
  • Switch 03c predict calls to REST API with explicit timeouts (0544e48)
  • Treat model-side 400 errors as warnings in 03c validation (c8a1fca)
  • Fix 03c — check for config_update IN_PROGRESS before testing endpoint (81b5cf2)
  • Fix search_model_versions — remove order_by for serverless compat (7117399)
  • Fix 03b — remove use_service_principal widget (not a job parameter) (9c26746)
  • Wrap ModelServing test cells in try/except (54d6a66)
  • Convert %sql ai_query test cell to Python with try/except (d4f95f6)
  • Exit ModelServing notebook after endpoint creation (9fc8ac5)
  • Fix dbx.pixels imports on GPU cluster (d4bfb86)
  • Add repo root to sys.path after restartPython for dbx.pixels (79c0141)
  • Enable Unity Catalog on the GPU cluster for the Vista3D task (40e51bf)
  • Fix 03a-register-model relative paths after move to resources/dabs/ (db24b3f)

DICOMweb gateway & STOW

  • Refactor DICOM web gateway STOW handler and update deployment notebooks (e8537b8)
  • Refactor STOW handling to derive stow_operations table dynamically (264cfca)
  • Update checkpoint paths in StowProcessor and related notebooks (ca0fd0d)
  • Refactor DICOM STOW processing — enhance directory structure and dynamic path handling (ac7b95a)
  • Update checkpoint paths to include dynamic table names (94aae22)
  • Implement STOW job creation and parameter handling in the gateway (d9bb547)
  • Refactor batch_put in InstancePathCache to accept list or dict (ffe47c3)
  • Refactor LakebaseUtils + InstancePathCache to return lists of dicts; include
    study + series UIDs in the cache key (5fbc6ce)
  • Update README_DICOMWEB.md to reflect the composite cache key (6031749)
  • Normalize STOW cache entry keys to canonical *_instance_uid names (f95f314)
  • Add unit tests for the STOW-RS handler and utilities; mock unavailable third-party
    modules to keep tests isolated (c878186, a08b82f)
  • Enhance DICOM metadata extraction by including file-meta header tags in JSON (27ac0ad)
  • Refactor query-params handling for DICOM web requests (ed77c83)
  • Refactor modalities handling in queries.py (multi-modality search fix) (1f34844)
  • Change branch name from main to production for the metrics gateway (27dd2d7)
  • Change OHIF staticWado from true to false (search fix) (52ca0f9)
  • Fix OHIF MONAI Label mixed content on HTTPS Databricks Apps origin (7a2e7cc)
  • Fix stow_processor job parameter name (b1e4a3f)
  • Fix base_path to files/workflow (was files/install/workflow) (0c47183)

Genie space & Vector Search

  • Add direct deploy engine; remove VS index from Genie tables (3d1500b)
  • Sort Genie space tables by identifier after catalog replacement (e0df9fd)
  • Fix Genie space to reference VS tables in the main catalog (d6fb434)
  • Use main catalog for VS dicom_tags to avoid Control Tower row filters (b5665f5)
  • Drop inherited row filter on dicom_tags before creating VS index (2ea7fea)
  • Make VS index creation idempotent (skip if exists) (83bbd10)
  • Fix Genie space parent_path to use the workspace directory (d910e4a)
  • Fix 3-part table name in GenieSpace dicom_tags saveAsTable (7984c85)
  • Improved Genie prompt (dffef0d)

Lakeview dashboard

  • Template dashboard params at make-deploy time (ce29876)
  • Sync dashboard template from live; match global filter structure (cbcb841)
  • Patch Global Filter widget selection defaults in post-install (3c6c1c8)
  • Add dashboard global filters; patch viewer_host in post-install (49f1110)
  • Remove study_time from queries (523c225)
  • Update dashboard, removing unused columns (85bf1f5)
  • Change dashboard default values (2813f91)
  • Fix dashboard default table parameter to main.pixels_solacc (7fc65251)
  • Restore ::STRING casts on VARIANT fields used in GROUP BY / ORDER BY / DISTINCT (723f4b4)
  • Remove ::STRING casts from 01-dcm-demo queries (env v4 VARIANT) (7c65251)
  • Fix dashboard defaults for dmoore.pixels deployment (bcd3ea1)

Performance

  • Refactor unzip to use mapInPandas for parallel processing (bbfd26f, 43ffa36)
  • Set default unzip workers to 1 (safer out-of-the-box default) (f92a7fc)
  • Enhance DicomMetaExtractor with concurrent I/O (4144429)

Data extraction & redaction

  • Enable permissive=True in 01-dcm-demo.ipynb (6f60baa)
  • Add permissive mode to DicomMetaExtractor and DicomAnonymizerExtractor (9c412a9)
  • Enhance README and DicomMetaExtractor to support removal of UN tags (846a83d)
  • Add function to remove UN VR elements from DICOM dataset (173cdef)
  • Update DicomMetaExtractor to support removal of UN VR elements (edf4501)
  • Refactor DicomMetaExtractor to rename parameter for clarity (599f8dd)
  • Notebook demonstrating metatag de-id by Presidio and Part 15 rules (b3c5fcc, e66a70a)

validate_install / observability

  • Convert 00-init-schema to pure SQL; fix 10_validate failures (6ecd2f0)
  • Fix 10-validate — use REST API for VS/Lakebase; fix Genie name match (0ff4e83)
  • Switch Model Serving info ping to REST API in 10-validate (6266223)
  • Move UC tagging from SQL notebook to 10-validate.py with try/except (58275b7)
  • Fix ALTER SET TAGS — remove IF EXISTS and VOLUME tag (not supported) (9bef583)
  • Add accelerator=pixels tag to all supported assets (665826a)

Dependencies & build

  • Bump Pillow 10.4.0 → 12.1.1 and MONAI 1.3.2 → 1.5.2rc1 (dependabot alerts) (5013598)
  • Pin Pillow, matplotlib, pyarrow with == in requirements.txt (ebb2075)
  • Bump pydicom 3.0.1 → 3.0.2 across requirements + conda envs (d33128a)
  • Align dcm_ingest databricks-sdk pin with the install pipeline (8d9e80c)
  • Consolidate install pip installs to read requirements.txt (7d17cd5)
  • Use >= for runtime-provided packages to avoid serverless immutable-constraint conflicts (7abd950)
  • Reformat with black 26.3.1 to match CI (028e9f9)
  • Apply black formatting to merged anonymizer + middleware (d35ad23)
  • Apply black formatting to bot_cache_builder and lakebase (a99dadd)

Documentation & developer experience

  • Add CLAUDE.md with project structure, build commands, and conventions (6cb6299)
  • Add post-install integration-test commands to CLAUDE.md (25f6224)
  • Update INSTALL.md to reflect the 14-task install job and add testing section (1b22009)
  • Update INSTALL.md with prerequisites, the correct task DAG, and scale_to_zero (0d8e6bc)
  • Tighten INSTALL.md resource descriptions (ddca462)
  • Replace architecture image in README (bd844bd)
  • Update viewer-app test to open OHIF with a MONAI Label study URL (d8e4692)
  • Remove deprecated 02-dcm-browser notebook in favor of OHIF viewer (89300eb)

Merged pull requests

  • #184 — bugfix/fix_ohif_search
  • #185 — bugfix/metrics_gateway
  • #186 — bugfix/search_multiple_modalities
  • #187 — feature/enable_h2 (HTTP/2 via hypercorn)
  • #188 — bugfix/lakebase_endpoints
  • #189 — feature/file_meta (file meta header tags in JSON)
  • #190 — feature/managed_file_events
  • #191 — bugfix/stow_dynamic_pixel_table
  • #192 — bugfix/dicom_identifiers
  • #193 — feature/auto_create_stow_job
  • #195 — perf/map_in_pandas (mapInPandas unzip)
  • #196 — bugfix/dashboard_v3
  • #197 — feature/remove_un_tags
  • #200 — perf/zip_pandas (default unzip workers)
  • #203 — bugfix/improve_genie
  • #206 — feat/permissive (permissive DICOM parsing)
  • #169 — feat/presidio_rules (Presidio + DICOM Part 15 de-id notebook)

Breaking changes since 3.0.0-rc.1

  • Repository layout. Anything importing dbx.pixels.* from a non-installed checkout
    must add src/ to PYTHONPATH (or install the wheel). Install notebooks call
    proxy_prep.init_env() to do this automatically.
  • Apps no longer deployed via DAB apps: sections. They are deployed by the install
    job tasks deploy_prep / deploy_lakebase / deploy_gateway / deploy_viewer /
    deploy_grants. The wheel and OHIF static bundle are loaded from the UC Volume.
  • Default catalog changed from dmoore to main. Override via --var catalog=....
  • Install task names dropped numeric prefixes (e.g. 01_dcm_ingest → dcm_ingest).
  • Catalog.init() no longer creates tables — DDL is owned by init_schema.
  • 02-dcm-browser notebook removed — use the OHIF viewer app instead.
  • models/vista3d/ renamed to models/monai/.

Upgrade notes

The simplest upgrade from 3.0.0-rc.* or earlier is a clean install:

git fetch --tags
git checkout 3.0.0
make deploy CATALOG=<your_catalog> SCHEMA=<your_schema> PROFILE=<your_profile> TARGET=prod
databricks bundle run pixels_install -t prod -p <your_profile> --var catalog=<your_catalog>

To remove an existing deployment first, run install/uninstall.py.

Contributors

Thanks to everyone who reviewed PRs and tested release candidates.


Full Changelog: 3.0.0-rc.1...3.0.0