Repository navigation
v3.0.0
Pixels 3.0.0
Pixels 3.0.0 is the GA release of the next-generation Databricks medical-imaging
accelerator. It builds on 3.0.0-rc.1 (and the nine release candidates that followed)
and ships the complete repository reorganization, a fully-declarative Databricks
Asset Bundle (DAB) installer with a 14-task DAG, and hardened Lakebase, DICOMweb
gateway, OHIF viewer, Vista3D model serving, Genie, and Lakeview
dashboard deployment, alongside a long list of stability, performance, and
documentation improvements.
For the feature highlights introduced over 2.2.0 (DICOMweb Gateway, Lakebase
integration, Variant data type, DICOM Redactor, Genie automation, OHIF v3.11.1,
MONAI 1.5.2, etc.), see the v3.0.0-rc.1 release notes.
This changelog covers every change since 3.0.0-rc.1 — 208 commits, 9 release
candidates, 14 merged pull requests.
Highlights since 3.0.0-rc.1
- Repository reorganization — Python library moved to
src/dbx/pixels/(src layout),
apps consolidated underapps/, model assets undermodels/monai/, install task
notebooks underinstall/, demos undernotebooks/, docs underdocs/. - One-click DAB installer — a single
make deploy+databricks bundle run pixels_install
provisions the entire stack: UC schema/volume/tables, Lakebase Postgres instance with
Reverse ETL sync, DICOMweb gateway and OHIF viewer apps, Vista3D model serving endpoint,
Genie space, Lakeview dashboard, STOW processor job, and grants. - 14-task install job with explicit dependencies, retry-free execution, and a final
validate_installtask that verifies every surface end-to-end. - Cross-workspace, cross-catalog ready — every notebook is parameterized via widgets
fed bydatabricks.ymlvariables. Default catalog is nowmain, schema ispixels. - Idempotent deployment — every install task can be re-run without breaking the stack
(schemas, endpoints, apps, Lakebase grants, VS index, Genie space). - Lakebase robustness — DML grants for app service principals, role-propagation race
fix, dedupedinstance_paths_vwto avoid PK violations on Reverse ETL sync, automated
synced-table creation via API, full connectivity/sync/grants/writes tests in
validate_install. - Model serving robustness — split task
03into03aregister /03bdeploy /
03cvalidate; endpoint creation and update are fully idempotent; predict calls use
the REST API with explicit timeouts; stale-config detection across schema/model
renames; champion alias on registration. - OHIF mixed-content fix on HTTPS Databricks Apps origins; viewer-side reverse proxy
for MONAIinfo/infercalls (cross-app OBO with request-correlated logging). - Wheel + OHIF served from UC Volume —
deploy_prepuploadsdatabricks_pixels-*.whl
andohif.tar.gzto/Volumes/<catalog>/<schema>/<volume>/. Appspip installthe
wheel from the Volume at startup and extract OHIF in-container, removing large
.wasmfiles from DAB sync. - STOW-RS hardening — gateway derives
stow_operationstable dynamically from the
configured Pixels table, normalizes cache keys to composite(study, series, sop)
instance UIDs, and ships unit tests for the handler. - Performance —
mapInPandas-based parallel unzip for large archives, concurrent
I/O inDicomMetaExtractor, configurable worker counts. - Documentation —
docs/INSTALL.mdrewritten for the 14-task DAG, prerequisites,
andscale_to_zero_enabled. NewCLAUDE.mdcapturing project structure, build
commands, conventions, and the mandatory post-install integration-test suite.
Repository reorganization (the reorg branch)
The repository was restructured in seven phases to clearly separate library code,
install notebooks, apps, models, and demo material:
- Phase 1 — consolidate install tasks into
install/(7d7f4d6) - Phase 2 — extract apps to
apps/, move shared app code todbx/pixels/common/(3dcef06) - Phase 3 — move the library to
src/dbx/pixels/(src layout); updatesetup.py(0c29e33) - Phase 5 — move the model to
models/vista3d/, demos tonotebooks/, docs todocs/(4e10863) - Phase 6 + 7 — style fixes, bundle validation, refresh
README.mdandCLAUDE.mdlinks (c0080b0) - Rename
models/vista3d/→models/monai/to reflect broader scope (MONAI Label + Vista3D bundle) (13e21bd) - Update
CLAUDE.mdfor the reorg; fix pre-commit paths for the src layout (b50130a) - Restore OHIF in package resources for wheel distribution (
a647b7f) - Decouple Vista3D model from DAB sync; move plan docs to
plans/(80ba431)
make dev, make build, make deploy, make test, make style, make check, and
make clean continue to work with the new layout. The Asset Bundle is configured to
sync the right subset of the new tree and to include dist/*.whl.
DAB installer & install job
A complete DAB installer replaces ad-hoc deployment notebooks. The install job is a
14-task DAG that provisions everything top-to-bottom.
Structure
- Add
resources/install-job.yml,resources/dashboard.yml,resources/unity-catalog.ymlDAB resources (9b7e40e) - Split task
03into03aregister-model,03bdeploy-endpoint,03cvalidate-model (09d46b6) - Split
deploy-appsinto 5 install tasks:deploy_prep,deploy_lakebase,
deploy_gateway,deploy_viewer,deploy_grants(1a0965b) - Drop numeric task prefixes from task names (
1a0965b) - Add
10_validateinstall task to verify every Pixels surface (73ea589) - Add
09-post-install-updatetask for dashboard params + app thumbnail (3685dec) - Add
stow_processortask — job creation + perms (65d2f76) - Move
init-schemainto the DAB; dedup Genie space; fix03arestart (65d2f76) - Add
install/uninstall.pyto tear down a Pixels deployment (2c62bf9) - Retry validation up to 12× at 3-min intervals until all checks pass (
3e39948) - Disable per-task retries (
max_retries: 0) so failures surface immediately (bf645fc) - Increase serverless compute environment from 1 to 4 (
d4f2860) - Wire
lakebase_instance_namethrough DAB → job → notebook widgets (6e35338) - Add
scale_to_zero_enabledvariable; pin deps; reorder job tasks (d1650c0) - Detect stale serving-endpoint config across schema/model name changes (
4bf62ef) - Skip trashed Pixels dashboards in
post_install_update(51ed6cf) - Split app-shell creation from code deploy to fix fresh-install grants race (
9a48438) - Template dashboard params at
make deploytime, not via post-install patch (ce29876) - Broaden pre-commit E2E test rule to apply to every commit (
f770544) - Address PR #211 review feedback for 3.0.0 release prep (
ecbbbd7) - Update
Makefileandrequirements.txtfor improved environment setup (7cc7041)
Configuration & parameterization
- Parameterize catalog references for cross-workspace installs (
22968df) - Parameterize VLM/LLM names (
f9d8e77) - Parameterize dashboard table, inline app deps, and upgrade CUDA/PyTorch stack (
7e90cbb) - Change default catalog from
dmooretomain; updateINSTALL.md(a030a9a) - Init DDL objects; move
init_tablesout ofCatalog.init()(7b77ff5,247d8fd) - Fix
init_schemanotebook to be self-contained on serverless (69a02f3) - Convert
00-init-schemato pure SQL; fix10_validatefailures (6ecd2f0) - Fix catalog creation on Default Storage workspaces (
9709d6a) - Handle ABAC policy error in
init_envtable existence check (e429de0)
App packaging & deployment
- Decouple OHIF from the wheel; serve from the UC Volume at app startup (
ae079ac) - Upload the wheel to the UC Volume for app
pip install(8434113) - Restore OHIF in package resources for wheel distribution (
a647b7f) - Clean up leftover wheel files from app dirs before deploy (
0698194) - Replace
git+httpsdep with the UC Volume wheel in thestow-processorjob (3fa1765) - Use the local wheel for app deps instead of
git+https(6857e16) - Switch to
git+https@fix/installduring fix-up, then back to the local wheel (633fbe7) - Fix BUILD cache: version comment in
requirements.txt; Makefileohif.tar.gz(38ea38c) - Add
app-ping.shsmoke test for the DICOMweb viewer app (aa85375) - Add
tests/deployment/pixels_inference.shmodel-serving smoke test (e6d3c4c) - Self-host landing page CSS; fix thumbnail API; improve app idempotency (
b1c0727)
Lakebase
- Add Lakebase tests to
validate-install: connectivity, sync, grants, writes (510ffd0) - Fix Lakebase grants: add UPDATE + DELETE on all tables, default privileges (
d07eecb) - Grant Lakebase DML privileges to the app service principal (
fd509e8) - Fix Lakebase role-propagation race in
get_or_create_sp_role(2d12f92) - Deduplicate
instance_paths_vwto prevent PK violation on Reverse ETL sync (b59d068) - Automate Reverse ETL synced-table creation via API (
92e5ef7) - Add synced-table verification, retry for
deploy_apps, clarify Reverse ETL docs (6e416e6) - Refactor endpoint retrieval logic in
lakebase.py(6c6ee2a, plus style passes) - Update hypercorn dependency to include HTTP/2 (
h2) support (2181742,1a309f1) - Enable Managed File Events config for Lakebase pipelines (
f2060ad)
Model serving (Vista3D / MONAI Label)
- Default
DICOMWEB_USE_USER_AUTHtotrue(OBO auth) (4d5047d) - Fix cross-app OBO and add request-correlated debug logging (
610b264) - Align
dcm_ingestdatabricks-sdkpin with install pipeline (0.60.0 → 0.88.0) (8d9e80c) - Vista3D conda env: drop
cuda-toolkit, bump pydicom, pinpylibjpeg-openjpeg(45c3c33) - Add champion alias, model description, and fix MONAI GET proxy (
f1a80fe) - Remove
vista3d_bundle/vista3dsync exclude to fix model packaging (cb7f6de) - Pin torch 2.5.1+cu124; align Python 3.12.3; fix
03aimport; removedabs/dbdemos(3dd25fe) - Rework task
03to useVISTA3D.ipynbwith serverless guards (626660b) - Fix
VISTA3D.ipynb— guard unconditional imports; broaden exception catches (a4bb479) - Make serving-endpoint creation idempotent (
a7a5285) - Make
03bskip update when endpoint already has correct model version (a8b4ae5) - Add volume-based error logging to
03cfor serverless debugging (7b45561) - Switch
03cpredict calls to REST API with explicit timeouts (0544e48) - Treat model-side 400 errors as warnings in
03cvalidation (c8a1fca) - Fix
03c— check forconfig_update IN_PROGRESSbefore testing endpoint (81b5cf2) - Fix
search_model_versions— removeorder_byfor serverless compat (7117399) - Fix
03b— removeuse_service_principalwidget (not a job parameter) (9c26746) - Wrap
ModelServingtest cells intry/except(54d6a66) - Convert
%sql ai_querytest cell to Python withtry/except(d4f95f6) - Exit
ModelServingnotebook after endpoint creation (9fc8ac5) - Fix
dbx.pixelsimports on GPU cluster (d4bfb86) - Add repo root to
sys.pathafterrestartPythonfordbx.pixels(79c0141) - Enable Unity Catalog on the GPU cluster for the Vista3D task (
40e51bf) - Fix
03a-register-modelrelative paths after move toresources/dabs/(db24b3f)
DICOMweb gateway & STOW
- Refactor DICOM web gateway STOW handler and update deployment notebooks (
e8537b8) - Refactor STOW handling to derive
stow_operationstable dynamically (264cfca) - Update checkpoint paths in
StowProcessorand related notebooks (ca0fd0d) - Refactor DICOM STOW processing — enhance directory structure and dynamic path handling (
ac7b95a) - Update checkpoint paths to include dynamic table names (
94aae22) - Implement STOW job creation and parameter handling in the gateway (
d9bb547) - Refactor
batch_putinInstancePathCacheto accept list or dict (ffe47c3) - Refactor
LakebaseUtils+InstancePathCacheto return lists of dicts; include
study + series UIDs in the cache key (5fbc6ce) - Update
README_DICOMWEB.mdto reflect the composite cache key (6031749) - Normalize STOW cache entry keys to canonical
*_instance_uidnames (f95f314) - Add unit tests for the STOW-RS handler and utilities; mock unavailable third-party
modules to keep tests isolated (c878186,a08b82f) - Enhance DICOM metadata extraction by including file-meta header tags in JSON (
27ac0ad) - Refactor query-params handling for DICOM web requests (
ed77c83) - Refactor modalities handling in
queries.py(multi-modality search fix) (1f34844) - Change branch name from
maintoproductionfor the metrics gateway (27dd2d7) - Change OHIF
staticWadofromtruetofalse(search fix) (52ca0f9) - Fix OHIF MONAI Label mixed content on HTTPS Databricks Apps origin (
7a2e7cc) - Fix
stow_processorjob parameter name (b1e4a3f) - Fix
base_pathtofiles/workflow(wasfiles/install/workflow) (0c47183)
Genie space & Vector Search
- Add direct deploy engine; remove VS index from Genie tables (
3d1500b) - Sort Genie space tables by identifier after catalog replacement (
e0df9fd) - Fix Genie space to reference VS tables in the
maincatalog (d6fb434) - Use
maincatalog for VSdicom_tagsto avoid Control Tower row filters (b5665f5) - Drop inherited row filter on
dicom_tagsbefore creating VS index (2ea7fea) - Make VS index creation idempotent (skip if exists) (
83bbd10) - Fix Genie space
parent_pathto use the workspace directory (d910e4a) - Fix 3-part table name in
GenieSpace dicom_tags saveAsTable(7984c85) - Improved Genie prompt (
dffef0d)
Lakeview dashboard
- Template dashboard params at make-deploy time (
ce29876) - Sync dashboard template from live; match global filter structure (
cbcb841) - Patch Global Filter widget selection defaults in post-install (
3c6c1c8) - Add dashboard global filters; patch
viewer_hostin post-install (49f1110) - Remove
study_timefrom queries (523c225) - Update dashboard, removing unused columns (
85bf1f5) - Change dashboard default values (
2813f91) - Fix dashboard default table parameter to
main.pixels_solacc(7fc65251) - Restore
::STRINGcasts on VARIANT fields used inGROUP BY/ORDER BY/DISTINCT(723f4b4) - Remove
::STRINGcasts from01-dcm-demoqueries (env v4 VARIANT) (7c65251) - Fix dashboard defaults for
dmoore.pixelsdeployment (bcd3ea1)
Performance
- Refactor unzip to use
mapInPandasfor parallel processing (bbfd26f,43ffa36) - Set default unzip workers to 1 (safer out-of-the-box default) (
f92a7fc) - Enhance
DicomMetaExtractorwith concurrent I/O (4144429)
Data extraction & redaction
- Enable
permissive=Truein01-dcm-demo.ipynb(6f60baa) - Add permissive mode to
DicomMetaExtractorandDicomAnonymizerExtractor(9c412a9) - Enhance README and
DicomMetaExtractorto support removal ofUNtags (846a83d) - Add function to remove
UNVR elements from DICOM dataset (173cdef) - Update
DicomMetaExtractorto support removal ofUNVR elements (edf4501) - Refactor
DicomMetaExtractorto rename parameter for clarity (599f8dd) - Notebook demonstrating metatag de-id by Presidio and Part 15 rules (
b3c5fcc,e66a70a)
validate_install / observability
- Convert
00-init-schemato pure SQL; fix10_validatefailures (6ecd2f0) - Fix
10-validate— use REST API for VS/Lakebase; fix Genie name match (0ff4e83) - Switch Model Serving info ping to REST API in
10-validate(6266223) - Move UC tagging from SQL notebook to
10-validate.pywithtry/except(58275b7) - Fix
ALTER SET TAGS— removeIF EXISTSandVOLUMEtag (not supported) (9bef583) - Add
accelerator=pixelstag to all supported assets (665826a)
Dependencies & build
- Bump Pillow 10.4.0 → 12.1.1 and MONAI 1.3.2 → 1.5.2rc1 (dependabot alerts) (
5013598) - Pin Pillow, matplotlib, pyarrow with
==inrequirements.txt(ebb2075) - Bump pydicom 3.0.1 → 3.0.2 across requirements + conda envs (
d33128a) - Align
dcm_ingestdatabricks-sdkpin with the install pipeline (8d9e80c) - Consolidate install pip installs to read
requirements.txt(7d17cd5) - Use
>=for runtime-provided packages to avoid serverless immutable-constraint conflicts (7abd950) - Reformat with black 26.3.1 to match CI (
028e9f9) - Apply black formatting to merged anonymizer + middleware (
d35ad23) - Apply black formatting to
bot_cache_builderandlakebase(a99dadd)
Documentation & developer experience
- Add
CLAUDE.mdwith project structure, build commands, and conventions (6cb6299) - Add post-install integration-test commands to
CLAUDE.md(25f6224) - Update
INSTALL.mdto reflect the 14-task install job and add testing section (1b22009) - Update
INSTALL.mdwith prerequisites, the correct task DAG, andscale_to_zero(0d8e6bc) - Tighten
INSTALL.mdresource descriptions (ddca462) - Replace architecture image in
README(bd844bd) - Update viewer-app test to open OHIF with a MONAI Label study URL (
d8e4692) - Remove deprecated
02-dcm-browsernotebook in favor of OHIF viewer (89300eb)
Merged pull requests
- #184 — bugfix/fix_ohif_search
- #185 — bugfix/metrics_gateway
- #186 — bugfix/search_multiple_modalities
- #187 — feature/enable_h2 (HTTP/2 via hypercorn)
- #188 — bugfix/lakebase_endpoints
- #189 — feature/file_meta (file meta header tags in JSON)
- #190 — feature/managed_file_events
- #191 — bugfix/stow_dynamic_pixel_table
- #192 — bugfix/dicom_identifiers
- #193 — feature/auto_create_stow_job
- #195 — perf/map_in_pandas (mapInPandas unzip)
- #196 — bugfix/dashboard_v3
- #197 — feature/remove_un_tags
- #200 — perf/zip_pandas (default unzip workers)
- #203 — bugfix/improve_genie
- #206 — feat/permissive (permissive DICOM parsing)
- #169 — feat/presidio_rules (Presidio + DICOM Part 15 de-id notebook)
Breaking changes since 3.0.0-rc.1
- Repository layout. Anything importing
dbx.pixels.*from a non-installed checkout
must addsrc/toPYTHONPATH(or install the wheel). Install notebooks call
proxy_prep.init_env()to do this automatically. - Apps no longer deployed via DAB
apps:sections. They are deployed by the install
job tasksdeploy_prep/deploy_lakebase/deploy_gateway/deploy_viewer/
deploy_grants. The wheel and OHIF static bundle are loaded from the UC Volume. - Default catalog changed from
dmooretomain. Override via--var catalog=.... - Install task names dropped numeric prefixes (e.g.
01_dcm_ingest→dcm_ingest). Catalog.init()no longer creates tables — DDL is owned byinit_schema.02-dcm-browsernotebook removed — use the OHIF viewer app instead.models/vista3d/renamed tomodels/monai/.
Upgrade notes
The simplest upgrade from 3.0.0-rc.* or earlier is a clean install:
git fetch --tags
git checkout 3.0.0
make deploy CATALOG=<your_catalog> SCHEMA=<your_schema> PROFILE=<your_profile> TARGET=prod
databricks bundle run pixels_install -t prod -p <your_profile> --var catalog=<your_catalog>To remove an existing deployment first, run install/uninstall.py.
Contributors
- @dmoore247 — 132 commits
- @erinaldidb — 74 commits
- @yenlow — 2 commits (Presidio + Part 15 de-id notebook)
Thanks to everyone who reviewed PRs and tested release candidates.
Full Changelog: 3.0.0-rc.1...3.0.0