Fix scale-lineage regressions and make napari auto-launch non-blocking in Qt#14
Merged
AdvancedImagingUTSW merged 10 commits intomainfrom Mar 24, 2026
Merged
Fix scale-lineage regressions and make napari auto-launch non-blocking in Qt#14AdvancedImagingUTSW merged 10 commits intomainfrom
AdvancedImagingUTSW merged 10 commits intomainfrom
Conversation
Introduce OME-Zarr v3 storage helpers and migrate ClearEx stores to a namespaced runtime-cache layout. Adds new clearex.io.ome_store module with helpers for namespaced metadata, runtime-cache source components, analysis cache/auxiliary roots, OME metadata handling, and copying/publishing utilities. Update pipelines and I/O to read/write the new SOURCE_CACHE_COMPONENT and analysis cache/auxiliary paths, migrate store spatial-calibration and metadata handling, and adjust resume/checkpoint logic to use cache/auxiliary groups. Add CLI flags for migrating legacy stores (--migrate-store/--migrate-output/--migrate-overwrite). Update pyproject dependencies to include bioio-ome-zarr, ome-zarr, ome-zarr-models and bump zarr requirement to >=3.1.1. These changes refactor how canonical source data and analysis outputs are stored and advertised (OME-Zarr v3), while preserving compatibility checks and pyramid generation using the new namespaced layout.
Update docs and README to adopt OME‑Zarr v3 (*.ome.zarr) as the canonical ClearEx store and formalize a public vs internal layout. Introduces a clear separation between public OME HCS collections (root A/1/<field>/<level> and results/<analysis>/latest) and ClearEx-owned runtime namespaces (clearex/runtime_cache/*, clearex/metadata, clearex/provenance, clearex/results). Add CLI flags and examples for migration and display-pyramid workflows (--display-pyramid, --migrate-store, --migrate-output, --migrate-overwrite) and update runtime/ingestion, provenance, module maps, and workflow docs to reference the new persistence and naming conventions. Provides migration guidance and changes many examples to use data_store.ome.zarr, clarifies spatial_calibration storage in clearex/metadata, and documents that legacy root data/data_pyramid layouts are migration-only.
Introduce TensorStore-based handling for Navigate BDV N5 sources and document that legacy .n5 is source-only (route via experiment.yml). Key changes: add tensorstore dependency (pyproject.toml, uv.lock), implement TensorStore N5 adapters and helpers in src/clearex/io/experiment.py (adapter, open-as-dask, N5 enumeration/summarization, BDV XML parsing, load_navigate_experiment_source_image_info), update GUI to use the new loader, and update docs to describe the required experiment.yml+XML workflow and that raw zarr.read on .n5 is unsupported. Tests were added/adjusted to create real N5 fixtures and verify behavior (tests/io/test_experiment.py). Overall this enforces canonical materialization into *.ome.zarr and keeps Dask ingestion parallelized on zarr>=3 by using TensorStore for N5 reads.
- add OME-model based validation helper for public source collections and use it when ingestion_progress metadata is stale/incomplete so valid canonical stores are reused instead of re-ingested - add Navigate multiposition sidecar fallback: when is_multiposition=true, resolve multi_positions.yml from both Saving.save_directory and the local experiment.yml directory to handle remote Windows save paths on Linux - filter BDV N5 setup discovery to include only setup*/timepoint*/s0 components with persisted chunk payload files (exclude attributes-only placeholder setups that read as implicit zeros) - extend io tests for stale progress acceptance/rejection, multiposition sidecar fallback, and placeholder setup exclusion; keep existing BDV N5 coverage - update io README materialization rules to document canonical reuse fallback, multiposition sidecar precedence, and placeholder setup filtering
…writes Stop provenance dedup from skipping mip_export runs by removing mip_export from the dedup operation set in _run_workflow. This ensures MIP export executes even when a matching provenance run exists, which avoids stale latest-output reuse during validation/export iterations. Fix a correctness bug in source-aligned canonical ingestion writes: when z_batch_depth does not align to the target z chunk size, concurrent region submissions can race on read-modify-write chunk updates and zero out sub-chunk ranges. Add _source_aligned_writes_require_serial_submission(...) and force regions_per_submission=1 for this misaligned case while preserving existing concurrency when writes are chunk-aligned. Plumb target chunk metadata into _write_dask_array_source_aligned_plane_batches and cover behavior with regression tests for serial-submission detection and serialized batch execution under misalignment. Add a workflow regression test that asserts mip_export is executed (not skipped) even with a matching completed provenance run.
Fix a data-loss bug in _resample_axis_linear_to_uint16 that affected multi-position TIFF outputs when resampling the z axis. For non-leading resample axes (for example axes p,z,x where z is axis 1), reshaping the moved destination could produce a non-shared copy, so interpolation writes never reached the destination buffer and outputs remained zero-filled after resample rewrite. Detect when the reshaped destination does not share memory with the moved destination view and explicitly copy the interpolated payload back into dst_moved before returning. Preserve existing blockwise interpolation behavior and memory budgeting. Add regression coverage for this path by validating non-leading-axis writeback and multi-position TIFF xz/yz resampled outputs. Update touched mip_export tests to use zarr create_array for zarr v3 compatibility and refresh legacy component-path assertions to the current clearex namespaced contract.
… attrs Fix registration startup failures on canonical OME-Zarr v3 stores where source_experiment/stage_rows/spatial_calibration are persisted under clearex/metadata instead of root attrs. run_registration_analysis now merges root attrs with namespaced store metadata before resolving spatial calibration and stage rows. Extend _parse_multiposition_stage_rows to accept list-of-dict payloads (x/y/z/theta/f or uppercase variants), and make _load_stage_rows prefer embedded stage_rows and navigate_experiment metadata before falling back to source_experiment sidecar loading. Add regression coverage for dict-row parsing and for run_registration_analysis progressing past the multiposition stage-metadata gate when metadata is provided via clearex/metadata.
Fix registration runtime failures caused by zarr v3 AsyncGroup.create_dataset requiring explicit shape for data payload writes. Replace registration output writes that used create_dataset(data=...) with create_array(...) so blend-weight profiles and edge/affine metadata arrays are written through the v3-compatible API. Also align registration test expectations with the canonical OME-Zarr v3 layout by asserting auxiliary artifacts under clearex/results/... and fused runtime data under clearex/runtime_cache/results/... . Validated with targeted registration tests that cover run_registration_analysis output materialization and namespaced stage-metadata handling.
This patch closes the recent scale-regression loop by centralizing voxel-size\nresolution and propagating it consistently through shear, registration,\nvisualization, flatfield, mip export, and OME publication.\n\nKey changes\n- Added shared voxel-size resolver helpers in io/ome_store.py with ordered\n fallback across:\n 1) source component attrs\n 2) source_component ancestry\n 3) namespaced store metadata\n 4) root attrs\n 5) Navigate metadata\n 6) default (1,1,1)\n- Added source provenance labeling (voxel_size_resolution_source) so downstream\n products can report exactly where scale was resolved.\n- Made load_store_metadata read-only safe (no metadata-group creation for\n mode='r' roots) to prevent resolver-side write attempts during read flows.\n- Updated flatfield/shear/registration/mip-export and OME publish paths to use\n the shared resolver and persist resolved scale lineage metadata.\n- Updated visualization scale resolution so non-base downsampled layers still\n apply shape-ratio scaling when only root/store-level voxel metadata is\n available (preserves physical spacing in napari).\n- Added subsystem guardrails and validation guidance in\n src/clearex/shear/README.md to prevent recurrence.\n\nTests\n- Added tests/io/test_ome_store_scale.py (resolver chain, metadata fallback,\n read-only metadata safety, publish scale transform lineage).\n- Added/updated scale lineage tests in flatfield, shear, registration, and\n visualization test suites.\n- Updated shear tests to assert runtime-cache output component access via\n summary.data_component.\n\nValidation executed\n- uv run ruff format src/clearex/io/ome_store.py src/clearex/visualization/pipeline.py tests/io/test_ome_store_scale.py tests/shear/test_pipeline.py\n- uv run ruff check src/clearex/io/ome_store.py src/clearex/visualization/pipeline.py tests/io/test_ome_store_scale.py tests/shear/test_pipeline.py\n- uv run --with pytest --with requests python -m pytest -q \\n tests/shear/test_pipeline.py \\n tests/io/test_ome_store_scale.py \\n tests/flatfield/test_pipeline.py::test_copy_source_array_attrs_preserves_voxel_size \\n tests/registration/test_pipeline.py::test_extract_voxel_size_uses_source_component_chain \\n tests/visualization/test_pipeline.py::test_run_visualization_analysis_prefers_voxel_size_um_attrs \\n tests/visualization/test_pipeline.py::test_run_visualization_analysis_resolves_scale_from_source_chain \\n tests/visualization/test_pipeline.py::test_launch_napari_viewer_resolves_per_layer_scale_for_downsampled_components\n- Result: all targeted tests passed.
Problem\n- Visualization progress could appear stuck at ~65% after napari opened when\n launch_mode=auto resolved to in_process on the main thread.\n- In GUI/Qt workflows this blocked run_visualization_analysis until the viewer\n closed, so workflow progress and metadata finalization did not advance while\n napari remained open.\n\nSolution\n- Updated launch-mode resolution in src/clearex/visualization/pipeline.py:\n - launch_mode=auto now prefers subprocess when a Qt application instance is\n active (PyQt6 QApplication.instance() is not None).\n - Existing behavior is preserved for non-Qt contexts:\n - main thread -> in_process\n - non-main thread -> subprocess\n- Kept explicit modes unchanged:\n - launch_mode=in_process remains blocking by design.\n - launch_mode=subprocess remains non-blocking by design.\n\nWhy this is safe\n- Subprocess launch path already supports keyframe manifest handoff by passing\n keyframe_manifest_path to the subprocess runner and forcing in_process only\n within the spawned viewer process.\n- This change affects only auto-resolution policy and does not alter payload\n construction, layer resolution, or metadata schema.\n\nTests\n- Added launch-mode policy tests in tests/visualization/test_pipeline.py:\n - test_resolve_effective_launch_mode_auto_prefers_subprocess_with_qt_app\n - test_resolve_effective_launch_mode_auto_prefers_in_process_without_qt_app\n - test_run_visualization_analysis_auto_uses_subprocess_with_active_qt_app\n\nDocs\n- Updated src/clearex/visualization/README.md GUI/workflow notes to document\n the new auto policy under Qt and retained non-Qt behavior.\n\nValidation executed\n- uv run ruff format src/clearex/visualization/pipeline.py tests/visualization/test_pipeline.py\n- uv run ruff check src/clearex/visualization/pipeline.py tests/visualization/test_pipeline.py\n- uv run --with pytest --with requests python -m pytest -q \\n tests/visualization/test_pipeline.py::test_resolve_effective_launch_mode_auto_prefers_subprocess_with_qt_app \\n tests/visualization/test_pipeline.py::test_resolve_effective_launch_mode_auto_prefers_in_process_without_qt_app \\n tests/visualization/test_pipeline.py::test_run_visualization_analysis_auto_uses_subprocess_with_active_qt_app\n- Result: 3 passed
There was a problem hiding this comment.
Pull request overview
This PR addresses two regressions observed during OME-Zarr migration testing: preserving physical voxel-size lineage through chained analyses, and preventing GUI workflows from blocking when napari is auto-launched under an active Qt application. It also introduces a canonical OME-Zarr v3 store layout with ClearEx-namespaced metadata/provenance/runtime-cache paths and provides a CLI migration path from legacy stores.
Changes:
- Centralize voxel-size resolution (with provenance) and propagate resolved scale metadata through multiple analysis and publication paths.
- Make
launch_mode=autoprefer non-blocking subprocess launch when a Qt app instance is active. - Adopt and document canonical OME-Zarr v3 store conventions (namespaced ClearEx groups), including legacy store migration tooling and expanded test coverage.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/visualization/test_pipeline.py | Adds coverage for Qt-aware auto launch mode and voxel-scale resolution through source_component ancestry; minor assertion formatting update. |
| tests/test_main.py | Adds regression coverage ensuring MIP export runs despite matching provenance in specific scenarios. |
| tests/shear/test_pipeline.py | Updates Zarr array creation API usage and adds shear voxel-size inheritance test; updates expected summary fields. |
| tests/registration/test_pipeline.py | Adds tests for voxel-size lookup via ancestry, mapping-based stage-row parsing, and namespaced metadata usage; adjusts expected output paths. |
| tests/mip_export/test_pipeline.py | Updates expected namespaced result/provenance paths, switches to create_array, and adds resampled-Z multi-position TIFF export coverage. |
| tests/io/test_ome_store_scale.py | New tests for voxel-size resolver ordering, read-only metadata loading behavior, and OME publication scale correctness. |
| tests/io/test_experiment.py | Adds TensorStore-backed N5 fixture helpers and new ingestion/resume cases; updates canonical component expectations and stale-progress handling. |
| tests/flatfield/test_pipeline.py | Adds test ensuring flatfield attr copying preserves voxel-size metadata. |
| src/clearex/workflow.py | Updates logical input/output component maps to runtime-cache / namespaced paths. |
| src/clearex/visualization/pipeline.py | Uses namespaced metadata, resolved voxel size provenance in napari payload, and Qt-aware non-blocking auto launch behavior; updates particle detection default component path. |
| src/clearex/visualization/README.md | Documents updated storage contract, namespaced calibration source, and Qt auto launch policy. |
| src/clearex/usegment3d/pipeline.py | Shifts outputs to runtime-cache + auxiliary roots; improves pyramid-factor discovery using component attrs; keeps public root publishing contract. |
| src/clearex/shear/pipeline.py | Switches voxel-size extraction to shared resolver, persists provenance label, and moves outputs to runtime-cache + auxiliary roots. |
| src/clearex/shear/README.md | New subsystem reference documenting shear scale invariants and resolver policy. |
| src/clearex/registration/pipeline.py | Switches voxel-size extraction to shared resolver, supports namespaced metadata for stage rows/calibration, and moves outputs/artifacts to runtime-cache + auxiliary roots with public publishing root. |
| src/clearex/registration/README.md | Documents OME-Zarr runtime contract, new output locations, and migration-only legacy paths. |
| src/clearex/mip_export/pipeline.py | Uses shared voxel-size resolver, fixes resampling writeback for non-leading axes, and stores metadata under namespaced latest root. |
| src/clearex/mip_export/README.md | Updates store/namespace contract, clarifies logical input alias behavior and non-public-image nature of MIP export. |
| src/clearex/main.py | Adopts namespaced component requirements, publishes image collections from runtime cache after image-producing steps, adds legacy-store migration CLI behavior, and updates canonical-store messaging. |
| src/clearex/io/read.py | Prefers public OME array selection when OME metadata is present and enriches returned metadata accordingly. |
| src/clearex/io/provenance.py | Moves provenance/gui-state storage into clearex/... namespaces and updates default output component selection for public-vs-auxiliary outputs. |
| src/clearex/io/ome_store.py | New OME-Zarr v3 helpers: namespace paths, metadata I/O, voxel-size resolution with provenance, public HCS publication from runtime cache, and legacy store migration. |
| src/clearex/io/cli.py | Adds migration-related CLI flags. |
| src/clearex/io/README.md | Updates canonical store policy to OME-Zarr v3, namespaced paths, TensorStore-backed BDV N5 ingestion rules, and migration guidance. |
| src/clearex/gui/app.py | Uses Navigate-specific BDV N5 source image-info summarization instead of generic reader open path. |
| src/clearex/gui/README.md | Updates GUI contract/docs for canonical .ome.zarr stores, namespaced calibration source, and BDV N5 ingestion expectations. |
| src/clearex/flatfield/pipeline.py | Moves outputs to runtime-cache + auxiliary roots, ensures voxel size is resolved and persisted in output attrs, and strengthens resume compatibility checks. |
| src/clearex/flatfield/README.md | Updates output layout and public-vs-internal contract documentation. |
| src/clearex/detect/pipeline.py | Moves particle detection results into namespaced auxiliary root and updates napari points component path. |
| src/clearex/detect/README.md | Updates particle detection storage contract to namespaced latest root and clarifies it is non-image output. |
| src/clearex/deconvolution/pipeline.py | Reads voxel sizes from runtime-cache source + namespaced metadata, writes runtime-cache outputs and auxiliary metadata, and updates synthetic PSF asset persistence target. |
| src/clearex/AGENTS.md | Updates agent-facing runtime invariants for canonical OME-Zarr v3, namespaced paths, BDV N5 ingestion rules, and publication behavior. |
| pyproject.toml | Updates dependencies for OME-Zarr v3 tooling (adds ome-zarr, ome-zarr-models, tensorstore, bioio-ome-zarr) and bumps zarr to v3. |
| docs/zarr_materialization_workflow.rst | Rewrites workflow docs for OME-Zarr materialization + legacy migration and public-vs-internal layout. |
| docs/source/runtime/provenance.rst | Updates provenance layout and publication model for OME-Zarr/namespaced paths. |
| docs/source/runtime/module-map.rst | Updates module map to include OME store helpers, publication, and expanded integrated analyses. |
| docs/source/runtime/ingestion-and-canonical-store.rst | Updates ingestion and canonical store contract to OME-Zarr v3 with namespaced internal execution arrays and public HCS layout. |
| docs/source/runtime/cli-and-execution.rst | Updates CLI flags list and describes OME-Zarr canonical-store behavior and migration options. |
| docs/source/runtime/architecture-overview.rst | Updates architectural invariants for OME-Zarr v3, public HCS, and runtime-cache execution model. |
| docs/source/getting-started.rst | Adds guidance for .ome.zarr materialization, BDV N5 routing via experiment.yml, and legacy migration usage. |
| docs/analysis_particle_detection_workflow.rst | Updates particle detection workflow docs for OME-Zarr and namespaced output paths. |
| docs/AGENTS.md | Updates documentation agent guidance for canonical .ome.zarr, namespaced paths, BDV N5/TensorStore constraints, and migration-only legacy layouts. |
| README.md | Updates top-level project description and runtime contract for OME-Zarr v3, namespaced paths, publication layout, and migration CLI flags. |
| AGENTS.md | Adds repository-level canonical store policy guidance (OME-Zarr v3, migration-only legacy layouts, BDV N5 rules). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+614
to
+633
| ome_selection = self._resolve_ome_array(grp) | ||
| if ome_selection is not None: | ||
| array_path, array, axes = ome_selection | ||
| meta = dict(getattr(grp, "attrs", {})) | ||
| meta.update(dict(getattr(array, "attrs", {}))) | ||
| meta["selected_array_path"] = array_path | ||
| meta["ome_selected"] = True | ||
| if prefer_dask: | ||
| darr = ( | ||
| da.from_zarr(array, chunks=chunks) if chunks else da.from_zarr(array) | ||
| ) | ||
| logger.info(f"Loaded public OME-Zarr array from {path.name}.") | ||
| info = ImageInfo( | ||
| path=path, | ||
| shape=tuple(darr.shape), | ||
| dtype=darr.dtype, | ||
| axes=axes, | ||
| metadata=meta, | ||
| ) | ||
| return darr, info |
There was a problem hiding this comment.
ImageInfo.axes is annotated/documented as Optional[str], but the new OME-Zarr selection path builds axes as Optional[list[str]] and passes it through to ImageInfo(...). This widens the runtime type of axes in a way that can break downstream callers that treat it as a string (e.g., calling string methods) and also makes the type contract inconsistent. Consider normalizing OME axes to a canonical string (e.g., "TCZYX") before storing, or updating ImageInfo.axes (and its docstring) to accept Sequence[str] | str and adjusting callers accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR delivers two runtime regression fixes observed during OME-Zarr migration testing.
voxel_size_resolution_sourcethrough shear, registration, visualization, flatfield, mip export, and OME publish paths.mode="r".src/clearex/shear/README.md.launch_mode=autopolicy to prefersubprocesswhenever a Qt app is active.in_processstays blocking,subprocessstays non-blocking.Key files
src/clearex/io/ome_store.pysrc/clearex/shear/pipeline.pysrc/clearex/registration/pipeline.pysrc/clearex/visualization/pipeline.pysrc/clearex/flatfield/pipeline.pysrc/clearex/mip_export/pipeline.pysrc/clearex/shear/README.mdsrc/clearex/visualization/README.mdtests/io/test_ome_store_scale.pytests/shear/test_pipeline.pytests/registration/test_pipeline.pytests/flatfield/test_pipeline.pytests/visualization/test_pipeline.pyValidation
Scale-lineage patch set
uv run ruff format src/clearex/io/ome_store.py src/clearex/visualization/pipeline.py tests/io/test_ome_store_scale.py tests/shear/test_pipeline.pyuv run ruff check src/clearex/io/ome_store.py src/clearex/visualization/pipeline.py tests/io/test_ome_store_scale.py tests/shear/test_pipeline.pyuv run --with pytest --with requests python -m pytest -q tests/shear/test_pipeline.py tests/io/test_ome_store_scale.py tests/flatfield/test_pipeline.py::test_copy_source_array_attrs_preserves_voxel_size tests/registration/test_pipeline.py::test_extract_voxel_size_uses_source_component_chain tests/visualization/test_pipeline.py::test_run_visualization_analysis_prefers_voxel_size_um_attrs tests/visualization/test_pipeline.py::test_run_visualization_analysis_resolves_scale_from_source_chain tests/visualization/test_pipeline.py::test_launch_napari_viewer_resolves_per_layer_scale_for_downsampled_componentsQt auto-launch non-blocking patch set
uv run ruff format src/clearex/visualization/pipeline.py tests/visualization/test_pipeline.pyuv run ruff check src/clearex/visualization/pipeline.py tests/visualization/test_pipeline.pyuv run --with pytest --with requests python -m pytest -q tests/visualization/test_pipeline.py::test_resolve_effective_launch_mode_auto_prefers_subprocess_with_qt_app tests/visualization/test_pipeline.py::test_resolve_effective_launch_mode_auto_prefers_in_process_without_qt_app tests/visualization/test_pipeline.py::test_run_visualization_analysis_auto_uses_subprocess_with_active_qt_appNotes