Refactor API for NGFF datasets #31

ziw-liu · 2023-01-27T04:52:31Z

Implements #19.
Fixes #29.

ziw-liu · 2023-01-27T04:56:03Z

At this point only iohub.ngff.OMEZarr works. (See examples/ome_zarr_writer.py and docstrings for intended usage patterns.)

Again early testing and suggestions are most welcome!

mattersoflight

Great iteration! The open_ome_zarr method with layout parameter is intuitive!
I have renamed & edited examples for clarity. Comments below are related to the following points.

Repurpose the nomenclature from our dependencies:

Accessing the 5D arrays (zarr arrays) via position['key'] syntax is not intuitive. Let's implement position.data (set or get property) syntax, which mimics napari's syntax.
Rename the ImageArray object to ZarrArray to make it obvious that the object is a zarr array.

Make it intuitive to read and write tiled acquisitions:

Introduce a tiles or tiled layout. I picked these words in favor of multiPos or multiFOV. But, I think either of them is a decent choice.

Make it intuitive to over-write channels:

setitem (position[array_name][:, channel_index] = array) is too complex a usage, let's provide a wrapper delete_channel or overwrite_channel. delete_channel is more broadly useful.

iohub/ngff.py

ziw-liu · 2023-02-11T22:46:50Z

@mattersoflight Thanks for the detailed review. At this point I think this PR has grown too large (100+ commits) and further refactoring makes it difficult to trace the history. I'm tempted to move your comments/edits to a new issue/branch pair and merge this into main at 6657abb.

mattersoflight · 2023-02-11T23:57:03Z

@mattersoflight Thanks for the detailed review. At this point, I think this PR has grown too large (100+ commits) and further refactoring makes it difficult to trace the history. I'm tempted to move your comments/edits to a new issue/branch pair and merge this into main at 6657abb.

You're welcome! If you want to take up improvements in API into a separate issue and PR, feel free.
Just refine the three examples (single FOV, hcs, tiled zarrs) with the current API and include them in this PR, starting at 61237f9

ziw-liu · 2023-02-13T17:15:52Z

Moved commits after 6657abb to ngff-api.

This reverts commits 6d0ba4c, b9516b1, 61237f9.

Calibration/Acquisition Plugin

* Merge pull request #14 from mehta-lab/QLIPP_pipeline recOrder pipeline infrastructure + qlipp pipeline submodule * Merge pull request #22 from mehta-lab/zarr_converter Zarr converter * Merge pull request #52 from mehta-lab/fluor_deconv Fluorescence Deconvolution and PhaseFromBF pipelines * Merge pull request #31 from mehta-lab/calibration_plugin Calibration/Acquisition Plugin * Merge pull request #112 from mehta-lab/cleanup-tests-and-readme Improved testing, configs, and dev tools * Merge pull request #123 from mehta-lab/zarr-converter-position-bug Fix ome-tif to zarr converter for acquisitions with micromanager beta * Merge pull request #140 from mehta-lab/restructure_tests Restructure tests * Merge pull request #148 from mehta-lab/pycromanager_converter_v2 Pycromanager converter v2 * Merge pull request #167 from mehta-lab/bkg-correction-fixes2 Fix background correction for non-square images and online mode * Merge pull request #157 from mehta-lab/RAM-warning Print RAM warning message in online and offline modes * Merge pull request #169 from mehta-lab/warn-bkg-averaging Remove background averaging * Merge pull request #174 from mehta-lab/show-ram-warning Show RAM warning in GUI / log in CLI * Merge pull request #175 from mehta-lab/better-messages Improved warning message * Merge pull request #181 from mehta-lab/error-mismatched-bkg-and-img Always load background and warn if background and image sizes are mismatched * Merge pull request #182 from mehta-lab/error-mismatched-bkg-and-img Mismatched x/y size of background and acquisition should error instead of warn * Merge pull request #210 from mehta-lab/0.2.0-gui-hide Remove fluorescence, preprocessing, and postprocessing; simplify GUI * Merge pull request #219 from mehta-lab/0.2.0-integration 0.2.0 integration PR * Introduce Black formatting into the code base (#229) * black-format all `.py` files except recorder_ui.py * create .git-blame-ignore-revs * Document how to ignore formatting commits * Document `git` version Co-authored-by: Talon Chandler <talonchandler@gmail.com> * draft refactor for imread * expose primary entry points * fix type check and tests Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * fix formatting * update `imread` docstring * define supported formats Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * rename pycromanager reader to ndtiff Pycromanager is no longer a dependency, and MM can also write NDTIFF. Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * remove 'doesnt work' code * improve error messages * rearrange files from patch * depend on tqdm for progress bar in conversion * move util function * delete other util functions * make util function private * remove redundant file * rename converter file * fix some of the references and docstrings * draft refactor * write with new ngff module * fix position grid * fix image check * close writer after conversion * test ome-tiff conversion * handle invalid stage position metadata * support labelling positions * fix typo * fix path type * convert single page tiff datasets This is rather broken: the reader does not handle metadata well * allow longer conversion time in tests Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * draft cli command Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * fix argument * detect flat ndtiff * sort imports Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> * limit bar length * add -h shortcut for help * test cli * fix get_image * test against a random choice of dataset --------- Signed-off-by: Ziwen Liu <67518483+ziw-liu@users.noreply.github.com> Co-authored-by: Cameron Foltz <56048121+camFoltz@users.noreply.github.com> Co-authored-by: Shalin Mehta <shalin.mehta@gmail.com> Co-authored-by: Talon Chandler <talonchandler@gmail.com> Co-authored-by: Ivan Ivanov <ivan.ivanov@czbiohub.org>

ziw-liu added 22 commits January 20, 2023 20:09

base class for ngff nodes

a63c55f

fix typo

7445eeb

sketch image container and position node

102ba31

create new image in position

4c8f77d

add docstrings

f57a368

combine ome-zarr reader and writer

c1a4b02

fix docstring length

4e42e88

append channel method

245c23c

mixin class for file mode methods

fe91734

handle metadata parsing in node classes

3e81089

move ome-zarr-py and hypothesis to dev dependency

0b8c9c0

move hypothesis to dev dependency

fad0225

match signature

279d04d

fix constructors

c25151c

fix metadata writing

2db7119

fix default chunking

c071835

fix context manager

1842570

fix channel axis check

63ffda2

fix shape check

eae58b4

update OMEZarr example

268959a

rename ngff file

4dcbf00

rename imports

dff09f2

ziw-liu added enhancement New feature or request NGFF OME-NGFF (OME-Zarr format) labels Jan 27, 2023

ziw-liu self-assigned this Jan 27, 2023

ziw-liu added this to the 0.0.1 milestone Jan 27, 2023

ziw-liu added 3 commits January 26, 2023 21:00

showcase ImageArray.numpy()

beff804

update docstring

09d953e

remove old writing code and add new well node

c9cbe27

ziw-liu added 6 commits February 10, 2023 14:31

move open store function to top level

c787958

rename open_store to aid auto-completion

b655f69

wrap zarr.Group.tree() for ngff nodes

a8300d4

remove mixin and container classes

ae5cd17

test against new api

a4a83ae

update examples

6657abb

ziw-liu requested a review from mattersoflight February 11, 2023 00:27

mattersoflight added 3 commits February 11, 2023 12:03

refined examples and delete_channel stub

6d0ba4c

rename (ome_zarr -> singleFOV_zarr)

b9516b1

proposal for simplified API for tiled acquisition

61237f9

mattersoflight reviewed Feb 11, 2023

View reviewed changes

iohub/ngff.py Show resolved Hide resolved

iohub/ngff.py Outdated Show resolved Hide resolved

iohub/ngff.py Show resolved Hide resolved

mattersoflight approved these changes Feb 11, 2023

View reviewed changes

Revert 3 commits

e9e83eb

This reverts commits 6d0ba4c, b9516b1, 61237f9.

ziw-liu mentioned this pull request Feb 13, 2023

Improve API for the NGFF module #45

Closed

3 tasks

ziw-liu added 4 commits February 13, 2023 09:39

rename single fov example file

2293405

modify example store path and array name

b08d107

edit HCS example

7dd36c5

rename hcs example file

9213136

ziw-liu merged commit 4a9e6fc into main Feb 13, 2023

ziw-liu deleted the ngff-dataset branch February 13, 2023 22:08

This was referenced Feb 15, 2023

Rename WaveorderReader to ImageReader #46

Merged

Support microDL data IO #10

Closed

Method to delete or rename channel in NGFF datasets #56

Closed

WaveorderWriter is not compatible with napari-ome-zarr reader mehta-lab/waveorder#94

Closed

ziw-liu pushed a commit that referenced this pull request Mar 3, 2023

Merge pull request #31 from mehta-lab/calibration_plugin

e02043f

Calibration/Acquisition Plugin

ziw-liu mentioned this pull request Apr 17, 2023

Add code coverage check on PRs #115

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor API for NGFF datasets #31

Refactor API for NGFF datasets #31

ziw-liu commented Jan 27, 2023

ziw-liu commented Jan 27, 2023

mattersoflight left a comment •

edited

ziw-liu commented Feb 11, 2023

mattersoflight commented Feb 11, 2023 •

edited

ziw-liu commented Feb 13, 2023

Refactor API for NGFF datasets #31

Refactor API for NGFF datasets #31

Conversation

ziw-liu commented Jan 27, 2023

ziw-liu commented Jan 27, 2023

mattersoflight left a comment • edited

Choose a reason for hiding this comment

ziw-liu commented Feb 11, 2023

mattersoflight commented Feb 11, 2023 • edited

ziw-liu commented Feb 13, 2023

mattersoflight left a comment •

edited

mattersoflight commented Feb 11, 2023 •

edited