Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor API for NGFF datasets #31

Merged
merged 111 commits into from
Feb 13, 2023
Merged
Show file tree
Hide file tree
Changes from 82 commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
a63c55f
base class for ngff nodes
ziw-liu Jan 21, 2023
7445eeb
fix typo
ziw-liu Jan 23, 2023
102ba31
sketch image container and position node
ziw-liu Jan 23, 2023
4c8f77d
create new image in position
ziw-liu Jan 26, 2023
f57a368
add docstrings
ziw-liu Jan 26, 2023
c1a4b02
combine ome-zarr reader and writer
ziw-liu Jan 26, 2023
4e42e88
fix docstring length
ziw-liu Jan 26, 2023
245c23c
append channel method
ziw-liu Jan 27, 2023
fe91734
mixin class for file mode methods
ziw-liu Jan 27, 2023
3e81089
handle metadata parsing in node classes
ziw-liu Jan 27, 2023
0b8c9c0
move ome-zarr-py and hypothesis to dev dependency
ziw-liu Jan 27, 2023
fad0225
move hypothesis to dev dependency
ziw-liu Jan 27, 2023
279d04d
match signature
ziw-liu Jan 27, 2023
c25151c
fix constructors
ziw-liu Jan 27, 2023
2db7119
fix metadata writing
ziw-liu Jan 27, 2023
c071835
fix default chunking
ziw-liu Jan 27, 2023
1842570
fix context manager
ziw-liu Jan 27, 2023
63ffda2
fix channel axis check
ziw-liu Jan 27, 2023
eae58b4
fix shape check
ziw-liu Jan 27, 2023
268959a
update OMEZarr example
ziw-liu Jan 27, 2023
4dcbf00
rename ngff file
ziw-liu Jan 27, 2023
dff09f2
rename imports
ziw-liu Jan 27, 2023
beff804
showcase `ImageArray.numpy()`
ziw-liu Jan 27, 2023
09d953e
update docstring
ziw-liu Jan 27, 2023
c9cbe27
remove old writing code and add new well node
ziw-liu Jan 27, 2023
a2f0267
update well constructor and docstring
ziw-liu Jan 27, 2023
0d6fa32
add row node
ziw-liu Jan 27, 2023
641459a
update docstring
ziw-liu Jan 27, 2023
8bef5dc
elevate iterator
ziw-liu Jan 27, 2023
4692bae
dummy metadata parsing for the row node
ziw-liu Jan 27, 2023
74ed6e0
update docstrings
ziw-liu Jan 27, 2023
dc00551
update example introduction
ziw-liu Jan 27, 2023
ca1681a
clarify read-only example
ziw-liu Jan 27, 2023
23e3e0c
fix hcs metadata
ziw-liu Jan 27, 2023
953bc9b
add back dump metadata method as placeholder
ziw-liu Jan 27, 2023
2ce0334
plate node
ziw-liu Jan 30, 2023
763138e
fix typo
ziw-liu Jan 30, 2023
5003887
init plate meta in its own method
ziw-liu Jan 30, 2023
acc800a
update docstring
ziw-liu Jan 30, 2023
dd7e6d7
replace old HCSWriter with a wrapper of plate node
ziw-liu Jan 30, 2023
d6f886a
remove reader adaptor
ziw-liu Jan 30, 2023
a43d71c
remove OMEZarrReader
ziw-liu Jan 30, 2023
a7f792d
update deprecation message
ziw-liu Jan 30, 2023
4330601
update docstring
ziw-liu Jan 30, 2023
a0bdfe7
rename example files
ziw-liu Jan 30, 2023
6a7ee3f
rearrange metadata attributes
ziw-liu Jan 30, 2023
68daef1
fix plate metadata
ziw-liu Jan 30, 2023
b01c5e6
fix metadata loading
ziw-liu Jan 31, 2023
3d62e6f
update hcs example
ziw-liu Jan 31, 2023
2784672
fix omero metadata
ziw-liu Jan 31, 2023
e66390a
rename test directory
ziw-liu Feb 1, 2023
34d20bc
rename test file
ziw-liu Feb 1, 2023
54ac842
change error type
ziw-liu Feb 2, 2023
87f7ee9
test util functions
ziw-liu Feb 2, 2023
d6efd5c
fix linting and formatting
ziw-liu Feb 2, 2023
6444f9c
test OMEZarr
ziw-liu Feb 2, 2023
a893cf0
warning against invalid plate metadata
ziw-liu Feb 2, 2023
ccab859
warn -> warning
ziw-liu Feb 2, 2023
fc2fc2e
warning against channel name not found
ziw-liu Feb 2, 2023
4a7b8cb
test `HCSZarr.open()`
ziw-liu Feb 2, 2023
9ae5ec3
handle missing axes and channel names
ziw-liu Feb 2, 2023
93b57bd
load axes from metadata if possible
ziw-liu Feb 2, 2023
21b89b8
fix check attribute logic
ziw-liu Feb 2, 2023
31d4489
fix typo
ziw-liu Feb 2, 2023
d70248c
test HCSZarr
ziw-liu Feb 2, 2023
e1347a9
return early if position is not found
ziw-liu Feb 2, 2023
403f5ec
match error type
ziw-liu Feb 2, 2023
b83be4d
restrict example count and loosen deadline
ziw-liu Feb 2, 2023
1c3128b
combine pytest commands
ziw-liu Feb 2, 2023
df3a823
fix fixture call
ziw-liu Feb 2, 2023
d104024
remove unused import
ziw-liu Feb 2, 2023
a51ab2a
update example comment
ziw-liu Feb 2, 2023
a0b04ba
fix linting
ziw-liu Feb 2, 2023
0723b33
fix linting
ziw-liu Feb 2, 2023
7a5e9f6
fix linting
ziw-liu Feb 2, 2023
c17b81e
Merge branch 'main' into ngff-dataset
ziw-liu Feb 2, 2023
90cdbc2
Revert "Merge branch 'main' into ngff-dataset"
ziw-liu Feb 2, 2023
9c6d0d2
isort
ziw-liu Feb 2, 2023
686b0af
Merge branch 'main' into ngff-dataset
ziw-liu Feb 2, 2023
0e995fd
remove empty files from merging
ziw-liu Feb 2, 2023
92fa0db
fix linting
ziw-liu Feb 2, 2023
94a20f9
Merge branch 'main' into ngff-dataset
ziw-liu Feb 3, 2023
ea17a61
simple refactors and an example that fails
mattersoflight Feb 8, 2023
4f58916
revert wrong example changes
ziw-liu Feb 8, 2023
efabd82
fix linting and formatting
ziw-liu Feb 8, 2023
7c20657
simplify code
ziw-liu Feb 8, 2023
a02ceba
revert to conditional operator
ziw-liu Feb 8, 2023
2789ea2
Merge branch 'main' into ngff-dataset
ziw-liu Feb 8, 2023
0255beb
use string as array key in examples
ziw-liu Feb 9, 2023
0e0b134
rename OMEZarr to OMEZarrFOV
ziw-liu Feb 9, 2023
919d951
Update docstrings
ziw-liu Feb 9, 2023
c9ee998
test append channel
ziw-liu Feb 9, 2023
036f205
check image shape with test
ziw-liu Feb 9, 2023
8dd85c7
test create position
ziw-liu Feb 10, 2023
9219273
test well metadata
ziw-liu Feb 10, 2023
a02a948
updated hcs_zarr example
mattersoflight Feb 10, 2023
702e945
Merge branch 'ngff-dataset' of https://github.com/czbiohub/iohub into…
mattersoflight Feb 10, 2023
c787958
move open store function to top level
ziw-liu Feb 10, 2023
b655f69
rename open_store to aid auto-completion
ziw-liu Feb 10, 2023
a8300d4
wrap zarr.Group.tree() for ngff nodes
ziw-liu Feb 10, 2023
ae5cd17
remove mixin and container classes
ziw-liu Feb 10, 2023
a4a83ae
test against new api
ziw-liu Feb 10, 2023
6657abb
update examples
ziw-liu Feb 11, 2023
6d0ba4c
refined examples and delete_channel stub
mattersoflight Feb 11, 2023
b9516b1
rename (ome_zarr -> singleFOV_zarr)
mattersoflight Feb 11, 2023
61237f9
proposal for simplified API for tiled acquisition
mattersoflight Feb 11, 2023
e9e83eb
Revert 3 commits
ziw-liu Feb 13, 2023
2293405
rename single fov example file
ziw-liu Feb 13, 2023
b08d107
modify example store path and array name
ziw-liu Feb 13, 2023
7dd36c5
edit HCS example
ziw-liu Feb 13, 2023
9213136
rename hcs example file
ziw-liu Feb 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ jobs:
python -m pip install --upgrade pip
pip install flake8
- name: Check code with Flake8
# E203 conflicts with black
run: |
flake8 iohub
flake8 iohub --extend-ignore=E203

isort:
name: isort Check
Expand All @@ -72,7 +73,7 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install isort
- name: Check code with Flake8
- name: Check code with isort
run: |
isort --check iohub

Expand Down
75 changes: 0 additions & 75 deletions examples/hcs_ome_zarr_writer.py

This file was deleted.

41 changes: 41 additions & 0 deletions examples/hcs_zarr.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# %%
# This script writes a high content screening (HCS) OME-Zarr dataset
# with a single FOV and a single scaling level per well,
# and adds an extra well-position to an existing dataset.
# It can be run as a plain Python script, or as interactive cells in some IDEs.

import numpy as np

from iohub.ngff import HCSZarr

# %%
# Write 5D data to multiple wells

position_list = (
("A", "1", "0"),
("H", 12, 0),
("Control", "Blank", 0),
)

with HCSZarr.open(
"hcs.zarr", mode="a", channel_names=["DAPI", "GFP"]
) as dataset:
for row, col, fov in position_list:
position = dataset.create_position(row, col, fov)
position[0] = np.random.randint(
0, np.iinfo(np.uint16).max, size=(5, 2, 3, 32, 32), dtype=np.uint16
)
ziw-liu marked this conversation as resolved.
Show resolved Hide resolved

# %%
# Append a channel to all the positions

with HCSZarr.open("hcs.zarr", mode="r+") as dataset:
for name, position in dataset.positions():
print(name)
position.append_channel("New", resize_arrays=True)
position[0][:, 2] = np.random.randint(
0, np.iinfo(np.uint16).max, size=(5, 3, 32, 32), dtype=np.uint16
)

# %%
# Try viewing the images with napari-ome-zarr
62 changes: 62 additions & 0 deletions examples/ome_zarr.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# %%
# This script shows how to create a single-FOV, single-scale OME-Zarr dataset,
# read data in read-only mode,
# append an extra time point to an existing dataset,
# and adding a new channel to an existing dataset.
# It can be run as a plain Python script,
# or as interactive cells in some IDEs.

import numpy as np

from iohub.ngff import OMEZarr

# %%
# Write 5D data to a new Zarr store

tczyx = np.random.randint(
0, np.iinfo(np.uint16).max, size=(5, 2, 3, 32, 32), dtype=np.uint16
)

with OMEZarr.open(
"ome.zarr", mode="a", channel_names=["DAPI", "GFP"]
) as dataset:
dataset[0] = tczyx
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since OMEZarr is by definition single-position ND array, the syntax of assigning a 5D array into a zeroth index of dataset is non-intuitive.

Copy link
Collaborator

@mattersoflight mattersoflight Feb 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data structure that mimics access pattern of napari will be lot more intuitive for users.

For example, dataset.channels['DAPI'].data is the numpy array whose shape, size, dtype can be examined.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it is single-position by definition, it is possible (and the intended use case) to have multiple items in the Zarr store. For example each resolution level in a pyramid is an array, and the labels are in a member Zarr group.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the NGFF specification and the current API of an OMEZarr dataset is identical to a position in the HCS hierarchy. In fact, a position in an HCS Zarr store can be opened in iohub and napari-ome-zarr as if it was a standalone dataset. I think it is beneficial to keep them synchronized

Copy link
Collaborator

@mattersoflight mattersoflight Feb 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per our discussion,

  • Update the example to use a string key rather than an integer index when assigning an array to a group or reading an array from a group.
  • Consider more intuitive names for OMEZarr and HCSZarr classes to imply single-position and multi-position groups or arrays.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed OMEZarr to OMEZarrFOV to show its single-FOV nature, and edited docstring to further explain its intended use.


# %%
# Opening in read-only mode prevents writing

with OMEZarr.open("ome.zarr", mode="r") as dataset:
img = dataset[0]
print(img.numpy())
try:
img[0, 0, 0, 0, 0] = 0
except Exception as e:
print(f"Writing was rejected: {e}")

# %%
# Append a new timepoint to an existing dataset

new_1czyx = np.random.randint(
0, np.iinfo(np.uint16).max, size=(1, 2, 3, 32, 32), dtype=np.uint16
)

with OMEZarr.open("ome.zarr", mode="r+") as dataset:
img = dataset[0]
print(img.shape)
img.append(new_1czyx, axis=0)
print(img.shape)

# %%
# Add a new channel and write a Z-stack

new_zyx = np.random.randint(
0, np.iinfo(np.uint16).max, size=(3, 32, 32), dtype=np.uint16
)

dataset = OMEZarr.open("ome.zarr", mode="r+")
dataset.append_channel("New", resize_arrays=True)
dataset[0][0, 2] = new_zyx
dataset.close()

# %%
# Try viewing the images with napari-ome-zarr
70 changes: 0 additions & 70 deletions examples/ome_zarr_writer.py

This file was deleted.

7 changes: 5 additions & 2 deletions iohub/lf_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ def channel_display_settings(
clim: Tuple[float, float, float, float] = None,
first_chan: bool = False,
):
"""This will create a dictionary used for OME-zarr metadata. Allows custom contrast limits and channel
"""This will create a dictionary used for OME-zarr metadata.
Allows custom contrast limits and channel.
names for display. Defaults everything to grayscale.

Parameters
Expand All @@ -22,7 +23,9 @@ def channel_display_settings(
clim : Tuple[float, float, float, float], optional
Contrast limits (start, end, min, max)
first_chan : bool, optional
Whether or not this is the first channel of the dataset (display will be set to active), by default False
Whether or not this is the first channel of the dataset
(display will be set to active),
by default False

Returns
-------
Expand Down
Loading