-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor API for NGFF datasets #31
Merged
Merged
Changes from 83 commits
Commits
Show all changes
111 commits
Select commit
Hold shift + click to select a range
a63c55f
base class for ngff nodes
ziw-liu 7445eeb
fix typo
ziw-liu 102ba31
sketch image container and position node
ziw-liu 4c8f77d
create new image in position
ziw-liu f57a368
add docstrings
ziw-liu c1a4b02
combine ome-zarr reader and writer
ziw-liu 4e42e88
fix docstring length
ziw-liu 245c23c
append channel method
ziw-liu fe91734
mixin class for file mode methods
ziw-liu 3e81089
handle metadata parsing in node classes
ziw-liu 0b8c9c0
move ome-zarr-py and hypothesis to dev dependency
ziw-liu fad0225
move hypothesis to dev dependency
ziw-liu 279d04d
match signature
ziw-liu c25151c
fix constructors
ziw-liu 2db7119
fix metadata writing
ziw-liu c071835
fix default chunking
ziw-liu 1842570
fix context manager
ziw-liu 63ffda2
fix channel axis check
ziw-liu eae58b4
fix shape check
ziw-liu 268959a
update OMEZarr example
ziw-liu 4dcbf00
rename ngff file
ziw-liu dff09f2
rename imports
ziw-liu beff804
showcase `ImageArray.numpy()`
ziw-liu 09d953e
update docstring
ziw-liu c9cbe27
remove old writing code and add new well node
ziw-liu a2f0267
update well constructor and docstring
ziw-liu 0d6fa32
add row node
ziw-liu 641459a
update docstring
ziw-liu 8bef5dc
elevate iterator
ziw-liu 4692bae
dummy metadata parsing for the row node
ziw-liu 74ed6e0
update docstrings
ziw-liu dc00551
update example introduction
ziw-liu ca1681a
clarify read-only example
ziw-liu 23e3e0c
fix hcs metadata
ziw-liu 953bc9b
add back dump metadata method as placeholder
ziw-liu 2ce0334
plate node
ziw-liu 763138e
fix typo
ziw-liu 5003887
init plate meta in its own method
ziw-liu acc800a
update docstring
ziw-liu dd7e6d7
replace old HCSWriter with a wrapper of plate node
ziw-liu d6f886a
remove reader adaptor
ziw-liu a43d71c
remove OMEZarrReader
ziw-liu a7f792d
update deprecation message
ziw-liu 4330601
update docstring
ziw-liu a0bdfe7
rename example files
ziw-liu 6a7ee3f
rearrange metadata attributes
ziw-liu 68daef1
fix plate metadata
ziw-liu b01c5e6
fix metadata loading
ziw-liu 3d62e6f
update hcs example
ziw-liu 2784672
fix omero metadata
ziw-liu e66390a
rename test directory
ziw-liu 34d20bc
rename test file
ziw-liu 54ac842
change error type
ziw-liu 87f7ee9
test util functions
ziw-liu d6efd5c
fix linting and formatting
ziw-liu 6444f9c
test OMEZarr
ziw-liu a893cf0
warning against invalid plate metadata
ziw-liu ccab859
warn -> warning
ziw-liu fc2fc2e
warning against channel name not found
ziw-liu 4a7b8cb
test `HCSZarr.open()`
ziw-liu 9ae5ec3
handle missing axes and channel names
ziw-liu 93b57bd
load axes from metadata if possible
ziw-liu 21b89b8
fix check attribute logic
ziw-liu 31d4489
fix typo
ziw-liu d70248c
test HCSZarr
ziw-liu e1347a9
return early if position is not found
ziw-liu 403f5ec
match error type
ziw-liu b83be4d
restrict example count and loosen deadline
ziw-liu 1c3128b
combine pytest commands
ziw-liu df3a823
fix fixture call
ziw-liu d104024
remove unused import
ziw-liu a51ab2a
update example comment
ziw-liu a0b04ba
fix linting
ziw-liu 0723b33
fix linting
ziw-liu 7a5e9f6
fix linting
ziw-liu c17b81e
Merge branch 'main' into ngff-dataset
ziw-liu 90cdbc2
Revert "Merge branch 'main' into ngff-dataset"
ziw-liu 9c6d0d2
isort
ziw-liu 686b0af
Merge branch 'main' into ngff-dataset
ziw-liu 0e995fd
remove empty files from merging
ziw-liu 92fa0db
fix linting
ziw-liu 94a20f9
Merge branch 'main' into ngff-dataset
ziw-liu ea17a61
simple refactors and an example that fails
mattersoflight 4f58916
revert wrong example changes
ziw-liu efabd82
fix linting and formatting
ziw-liu 7c20657
simplify code
ziw-liu a02ceba
revert to conditional operator
ziw-liu 2789ea2
Merge branch 'main' into ngff-dataset
ziw-liu 0255beb
use string as array key in examples
ziw-liu 0e0b134
rename OMEZarr to OMEZarrFOV
ziw-liu 919d951
Update docstrings
ziw-liu c9ee998
test append channel
ziw-liu 036f205
check image shape with test
ziw-liu 8dd85c7
test create position
ziw-liu 9219273
test well metadata
ziw-liu a02a948
updated hcs_zarr example
mattersoflight 702e945
Merge branch 'ngff-dataset' of https://github.com/czbiohub/iohub into…
mattersoflight c787958
move open store function to top level
ziw-liu b655f69
rename open_store to aid auto-completion
ziw-liu a8300d4
wrap zarr.Group.tree() for ngff nodes
ziw-liu ae5cd17
remove mixin and container classes
ziw-liu a4a83ae
test against new api
ziw-liu 6657abb
update examples
ziw-liu 6d0ba4c
refined examples and delete_channel stub
mattersoflight b9516b1
rename (ome_zarr -> singleFOV_zarr)
mattersoflight 61237f9
proposal for simplified API for tiled acquisition
mattersoflight e9e83eb
Revert 3 commits
ziw-liu 2293405
rename single fov example file
ziw-liu b08d107
modify example store path and array name
ziw-liu 7dd36c5
edit HCS example
ziw-liu 9213136
rename hcs example file
ziw-liu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
# %% | ||
# This script writes a high content screening (HCS) OME-Zarr dataset | ||
# with a single FOV and a single scaling level per well, | ||
# and adds an extra well-position to an existing dataset. | ||
# It can be run as a plain Python script, or as interactive cells in some IDEs. | ||
|
||
import numpy as np | ||
|
||
from iohub.ngff import HCSZarr | ||
|
||
# %% | ||
# Write 5D data to multiple wells | ||
|
||
position_list = ( | ||
("A", "1", "0"), | ||
("H", 12, 0), | ||
("Control", "Blank", 0), | ||
) | ||
|
||
with HCSZarr.open( | ||
"hcs_wrong.zarr", mode="a", channel_names=["DAPI", "GFP"] | ||
) as dataset: | ||
for row, col, fov in position_list: | ||
position = dataset.create_position(row, col, fov) | ||
position[0] = np.random.randint( | ||
0, np.iinfo(np.uint16).max, size=(5, 3, 3, 32, 32), dtype=np.uint16 | ||
) | ||
|
||
# %% | ||
# Append a channel to all the positions | ||
|
||
with HCSZarr.open("hcs.zarr", mode="r+") as dataset: | ||
for name, position in dataset.positions(): | ||
print(name) | ||
position.append_channel("New", resize_arrays=True) | ||
position[0][:, 2] = np.random.randint( | ||
0, np.iinfo(np.uint16).max, size=(5, 3, 32, 32), dtype=np.uint16 | ||
) | ||
|
||
# %% | ||
# Try viewing the images with napari-ome-zarr |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
# %% | ||
# This script shows how to create a single-FOV, single-scale OME-Zarr dataset, | ||
# read data in read-only mode, | ||
# append an extra time point to an existing dataset, | ||
# and adding a new channel to an existing dataset. | ||
# It can be run as a plain Python script, | ||
# or as interactive cells in some IDEs. | ||
|
||
import numpy as np | ||
|
||
from iohub.ngff import OMEZarr | ||
|
||
# %% | ||
# Write 5D data to a new Zarr store | ||
|
||
tczyx = np.random.randint( | ||
0, np.iinfo(np.uint16).max, size=(5, 2, 3, 32, 32), dtype=np.uint16 | ||
) | ||
|
||
with OMEZarr.open( | ||
"ome.zarr", mode="a", channel_names=["DAPI", "GFP"] | ||
) as dataset: | ||
dataset[0] = tczyx | ||
|
||
# %% | ||
# Opening in read-only mode prevents writing | ||
|
||
with OMEZarr.open("ome.zarr", mode="r") as dataset: | ||
img = dataset[0] | ||
print(img.numpy()) | ||
try: | ||
img[0, 0, 0, 0, 0] = 0 | ||
except Exception as e: | ||
print(f"Writing was rejected: {e}") | ||
|
||
# %% | ||
# Append a new timepoint to an existing dataset | ||
|
||
new_1czyx = np.random.randint( | ||
0, np.iinfo(np.uint16).max, size=(1, 2, 3, 32, 32), dtype=np.uint16 | ||
) | ||
|
||
with OMEZarr.open("ome.zarr", mode="r+") as dataset: | ||
img = dataset[0] | ||
print(img.shape) | ||
img.append(new_1czyx, axis=0) | ||
print(img.shape) | ||
|
||
# %% | ||
# Add a new channel and write a Z-stack | ||
|
||
new_zyx = np.random.randint( | ||
0, np.iinfo(np.uint16).max, size=(3, 32, 32), dtype=np.uint16 | ||
) | ||
|
||
dataset = OMEZarr.open("ome.zarr", mode="r+") | ||
dataset.append_channel("New", resize_arrays=True) | ||
dataset[0][0, 2] = new_zyx | ||
dataset.close() | ||
|
||
# %% | ||
# Try viewing the images with napari-ome-zarr |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since OMEZarr is by definition single-position ND array, the syntax of assigning a 5D array into a zeroth index of dataset is non-intuitive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The data structure that mimics access pattern of napari will be lot more intuitive for users.
For example,
dataset.channels['DAPI'].data
is the numpy array whose shape, size, dtype can be examined.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While it is single-position by definition, it is possible (and the intended use case) to have multiple items in the Zarr store. For example each resolution level in a pyramid is an array, and the labels are in a member Zarr group.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, the NGFF specification and the current API of an
OMEZarr
dataset is identical to a position in the HCS hierarchy. In fact, a position in an HCS Zarr store can be opened in iohub and napari-ome-zarr as if it was a standalone dataset. I think it is beneficial to keep them synchronizedThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Per our discussion,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I renamed
OMEZarr
toOMEZarrFOV
to show its single-FOV nature, and edited docstring to further explain its intended use.