Reading nd datasets #966

markbader · 2023-11-23T09:10:32Z

Description:

This PR adds functionality to convert ND Tiff Files to Zarr3 Datasets.
Adds the concept of NDBoundingBoxes to interact with more than three dimensions.
Supports reading and writing ND Zarr3 Datasets and persist their metadata in datasource-properties.json

Issues:

fixes Support reading ND data with webknossos-libs #928

Todos:

Make sure to delete unnecessary points or to check all before merging:

Updated Changelog
Updated Documentation
Added / Updated Tests
Considered adding this to the Examples

webknossos/webknossos/geometry/nd_bounding_box.py

webknossos/webknossos/geometry/vecn_int.py

…ible.

…s images.

…om_images.

…ing the dataset.

…access.

…ween old BoundingBoxes and new NDBoundingBoxes.

…mages.

webknossos/examples/download_tiff_stack.py

webknossos/script_collection/reading_nd_data.py

hotzenklotz

I did a first round of review with some suggestions and many questions.

webknossos/script_collection/reading_nd_data.py

webknossos/tests/dataset/test_add_layer_from_images.py

webknossos/tests/test_examples.py

hotzenklotz · 2024-02-16T09:30:09Z

webknossos/webknossos/dataset/_array.py

@@ -60,6 +70,9 @@ class ArrayInfo:
    voxel_type: np.dtype
    chunk_shape: Vec3Int
    chunks_per_shard: Vec3Int
+    shape: VecInt = VecInt(1, 1, 1, 1)
+    dimension_names: Tuple[str, ...] = ("c", "x", "y", "z")


Why is called dimension_...instead of axis_...? Does it make sense to use a different terminology herre?

axes_... is used in the json, webknossos and the libs so far, dimension_... is terminology of the Zarrita Array create method.

webknossos/webknossos/dataset/_array.py

webknossos/webknossos/dataset/_utils/buffered_slice_reader.py

webknossos/webknossos/dataset/_utils/buffered_slice_writer.py

hotzenklotz · 2024-02-16T14:14:56Z

webknossos/webknossos/dataset/_utils/buffered_slice_writer.py

+                    chunk_bbox.get_3d("topleft") - bbox.get_3d("topleft")
+                )
+                section_bottomright = Vec3Int(
+                    chunk_bbox.get_3d("bottomright") - bbox.get_3d("topleft")


Shouldn't it be ... - bbox.get_3d("bottomright")? or is this an offset in which case chunk_bbox.get_3d("bottomright").offset(...) might make this more clear?

It is an offset. But the Vec3Int class currently doesn't have an offset method implemented. I could apply the offset to the chunk_bbox but, as I need to modify both Vec3Ints again, after adding the offset, I think the readability wouldn't be much better.

webknossos/webknossos/dataset/_utils/buffered_slice_writer.py

…edback.

…cInts.

philippotto

wow, impressive stuff 🥇 kudos for towing this towards the finish line. I left a couple of comments and several questions.

the biggest impact would probably be to add the channel axis to the ND bbox. I don't know whether there was a discussion which decided against this? or is it only because WK doesn't list the channel as an additional coordinate? I think, it could simplify a lot of code if the channel was included in the nd bbox, but it's probably also quite some work to do so...

Overall, I'd like to see a bit more documention:

a small doc section which explains how ND support works. for example, I still don't have a super good grasp of the axes order stuff.
are there breaking changes for users of wklibs? e.g., topleft -> topleft_xyz? this would be quite important to know.

Also, maybe you could add a few more sentences or bullet points to the PR description to explain what was changed in the PR.

webknossos/examples/convert_4d_tiff.py

webknossos/examples/download_segments.py

webknossos/script_collection/globalize_floodfill.py

webknossos/webknossos/cli/export_wkw_as_tiff.py

webknossos/webknossos/dataset/_utils/buffered_slice_reader.py

webknossos/webknossos/dataset/_utils/buffered_slice_writer.py

webknossos/webknossos/dataset/_utils/pims_images.py

webknossos/webknossos/geometry/vec_int.py

…ecks.

hotzenklotz

Let's get this merged and released so we can startet dog fooding these changes.

markbader added 3 commits November 7, 2023 16:58

Started debuging for reading nd datasets.

6269a11

Implementation of VecNInt and NDBoundingBox.

f891606

Merge branch 'master' into reading_nd_datasets

0681252

normanrz reviewed Nov 23, 2023

View reviewed changes

webknossos/webknossos/geometry/nd_bounding_box.py Show resolved Hide resolved

webknossos/webknossos/geometry/vecn_int.py Outdated Show resolved Hide resolved

markbader added 4 commits November 27, 2023 15:40

Rename VecInt and fix some issues.

7cb395e

Work on import of existing zarr dataset.

b3c972e

Merge branch 'master' into reading_nd_datasets

fdf9c03

Add nd_bounding_box to properties of Layer.

89a38e6

hotzenklotz assigned markbader Jan 4, 2024

hotzenklotz added the enhancement label Jan 4, 2024

markbader added 2 commits January 4, 2024 18:00

Update hooks in properties.py to make import of 4d zarr datasets poss…

5580631

…ible.

Add axis_order and index of additional axes to nd_bounding box creation.

c4d2838

fm3 mentioned this pull request Jan 11, 2024

Additional axes bounds as semi-open interval scalableminds/webknossos#7547

Merged

18 tasks

markbader added 17 commits January 11, 2024 18:51

Working on reading nd data with pims images.

cf36c58

Modify pims images to support more _iter_dims.

4033eb0

Add method for expected bounding box instead of expected shape in pim…

ae29105

…s images.

Adding functions for editing ndboundingbox in 3d and update import fr…

f7ffb6b

…om_images.

Propagade nd-bounding box to different methods that take care of writ…

21bb0ea

…ing the dataset.

Update object unstructuring for json and start implementing nd array …

51631c0

…access.

Adapt array classes and add axes information to ArrayInfo.

2a755d3

Updated zarr writing for nd arrays. Axes order still get mixed up bet…

58c728b

…ween old BoundingBoxes and new NDBoundingBoxes.

Adapted buffered_slice_reader and test behaviour with different tif i…

af249e7

…mages.

Adding testdata and fix bugs with axes operation for writing zarr array.

ad3b22e

Fixing issues with axis order.

e975f08

Rewrite buffered_slice_writer and fix bugs to get old tests working.

61146c9

Fix chunking in buffered slice writer.

6fff638

Fix issues with old tests.

6e9efd6

Working on fixing all tests.

0cb8241

Merge branch 'master' into reading_nd_datasets

bb16e83

Fixing issues with alignment, writing with offsets and different mags.

0d8a4f7

hotzenklotz reviewed Feb 16, 2024

View reviewed changes

webknossos/examples/download_tiff_stack.py Outdated Show resolved Hide resolved

hotzenklotz reviewed Feb 16, 2024

View reviewed changes

webknossos/script_collection/reading_nd_data.py Outdated Show resolved Hide resolved

hotzenklotz reviewed Feb 16, 2024

View reviewed changes

webknossos/script_collection/reading_nd_data.py Outdated Show resolved Hide resolved

hotzenklotz requested changes Feb 16, 2024

View reviewed changes

markbader added 13 commits February 20, 2024 01:26

Move script reading_nd_data to examples in docs and implement some fe…

51f173c

…edback.

Do some formatting and implement requested changes.

7340073

Update naming for accessing attributes of nd bounding boxes.

1a94714

Fix buffered_slice_writer for different axis and typechecking.

e1d522f

Fix issue with ensure_size in initialization.

e614525

Merge branch 'master' into reading_nd_datasets

23b992a

Adapt pims_images to support xyz images with channels and timepoint.

27d3a3d

run formatter

8a1ca27

Fix issues with failed tests and add comments.

4167b9b

Fix statement with wrong VecInt initialization.

2f69c30

Insert previously deleted assertions.

31255e3

Merge branch 'master' into reading_nd_datasets

8bedec5

Add docstrings and use bounding box for read in nd case instead of Ve…

c354dd6

…cInts.

philippotto reviewed Mar 12, 2024

View reviewed changes

markbader added 8 commits March 18, 2024 10:06

Implement requested changes.

ffbe7f7

Merge branch 'master' into reading_nd_datasets

36fef79

Changes init of NDBoundingBoxes.

2bfe961

Add converter for VecInt attributes of nd bounding box to pass typech…

0575eb9

…ecks.

Merge branch 'master' into reading_nd_datasets

2ce18df

Merge branch 'master' into reading_nd_datasets

e6781d7

Enhance documentation.

1a51e9b

Update Changelog.md

f21ac09

hotzenklotz approved these changes Apr 2, 2024

View reviewed changes

Merge branch 'master' into reading_nd_datasets

092af26

markbader merged commit c68f9a8 into master Apr 2, 2024
19 checks passed

markbader deleted the reading_nd_datasets branch April 2, 2024 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reading nd datasets #966

Reading nd datasets #966

markbader commented Nov 23, 2023 •

edited by hotzenklotz

Loading

hotzenklotz left a comment

hotzenklotz Feb 16, 2024

markbader Feb 19, 2024

hotzenklotz Feb 16, 2024

markbader Mar 8, 2024

philippotto left a comment

hotzenklotz left a comment

Reading nd datasets #966

Reading nd datasets #966

Conversation

markbader commented Nov 23, 2023 • edited by hotzenklotz Loading

Description:

Issues:

Todos:

hotzenklotz left a comment

Choose a reason for hiding this comment

hotzenklotz Feb 16, 2024

Choose a reason for hiding this comment

markbader Feb 19, 2024

Choose a reason for hiding this comment

hotzenklotz Feb 16, 2024

Choose a reason for hiding this comment

markbader Mar 8, 2024

Choose a reason for hiding this comment

philippotto left a comment

Choose a reason for hiding this comment

hotzenklotz left a comment

Choose a reason for hiding this comment

markbader commented Nov 23, 2023 •

edited by hotzenklotz

Loading