partial loading of large ome.tiff files

[bryantChhun]

problem

For very large ome.tiff files, the data loading time can be very long. This is primarily because we use tifffile library to parse the ome.xml structure and tiff page locations. Even if we open as zarr, the page locations need to be found, which requires highly iterative search through the data, even before data is loaded (zarr facilitates the lazy loading when you want the data).

this issue is running documentation of research and attempts at solving this problem

research

I created this issue on tifffile repo: cgohlke/tifffile#83
But the answer is that no high level API exists for this kind of loading and that we'd have to implement this manually.

Here is an example of someone's manual tifffile parsing to extract cropped regions of a large multi-tile image:
https://stackoverflow.com/questions/62747767/how-to-efficiently-read-part-of-a-huge-tiff-image
and the corresponding answer from Christoph
https://gist.github.com/rfezzani/b4b8852c5a48a901c1e94e09feb34743

[bryantChhun]

Consider accessing the pages directly (this requires assumptions about the underlying data, but is more performant than calls to tiff.series). Also, it's possible this is not compatible with micro-manager multi-file, ome-tiff series.

with tifffile.TiffFile(filename) as tif:
    data = tif.pages[pageindex].asarray()

cgohlke/tifffile#52

Would require some rigorous validation and testing, but could boost speed substantially.

[camFoltz]

closed with #39