Memory efficient writing to TIFF and pyramidal tiffs

[eddienko]

Hi!

I wanted to discuss two (somewhat related) issues. At the moment we need to load the whole array into memory in order to save it to TIFF. I would like to have an option to write by chunks, this would be memory efficient, but slower since we cannot write in parallel to TIFF.

Related to this, I would also like to have support for writing pyramidal tiffs, so I can use applications that support them (e.g. QuPath) to visualise really large datasets. Writing the pyramidal structure would again be memory efficient at expense of speed.

I have code that does this, but I wonder (and hence this issue) if that is something that you want added to xtiff.

Cheers

[jwindhager]

Hi Ed, thanks a lot for opening this issue!

Having support for writing chunks would be lovely. The to_tiff function in xtiff merely is a simplified interface to tifffile with restrictions on the input data and additional sanity checks - if there is a way to get chunking to work with tifffile that would be awesome! If I remember correctly, tifffile now supports zarr, maybe that would be an option to look into?

Re pyramidal TIFFs: this is definitely supported by the tifffile, but not (yet) by xtiff. Would certainly be nice to have for xtiff as well!

I'd be more than happy to take pull requests in both cases. Only thing to keep in mind is that it's key to xtiff to keep the to_tiff signature as simple and straight-forward as possible (one can always resort to tifffile for more complex use cases). Please ping me anytime if you have questions or would like to discuss!

[jwindhager]

Closing due to inactivity