Removing garbage collection speeds up tiffs loading

[zhaobinlidev]

Removing gc.collect() on individual tiffs speeds up converting to binary and suite2p pipeline significantly.

Hi suite2p, I profiled suite2p using python cProfile for loading 4500 individual tiffs and gc.collect() takes up to 1/3 of the whole pipeline (screenshot below). The tiffs sequence are exported using ImageJ from gt1.tif, the sample data on google colab. Removing gc.collect() in suite2p/io/tiff.py speeds up the converting to binary step from 188.14 sec to 3.29 sec, which is similar to the 3.03 sec taken for gt1.tif. The total time reduces from 316.36 sec to 200.82 sec.

Would removing gc.collect() help suite2p? #160 mentioned that garbage collection was added to resolve the memory issue, though I am not sure about the context and I am using a computer with 32gb ram.

[carsen-stringer]

you have single page tiffs that aren't ome's? we do not recommend using single page tiffs unless you have to, like from a bruker scope. if this speeds up reading for bruker we can remove it from the ome_to_binary function

[jmdelahanty]

Related to this, Bruker's newest version of their software implements multipage tiff outputs apparently. Haven't tried it yet...

[zhaobinlidev]

The bruker scope produces single-page ome tiffs. Would be nice to remove the garbage collector

[jmdelahanty]

I just tried the multi-page tiff output from 5.8.64.200 from Prairie View and it looks like it works, just beware @zhaobin-li when upgrading your software... I'm running into some bugs with how triggers work and the reported framerate of the scope is incorrect when compared against an oscilloscope.

Also, converting single page ome tiffs to zarrs/H5 is really speedy with Dask. As a sort of junior IT person in the lab I'm currently in, our data storage/backups greatly appreciate not having to manage thousands of individual files. This doesn't yet retain the metatadata from the ome tiffs, but dumping it to an xml of its own wouldn't be that hard. Spawned from this thread on imagesc. Just figured I'd share this on the thread in case others working with Bruker stuff find it interesting or useful.