java.lang.OutOfMemoryError: Java heap space, when displaying (open virtual) large (>20gb) ndtiff stores (recordings)

[LukasMicroscopy]

Hi!
When opening large ndtiff recordings (1000 frames 5000x5000px) or creating a display (createDisplay(ndtiffstore)) for the ndtiffstore a lot of memory is needed, or the system runs out of memory.

Example:
Record 1000 frames with Camera-OnCameraCCDXSize:5120, Camera-OnCameraCCDYSize:5120 (MMConfig_Demo.cfg). When trying to open with open(virtual) the system runs out of memory (>20Gb memory usage).

[UI]  "Open" clicked in JDialog.
2023-11-16T14:09:25.184986 tid6000 [IFO,App] Class: class org.micromanager.display.internal.displaywindow.DisplayController
2023-11-16T14:09:25.185022 tid6000 [IFO,App] Classloader: ij.io.PluginClassLoader@23594b45
2023-11-16T14:09:29.855803 tid8216 [dbg,App] [UI] JMenu "File" clicked in MainFrame.
2023-11-16T14:09:30.223334 tid6000 [IFO,App] Thread 150 (Thread-8) terminated with uncaught exception
2023-11-16T14:09:30.223367 tid6000 [IFO,App] java.lang.OutOfMemoryError: Java heap space
2023-11-16T14:09:30.223511 tid6000 [IFO,App]   at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
2023-11-16T14:09:30.223514 tid6000 [IFO,App]   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
2023-11-16T14:09:30.223517 tid6000 [IFO,App]   at org.micromanager.ndtiffstorage.NDTiffReader.readTaggedImage(NDTiffReader.java:299)
2023-11-16T14:09:30.223521 tid6000 [IFO,App]   at org.micromanager.ndtiffstorage.NDTiffReader.readImage(NDTiffReader.java:173)
2023-11-16T14:09:30.223522 tid6000 [IFO,App]   at org.micromanager.ndtiffstorage.ResolutionLevel.getImage(ResolutionLevel.java:178)
2023-11-16T14:09:30.223524 tid6000 [IFO,App]   at org.micromanager.ndtiffstorage.NDTiffStorage.getImage(NDTiffStorage.java:996)
2023-11-16T14:09:30.223525 tid6000 [IFO,App]   at org.micromanager.ndtiffstorage.NDTiffStorage.getImage(NDTiffStorage.java:943)
2023-11-16T14:09:30.223529 tid6000 [IFO,App]   at org.micromanager.data.internal.ndtiff.NDTiffAdapter$1.apply(NDTiffAdapter.java:244)
2023-11-16T14:09:30.223530 tid6000 [IFO,App]   at org.micromanager.data.internal.ndtiff.NDTiffAdapter$1.apply(NDTiffAdapter.java:241)
2023-11-16T14:09:30.223531 tid6000 [IFO,App]   at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
2023-11-16T14:09:30.223534 tid6000 [IFO,App]   at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
2023-11-16T14:09:30.223536 tid6000 [IFO,App]   at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3527)
2023-11-16T14:09:30.223537 tid6000 [IFO,App]   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
2023-11-16T14:09:30.223540 tid6000 [IFO,App]   at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
2023-11-16T14:09:30.223542 tid6000 [IFO,App]   at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
2023-11-16T14:09:30.223544 tid6000 [IFO,App]   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2023-11-16T14:09:30.223547 tid6000 [IFO,App]   at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
2023-11-16T14:09:30.223549 tid6000 [IFO,App]   at org.micromanager.data.internal.ndtiff.NDTiffAdapter.getImagesMatching(NDTiffAdapter.java:248)
2023-11-16T14:09:30.223550 tid6000 [IFO,App]   at org.micromanager.data.internal.ndtiff.NDTiffAdapter.getImagesIgnoringAxes(NDTiffAdapter.java:256)
2023-11-16T14:09:30.223553 tid6000 [IFO,App]   at org.micromanager.data.internal.DefaultDatastore.getImagesIgnoringAxes(DefaultDatastore.java:204)
2023-11-16T14:09:30.223555 tid6000 [IFO,App]   at org.micromanager.display.internal.displaywindow.DisplayController.handleDisplayPosition(DisplayController.java:561)
2023-11-16T14:09:30.223558 tid6000 [IFO,App]   at org.micromanager.display.AbstractDataViewer.setDisplayPosition(AbstractDataViewer.java:229)
2023-11-16T14:09:30.223560 tid6000 [IFO,App]   at org.micromanager.display.AbstractDataViewer.setDisplayPosition(AbstractDataViewer.java:242)
2023-11-16T14:09:30.223561 tid6000 [IFO,App]   at org.micromanager.display.internal.displaywindow.DisplayController.create(DisplayController.java:241)
2023-11-16T14:09:30.223565 tid6000 [IFO,App]   at org.micromanager.display.internal.displaywindow.DisplayController.access$000(DisplayController.java:86)
2023-11-16T14:09:30.223566 tid6000 [IFO,App]   at org.micromanager.display.internal.displaywindow.DisplayController$Builder.build(DisplayController.java:211)
2023-11-16T14:09:30.223569 tid6000 [IFO,App]   at org.micromanager.display.internal.DefaultDisplayManager.loadDisplays(DefaultDisplayManager.java:364)
2023-11-16T14:09:30.223571 tid6000 [IFO,App]   at org.micromanager.internal.menus.FileMenu.lambda$promptToOpenFile$4(FileMenu.java:117)
2023-11-16T14:09:30.223573 tid6000 [IFO,App]   at org.micromanager.internal.menus.FileMenu$$Lambda$233/1515206427.run(Unknown Source)
2023-11-16T14:09:30.223576 tid6000 [IFO,App]   at java.lang.Thread.run(Thread.java:748)

[nicost]

This seems to be due to this function:
https://github.com/micro-manager/micro-manager/blob/main/mmstudio/src/main/java/org/micromanager/data/internal/ndtiff/NDTiffAdapter.java#L226
@henrypinkard, any ideas how to get a catalog of the images in the data sets (as defined by their axes), without actually reading the complete dataset?

[nicost]

Possible fix in PR #1840

[henrypinkard]

I think the problem is that that function is reading all of the images, rather than just their metadata. I just added a function to the java NDTiff API to read the metadata, without reading the pixels:

micro-manager/NDStorage#142