Cannot enlarge memory arrays.

[HolgerWillems]

I don't succeed in uploading a 800MB DICOM file to paraview-glance because of memory issues. How can I upload these larger files, without upgrading my RAM or VRAM? Thank you!

Error on a development server on Windows 10, 16GB RAM, 8GB VRAM:

Cannot enlarge memory arrays. Either (1) compile with  -s TOTAL_MEMORY=X  with X higher than the current value 1694498816, (2) compile with  -s ALLOW_MEMORY_GROWTH=1  which allows increasing the size at runtime, or (3) if you want malloc to return NULL (0) instead of this abort, compile with  -s ABORTING_MALLOC=0

Error on Ubuntu 18.04, 32GB RAM, 11GB VRAM:

Exception thrown while writing the image itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
<empty string> itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
itk::MemoryAllocationError (0x956ed0) itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
Location: "unknown" itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
File: /ITK/Modules/Core/Common/include/itkImportImageContainer.hxx itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
uncaught exception: Object
Line: 192 itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
Description: Failed to allocate memory for image. itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
<empty string> 2 itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594

[floryst]

Hi! Seems like you're running up against whatever memory allocation is set for itk.js. @thewtex Do you encounter this issue when you first select your dataset, or after you hit "Load"?

While Glance isn't really meant to view really large datasets due to constraints set by the browser, it should probably work with your Ubuntu machine specs. I don't have an ~800MB dataset on hand right now to see if I can reproduce the issue, but I have loaded 1GB of DICOM data (multiple series) into itk.js before.

[HolgerWillems]

Good to hear that it could be able to work. I encounter this issue while selecting the DICOM. The load button doesn't become active.

[floryst]

Is your data a single series, or multiple series? Multiple files, or one single (multi-frame) file? I might get a better idea of where to look if I know this info.

[thewtex]

@floryst this could likely be resolved, along with faster loading, with your WIP that enables separate series sorting / loading. In this case the pixel data is loaded incrementally, in slabs, in parallel, reducing memory usage.

[HolgerWillems]

@floryst Good suggestion, for practical reasons I always use the single file format. I now also tried the multiple files format and this worked on my 16GB RAM Windows machine. Selecting the file was slow, but once loaded, the interaction was quite fast. So the issue is related to 800MB DICOM single file format, as e.g. 250MB single file format works perfectly.

[thewtex]

If it is a single, multi-frame file, it may help if glance used itk/readImageFile instead of the series reader as indicated by:

itkDICOMImageSeriesReaderJSBindingWasm.js

[HolgerWillems]

Thank you very much for these useful suggestions! I succeeded in uploading a single Dicom file of 800MB with itk/readImageFile, while also keeping the original multifile upload.

I added it like this in my code in ITKReader/ITKDicomImageReader.js.:

  publicAPI.readFileSeries = (files) => {
    if (!files || !files.length || files === model.files) {
      return Promise.resolve();
    }

    model.files = files;
    let promise = null;

    if (files.length === 1) {
      promise = readImageFile(null, files[0]);
    } else {
      promise = readImageDICOMFileSeries(null, files);
    }

[floryst]

That's great! I'm sure others can benefit from what you've added, so if you submit a PR to Glance I can review and merge your changes. :)

[HolgerWillems]

Hi, unfortunately, I am not able to integrate this patch in the most recent version of ParaView-glance as I only fixed it for the older version of "itk": "10.2.1" .

With the new version of "itk": "12.2.0" I experience the problems for 800MB files as stated earlier (see following error), but now for both single file and for multiple file DICOM and the fix doesn't work. It could be that I switch in the future to the newer version of ITK and can perhaps look deeper into it. In any case, thank you!

Exception thrown while writing the image itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
<empty string> itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
itk::MemoryAllocationError (0x94d058) itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
Location: "unknown" itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
File: /ITK/Modules/Core/Common/include/itkImportImageContainer.hxx itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
Line: 192 itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594
Description: Failed to allocate memory for image. itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594

[floryst]

Oh, hm that might be the switch from workerfs to memfs for itk.js virtual filesystem. Will have to investigate further to see how we can handle large files.

[thewtex]

Line: 192 itkDICOMImageSeriesReaderJSBindingWasm.js:9:28594

This is indicating that we are not yet using itk/readImageFile.