Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NumPyDataLoader for reading .npy and .npz files #19

Merged
merged 1 commit into from
Mar 21, 2024
Merged

Add NumPyDataLoader for reading .npy and .npz files #19

merged 1 commit into from
Mar 21, 2024

Conversation

jcfr
Copy link
Contributor

@jcfr jcfr commented Aug 1, 2023

This pull request was created based on these comments:

to follow up on Slicer forum discussion1 related to import of NumPy file as image volume.

The .npy or npz file may contain an array of dimension between 1 to 5 (axes: time, K, J, I, component).

Notes:

  • User is responsible for setting the correct IJK to RAS matrix.
  • For the 4D and 5D cases, having the channel-last convention corresponds to the ITK/VTK memory layout and is different from the convention used in PyTorch for NCHW for 4D tensors/arrays and NCDHW for 5D.

Based on:

Footnotes

  1. https://discourse.slicer.org/t/import-numpy-file-in-as-image-volume/3653/5 ↩

@dzenanz @lassoan @jcfr
ENH: Add NumPyDataLoader for reading .npy and .npz files
7d8fc63 
The `.npy` or `npz` file may contain an array of dimension
between 1 to 5 (axes: time, K, J, I, component).

The `NumPyDataLoader` reader was created to follow-up on Slicer forum
discussions.
See https://discourse.slicer.org/t/import-numpy-file-in-as-image-volume/3653/5

Notes:
* User is responsible for setting the correct IJK to RAS matrix.
* For the 4D and 5D cases, having the channel-last convention corresponds to the
  ITK/VTK memory layout and is different from the convention used in `PyTorch`
  for [NCHW for 4D][NCHW] tensors/arrays and [NCDHW for 5D][NCDHW].

[NCHW]: https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d
[NCDHW]: https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#torch.nn.Conv3d

Based on:
* https://github.com/pieper/SlicerDMRI/blob/9565f89d16cd72618cd87f1c9046542450e4937b/Modules/Scripted/NIfTIFile/NIfTIFile.py
* https://github.com/Slicer/Slicer/blob/9390be6bf76e048e6fd384d8bfd607eba2857b5a/Applications/SlicerApp/Testing/Python/SlicerScriptedFileReaderWriterTest.py

Co-authored-by: Andras Lasso <lasso@queensu.ca>
Co-authored-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>
@jcfr
Copy link
Contributor Author

jcfr commented Aug 1, 2023 •
edited
Loading

Summary of changes compared to version proposed in Slicer/Slicer#6733 (Slicer/Slicer@aaba225):

  • Renamed from NumpyDataLoader to NumPyDataLoader
  • Split test functions by dimension.
  • Updated CMakeLists.txt adding call to slicer_add_python_unittest, this ensures the tests are executed using ctest
  • Renamed function from testWriteReadOne to _testWriteReadOne, this avoids the helper function from being discovered as unit test
  • Updated helper function _testWriteReadOne to
    • check for expected input dimension
    • avoid catching exceptions. This allows to use assertRaisesRegex and check for exception message.
  • Updated 0D to test to ensure a 0 dimensional input is given (np.array(0) instead of np.array([]))

This was referenced Aug 1, 2023
Closed
Merged
@jcfr
Copy link
Contributor Author

jcfr commented Aug 1, 2023 •
edited
Loading

I confirm that running the tests on Ubuntu 20.04 against Slicer/Slicer@ce1ab3d work as expected:

$ctest -R "(py_ImportNumPyArray|py_nomainwindow_qSlicerImportNumPyArrayModuleGenericTest)" -VV
[...]
test 11
    Start 11: py_nomainwindow_qSlicerImportNumPyArrayModuleGenericTest

11: Test command: /home/jcfr/Projects/Slicer-Release/Slicer-build/Slicer "--no-splash" "--testing" "--launcher-additional-settings" "/tmp/SlicerSandbox-Release/AdditionalLauncherSettings.ini" "--no-main-window" "--disable-cli-modules" "--additional-module-path" "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-scripted-modules" "--additional-module-paths" "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-scripted-modules" "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-loadable-modules" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['/tmp/SlicerSandbox-Release/ImportNumPyArray', '/tmp/SlicerSandbox/ImportNumPyArray'], 'qSlicerImportNumPyArrayModuleGenericTest')"
11: Working Directory: /tmp/SlicerSandbox-Release/ImportNumPyArray
11: Test timeout computed to be: 1500
11: When loading module  "CropVolume" , the dependency "ResampleScalarVectorDWIVolume" failed to be loaded.
11: qSlicerMarkupsModulePrivate::addToolBar: no main window is available, toolbar is not added
11: qSlicerSequencesModulePrivate::addToolBar: no main window is available, toolbar is not added
11: -------------------------------------------
11: path: ['/tmp/SlicerSandbox-Release/ImportNumPyArray', '/tmp/SlicerSandbox/ImportNumPyArray']
11: testname: qSlicerImportNumPyArrayModuleGenericTest
11: -------------------------------------------
11: test_acknowledgementText (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_application (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_categories (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_contributors (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_file_attribute (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_logic (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_module_valid (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_name_attribute (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_setCurrentModule (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_setMRMLScene (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_settings (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: test_widgetRepresentation (qSlicerImportNumPyArrayModuleGenericTest.qSlicerImportNumPyArrayModuleGenericTest) ... ok
11: 
11: ----------------------------------------------------------------------
11: Ran 12 tests in 0.016s
11: 
11: OK
1/2 Test #11: py_nomainwindow_qSlicerImportNumPyArrayModuleGenericTest ...   Passed    4.01 sec
test 12
    Start 12: py_ImportNumPyArray

12: Test command: /home/jcfr/Projects/Slicer-Release/Slicer-build/Slicer "--no-splash" "--testing" "--launcher-additional-settings" "/tmp/SlicerSandbox-Release/AdditionalLauncherSettings.ini" "--additional-module-paths" "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-scripted-modules" "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/cli-modules" "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-loadable-modules" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['/tmp/SlicerSandbox-Release/ImportNumPyArray', '/tmp/SlicerSandbox/ImportNumPyArray'], 'ImportNumPyArray')"
12: Working Directory: /tmp/SlicerSandbox-Release/ImportNumPyArray
12: Test timeout computed to be: 1500
12: Switch to module:  "Welcome"
12: -------------------------------------------
12: path: ['/tmp/SlicerSandbox-Release/ImportNumPyArray', '/tmp/SlicerSandbox/ImportNumPyArray']
12: testname: ImportNumPyArray
12: -------------------------------------------
12: testWriterReader0D (ImportNumPyArray.ImportNumPyArrayTest) ... Traceback (most recent call last):
12:   File "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-scripted-modules/ImportNumPyArray.py", line 103, in load
12:     raise RuntimeError("Zero dimensional arrays are not supported")
12: RuntimeError: Zero dimensional arrays are not supported
12: ok
12: "Volume" Reader has successfully read the file "/home/jcfr/.cache/slicer.org/Slicer-tmp/Slicer-tmpIO/MR-head.nrrd" "[0.12s]"
12: "NumPy Array Image" Reader has successfully read the file "/tmp/Slicer-tmp-jcfr/__SlicerTemp__2023-08-01_02+46+30.745/TestImportNumPyArray_3d.npy" "[0.04s]"
12: "NumPy Array Image" Reader has successfully read the file "/tmp/Slicer-tmp-jcfr/__SlicerTemp__2023-08-01_02+46+30.745/TestImportNumPyArray_3d.npz" "[0.04s]"
12: "NumPy Array Image" Reader has successfully read the file "/tmp/Slicer-tmp-jcfr/__SlicerTemp__2023-08-01_02+46+30.745/TestImportNumPyArray_2d.npy" "[0.02s]"
12: "NumPy Array Image" Reader has successfully read the file "/tmp/Slicer-tmp-jcfr/__SlicerTemp__2023-08-01_02+46+30.745/TestImportNumPyArray_1d.npy" "[0.02s]"
12: testWriterReader1D2D3D (ImportNumPyArray.ImportNumPyArrayTest) ... ok
12: "Volume" Reader has successfully read the file "/home/jcfr/.cache/slicer.org/Slicer-tmp/Slicer-tmpIO/MR-head.nrrd" "[0.12s]"
12: "NumPy Array Image" Reader has successfully read the file "/tmp/Slicer-tmp-jcfr/__SlicerTemp__2023-08-01_02+46+31.610/TestImportNumPyArray_4d.npy" "[0.27s]"
12: testWriterReader4D (ImportNumPyArray.ImportNumPyArrayTest) ... ok
12: "Sequence" Reader has successfully read the file "/home/jcfr/.cache/slicer.org/Slicer-tmp/Slicer-tmpIO/CT-cardio.seq.nrrd" "[0.22s]"
12: "NumPy Array Image" Reader has successfully read the file "/tmp/Slicer-tmp-jcfr/__SlicerTemp__2023-08-01_02+46+32.841/TestImportNumPyArray_5d.npy" "[0.09s]"
12: testWriterReader5D (ImportNumPyArray.ImportNumPyArrayTest) ... ok
12: testWriterReader6D (ImportNumPyArray.ImportNumPyArrayTest) ... Traceback (most recent call last):
12:   File "/tmp/SlicerSandbox-Release/lib/Slicer-5.3/qt-scripted-modules/ImportNumPyArray.py", line 101, in load
12:     raise RuntimeError("Arrays larger than 5 dimensions are not supported")
12: RuntimeError: Arrays larger than 5 dimensions are not supported
12: ok
12: runTest (slicer.ScriptedLoadableModule.ScriptedLoadableModuleTest)
12: Run a default selection of tests here. ... ok
12: 
12: ----------------------------------------------------------------------
12: Ran 6 tests in 3.173s
12: 
12: OK
12: Switch to module:  ""
12: Switch to module:  ""
2/2 Test #12: py_ImportNumPyArray ........................................   Passed    9.57 sec

The following tests passed:
	py_nomainwindow_qSlicerImportNumPyArrayModuleGenericTest
	py_ImportNumPyArray

100% tests passed, 0 tests failed out of 2

Total Test time (real) =  13.58 sec

@jcfr jcfr changed the title ENH: Add NumPyDataLoader for reading .npy and .npz files Add NumPyDataLoader for reading .npy and .npz files Aug 1, 2023
@jcfr jcfr mentioned this pull request Aug 1, 2023
Open
lassoan
lassoan approved these changes Mar 21, 2024
View reviewed changes
Copy link
Contributor

@lassoan lassoan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to current limitations (incorrect spacing by default, potentially incorrect axis order) the reader is not ready to be included in Slicer core, it makes sense to make it available in the Sandbox to let users experiment with it.

@lassoan lassoan merged commit 9c4b2b8 into PerkLab:master Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lassoan lassoan lassoan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@jcfr @lassoan @dzenanz