Add support for 4D images

[fepegar]

🚀 Feature

Add support to read, write, sample and transform 4D images.

Motivation

It would be convenient to support 4D images out of the box, as tensors of shape (C, S1, S2, S3) where C is the number of channels and SX are spatial dimensions (S1 = 1 for 2D images).

Examples of what the 4th dimension might encode

Gradient directions

• Diffusion MRI

Time

• Functional MRI
• Ultrasound sequence

Modalities or MRI sequences

• T1, CT, US
• T1, T2, PD

EM frequency bands

• RGB
• Multispectral

Labels

• Discrete: one-hot encoded labels
• Continuous: fractional labelmap aka fuzzy volumes aka tissue probability maps

Pitch

Support of 4D images in:

• torchio.Image
• torchio.io
• Samplers
• Intensity transforms
• Spatial transforms
• Docs

Alternatives

Pass the channels independently in different instances of Image.

Additional context

Considerations

torchio.io

Possible image shapes when reading

How do we know if 2D or 3D when there are 3 dimensions in the read data? Lookup table of file extensions?

2D

• (height, width)
• (height, width, channels): how do we know whether the 3rd dimension is depth or channels? Easy to infer if it's 3 as probably RGB, harder otherwise (e.g. multispectral images).

3D

• (height, width, depth): how do we know whether the 3rd dimension is depth or channels?
• (height, width, depth, channels): is this typical for dMRI and fMRI? What about 4D ultrasound?

Transforms

Consider whether to apply the same or different transformations to each channel of each image, as pointed out by @romainVala in #238 (comment).

Related issues

• Consider multi-channel output in the Grid Aggregator! #100: Consider multi-channel output in the GridAggregator
• Support 4D tensors as torchio.Image #213: Support 4D tensors as torchio.Image
• transform torchio subject - mask with multiple masks and different shapes than image #215: Transform torchio.Subject - mask with multiple masks and different shapes than image
• Passing mean and std to normalization transform.  #228: Passing mean and std to normalization transform
• Inconsistent shapes for diffusion MRI #234: Inconsistent shapes for diffusion MRI
• How to handle multi-modal images? #236: How to handle multi-modal images?
• How to handle multi-class segmentation? #237: How to handle multi-class segmentation?

[romainVala]

Great

There is no obvious difficulties for read write and sample since it is just adding multi-channel ...

Here are just a few thought for what to do with transform

For spatial transform :
one would like to have the same transfo apply to each channel. Either a for loop on the channel, on directly apply on 4D, (which is strangely slower for resample as shown in #213 )

Although in the case of diffusion one may be interested, to produce different elastic transform (as there are small non linear deformation between diffusion volume) ... so may be the same option as for intensity ?

For intensity transform:
we should have the choice (with a user define parameter) between applying the same transform for each channel or a different one

As an example for Motion, with diffusion or fMRI data one expect to have a different motion for each volume, but with a multi echo sequence, we expect the same motion for each volume

It seem logical that for random_skipke we use a different one, but for random_bias the same for each channel ... but I suggest to let the user decide

this may be quite a lot of code change in each transform ...

[romainVala]

Actually the same choice of behaviour should also be possible when multiple images are load in the subject sample

For instance in randomAffine the same affine parameter is applied to all images (one could want a different one too)
For RandomNoise it is the opposite ...

It would be great and flexible if this can also be a user choice (as input param of the transform)

Because the choice of loading several 3D volume or one 4D volume will mainly be done depending of how the file are stored, but in this is 2 different internal representation of the same data ... so It make sens to have the same behavior

[fepegar]

this may be quite a lot of code change in each transform ...

Yes, this won't be easy! We need to plan this feature carefully.

[fepegar]

Would it be a good idea to add new subclasses of Image? That would help interpret the shape of the read image. Or maybe a kwarg in Image specifying the number of spatial dimensions (2 or 3).

[fepegar]

I've done most of the work. The parameters are computed per-channel, but I think it's good to merge that for now and we'll add support to have the choice later.

@GFabien after merging this, could you refactor RandomLabelsToImage? I think the code will get much more elegant.

[GFabien]

@GFabien after merging this, could you refactor RandomLabelsToImage? I think the code will get much more elegant.

I agree. If I do this I may add a kwarg to choose the channels used to create the new image because I really like the modularity brought by the fact of having the labels as different keys in the sample. For example, in some of the models I'm currently running I use a OneOf between two RandomLabelsToImage, one that includes extra brain mask and samples a gaussian in these regions and the other one that excludes them and takes the values from the original T1 image. Without such a kwarg I would need to create two different label volumes...

[fepegar]

If I do this I may add a kwarg to choose the channels used to create the new image because I really like the modularity brought by the fact of having the labels as different keys in the sample.

Sounds good!

[fepegar]

FYI, most transforms seem ok: https://drive.google.com/file/d/1Gc8kzwKQR-bYA_ifqTnA6v0N5J_hPUeO/view?usp=sharing

[romainVala]

Sorry If I misunderstand, but I only had a quick look at the code, (i am in holiday ...)
I wonder what is the choice you have made, for all transform when you have 4D image
If I correctly understand, for random_biasfield, you apply a different bias field to each 4D image ?
Is that correct ?
I thing this should be a user choice of the transform whether you apply the same bias field to all 4D images or a different one

(from a physical point of view it makes sense to apply the same bias field to all 4D images, as it is what happen during the acquisition (if the subject is no moving too much into the coil)

[fepegar]

Salut Romain,

I wonder what is the choice you have made, for all transform when you have 4D image

For now, I haven't made any choice. I just adapted Image and the transforms to take 4D images.

If I correctly understand, for random_biasfield, you apply a different bias field to each 4D image ?
Is that correct ?

I think that's not what happens in the current implementation. I wrote to visualize the 4D transforms, look: https://colab.research.google.com/drive/1Gc8kzwKQR-bYA_ifqTnA6v0N5J_hPUeO#scrollTo=Dy2v05LPVCvA&line=2&uniqifier=1

I also think it should be the user's choice, but I won't have time to work on that anytime soon. Contributions are welcome, this shouldn't be difficult to implement for most of the transforms.

[fepegar]

Enjoy your holidays!

[meghbhalerao]

Hi all,
Thank you so much for adding 4D image support! I was wondering if there is support for directly converting a 3D multi class label map (for example in BraTS you have pixel values as 0,1,2,4) into a one-hot encoded label map which is used for model training in semantic segmentation? As in if I just instantiate an object of the class Subject with torchio.LABEL as some .nii.gz file which has a BraTS mask, would it convert this mask into a one-hot mask when I read torchio.LABEL while iterating through the DataLoader? Please let me know if something is not clear.
Thank you

[fepegar]

Hi, @meghbhalerao.

I was wondering if there is support for directly converting a 3D multi class label map (for example in BraTS you have pixel values as 0,1,2,4) into a one-hot encoded label map which is used for model training in semantic segmentation? As in if I just instantiate an object of the class Subject with torchio.LABEL as some .nii.gz file which has a BraTS mask, would it convert this mask into a one-hot mask when I read torchio.LABEL while iterating through the DataLoader?

There's nothing one-hot-encoding-related in the library, but we could add it if necessary. You can use torch.nn.functional.one_hot.

Note that you can now use the LabelMap class and forget about torchio.LABEL.

[meghbhalerao]

Thanks for the clarification @fepegar!