differentiable transforms

[mhubii]

the transforms are not differentiable, are they?

Edit: I was primarily looking at transforms.AffineGrid which generates the affine transformation from numpy arrays and does not allow the autograd mechanism of PyTorch to track the gradient.

Much of what you want to implement is already available here https://github.com/kornia/kornia, including the dice loss, and a good amount of the transforms, but differentiable as well.

Its not even the alpha release yet but reading the docs I had trouble to understand which type of parameter is being passed. Often there is just an explanation what it stands for rather than what data type it actually is. Makes it hard to use the functions

[tvercaut]

This relates to #95 as differentiable spatial transformations are required for most learning based registration approaches.

[tvercaut]

At first look, kornia seems focused/restricted to 2D images.

[wyli]

this feature is almost there, numeric logic is in place, now we need to make decision whether we want to make these trainable layers a part of network layer module or transform module

[mhubii]

At first look, kornia seems focused/restricted to 2D images.

Fair point but it works on RGB, should work similarly on 3D as well. However, it does not support more complex transforms

[mhubii]

this feature is almost there, numeric logic is in place, now we need to make decision whether we want to make these trainable layers a part of network layer module or transform module

The transform module is meant to be pre-processing step in monai, right? We got the torch.Dataset which takes a compose of these transforms. By design, the monai.transforms take a dict. PyTorch shouldnt really care whether we extend the torch.nn.Module or invoke calls on monai.transform, but building a network will probably look like a weird mixture of transforms and layers

[tvercaut]

For the record, it's also good to have a look at how PhoenixDL/rising did it with a common split into functionals and transforms:
https://github.com/PhoenixDL/rising/tree/master/rising/transforms
https://github.com/PhoenixDL/rising/tree/master/rising/transforms/functional

Note that quite a few 3D spatial transforms are already available there.

[wyli]

yes, @mibaumgartner from PhoenixDL/rising is also part of our working group

[mibaumgartner]

@justusschock
Small side note: rising's functional interface assumes batched tensors and applies them inside the Dataloader. This was necessary to ensure max performance on CPU AND GPU :) By this design decision the transforms could also be added as "normal" network layers. [I think a more detailed discussion about this will also be part of the next working group meeting]

So there might be some changes that need to be done to the transforms (this is a general consideration when subclassing nn.Module which generally also operate on batches).

Also: operating on RGB is not equivalent to volumetric data which would be 4D with channels.

[vikashg]

closed