4D and higher dimensional convolutional layers

[alexgkendall]

Are there any plans to implement 4D or higher convolutional layers?

[aselle]

Not that I am aware of. Conv2d's are clearly used a ton and Conv3ds much less so. What kinds of applications do you have in mind? Do you have any intention to work on them yourself?

[alexgkendall]

The specific research is unpublished but it is for computer vision. The models fundamentally require 4-D convolution and it would be great to get them working in TF. I understand cudnn supports this so i thought it might not be too much effort for someone who understands the inner workings of TF to implement it.

[aselle]

It probably isn't a ton of effort, but we are not working on it currently, so I'm leaving it as contributions welcome, in case anybody wants to work on it. Thanks for bringing it up. @zheng-xq, confirming that we don't have any effort.

[alexgkendall]

@aselle I'm thinking of giving this a go myself, would you please be able to detail the steps to doing this?

[aselle]

Best way is to look at another op that is similar (like conv). Try to get a prototype working quickly and submit a draft pull request. There is an API review process where we decide where this goes in the API (and/or cotnrib). @martinwicke, do we have a design document process yet?

[martinwicke]

We don't. I think since this is a straightforward extension of existing ops, I don't think there's a huge design space to consider anyway. I'd guess it would go into contrib first, although if it is basically parallel to the 3D ops, we may just add them to core directly.

[bhack]

Is it a duplicate of #1661?

[bhack]

@alexgkendall I don't know if you can divulgate this info but is this for a follow up of GC-NET?

[girving]

I think this is unlikely to happen in a way that makes anyone happy, so closing. All applications of 4D convs that I know of will almost certainly be forced to use special factorizations for efficiency.

[QuntuamLoop]

@aselle I think the high dimention in dataset is a good application. You can see about the future computer vision have the colorful 3-D sight in the time series. [batch, [x,y,z ],rgb-channel, time]

[bhack]

I think that we can investigate to expand separable approaches.
What do you think?

[funkey]

nD convolutions can be realized by summing a sequence of (n-1)D convolutions. See here for an implementation of a 4D convolution layer using TensorFlow's conv3d layer. Details can be found in the docstring.

[funkey]

By the way, this issue is a duplicate of #1661.

[OrodRazeghi]

@funkey thank you for sharing a nice clean implementation of a 4D conv layer in tensorflow. I was wondering if you could help me with the transpose version of this 4D conv layer. How would that look like in this implementation? Also, do you think your suggested 4D conv can be used for image+time data, where the image is 3D and the 4th dimension is time?

[funkey]

A transposed convolution version of conv4d would be great. Happy to review your solution.

Yes, all our use cases are 3D+t.

[OrodRazeghi]

@funkey thanks for your response. Do you have any suggestions on how the transpose would look like in your implementation? It seems that in the case of conv4d, frame_results is the output and is the result of summing the convolution of the current input frame with its previous kernel frame. The reverse would be the result of summing the 3D transpose layers? or there is more to it? any comments will be very appreciated, thanks.