Can this idea be used for 3D voxel convolutional NN?

[xuzhang5788]

It will be great If it can be applied to 3D voxel convolutional NN. Any concerns? Thanks

[lucidrains]

Sure! It's the same concept, but with an added dimension

[xuzhang5788]

@lucidrains
Thank you for your reply. Will you have any plans to implement it in your library?
By the way, I don't really understand how to deal with channels. For 2D images, they have RGB 3 channels. Thanks a lot

[lucidrains]

Not really for this library, as I'd like to keep it image specific. But I'd be happy to share the code snippet you need if you show me the shape of your input tensor. It won't amount to more than 10 lines, before it goes into a standard transformer

[xuzhang5788]

Thank you so much. My data shape is (31, 20, 20, 20). Voxel size is (20, 20, 20) with 31 channels.

[lucidrains]

@xuzhang5788 ok, it was a 3 line change https://gist.github.com/lucidrains/213d2be85d67d71147d807737460baf4

[xuzhang5788]

@lucidrains Thank you very much. Does Linformer library need some changes for 3D? If I want to have 10 patches, is it okay to change
efficient_transformer = Linformer(
dim=128,
seq_len=49+1, # 7x7 patches + 1 cls-token
depth=12,
heads=8,
k=64
)

into

efficient_transformer = Linformer(
dim=128,
seq_len=1000+1, # 7x7 patches + 1 cls-token
depth=12,
heads=8,
k=64
)

[lucidrains]

@xuzhang5788 you just have to make sure the sequence length is correct

yup, 10 patches would be 10 ** 3 + 1 (for cls token)

[lucidrains]

for linformer, k is recommended to be around 256 at that length

[xuzhang5788]

Thank you a lot