How to transfer single-modal pre-trained model to multi-modal dataset?

[HongxiY]

Hello, I'm interested in how VoCo manages different numbers of modalities in your pre-training and downstream datasets. Your paper mentions transferring a pre-trained model (trained on CT datasets) to BraTS 21 dataset, which includes four modalities (T1, T2, T1CE, FLAIR). Could you provide more details on the strategies used for this adaptation?

[Luffy03]

Hi, many thanks for your attention to our work!
For Brats with four channels of input, we do not load the parameters of the 'patch_embed', while the other parameters are still loaded. For example:

[HongxiY]

Thanks for your quick response. May I also inquire about your 3D UNet backbone? Do you skip the first layer and then load the remaining parameters accordingly?

[Luffy03]

Thanks for your quick response. May I also inquire about your 3D UNet backbone? Do you skip the first layer and then load the remaining parameters accordingly?

Yes! You are right.

[Luffy03]

Dear researchers, our work is now available at Large-Scale-Medical, if you are still interested in this topic. Thank you very much for your attention to our work, it does encourage me a lot!