image reconstruction result comparision of the pretrained ConvNeXt-V1/V2 model

[songkq]

@shwoo93 @s9xie hi, did you make a comparision about the effect of image reconstruction of the pretrained V1/V2 model? How about comparing with MAE-pretrained ViT?

[s9xie]

Yes we have visualized the reconstruction results several times and found the reconstructed results very similar to those obtained by ViT MAE (or Swin SimMIM), even before adding the GRN layer (ie. using the v1 model). In other words, it seems one cannot really judge the representation quality from the pixel space reconstruction quality: even we get “perfect” reconstructions, the finetuning results can still have a large gap.

[songkq]

@s9xie Does ConvNeXt-v1/v2 model equipped without sparse conv also work well in image reconstruction?

[songkq]

Yes we have visualized the reconstruction results several times and found the reconstructed results very similar to those obtained by ViT MAE (or Swin SimMIM), even before adding the GRN layer (ie. using the v1 model). In other words, it seems one cannot really judge the representation quality from the pixel space reconstruction quality: even we get “perfect” reconstructions, the finetuning results can still have a large gap.

Since the gap between pretraining and finetuning in the self-supervised paradigm, could we introcude the MAE-based ConvNeXt Encoder in a Semi-supvised framework such as FixMatch?

[songkq]

@s9xie @shwoo93 Did you make a comparison among the supervised ConvNeXt-v2 models across different model sizes?
As model size shrinks, does the superiority of MAE-pretrained vanish compared with the supervised ones?