Distilling VIT-L from VIT-G on imagenet

[nighting0le01]

Hi, How can modify the config files to Distill from a ViT-g model pretrained on Imagenet21k& finetuned on Imagenet 1k to a VIT-L model.
What All do I need to modify
both checkpoints are avaialble here https://github.com/huggingface/pytorch-image-models

[yoshitomo-matsubara]

Hi @arpitsahni04

Do you just want to change models in some ImageNet configs?
If so, with the current PyPI package (torchdistill==0.3.3), your model entry in a yaml config will look like this as shown in README.md

models:
  teacher_model:
    name: 'vit_giant_patch14_224'
    repo_or_dir: 'huggingface/pytorch-image-models'
    params:
      pretrained: True
      other kwargs here if necessary ...
  student_model:
    name: 'vit_large_patch14_clip_224'
    repo_or_dir: 'huggingface/pytorch-image-models'
    params:
      pretrained: False
      other kwargs here if necessary ...

For the teacher and student names, I referred to https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py

Note that if you'll need to tune hyperparameters if you reuse existing yaml config files under sample/ like this

[nighting0le01]

Thanks @yoshitomo-matsubara but would using teachers transform be good enough since i'm finetuning student on imagenet1k during distillation anyway? I know wrong transformation gives out garbage so can you suggest any way to modify the image_classification.py to apply model dependent transforms

[yoshitomo-matsubara]

If your student model is trained from scratch, it may be ok to use teacher's transform. I feel that it's not easy for the torchdistill v0.3.3 to support two separate batches for teacher and student models.

[nighting0le01]

would it be feasible to apply the student and teacher transforms after the data comes out of the dataloader for student and teacher? can i make this change easily in torchdistill?

[yoshitomo-matsubara]

If your teacher and student models have their own transforms inside, it could be. But it's still not easy, and I don't know if it's worth spending time. I'd try to find a different pair of teacher and student models using the same transform

[nighting0le01]

i see @yoshitomo-matsubara thanks. but this can be the case for transformer student teacher models having different pretraining and finetuning. in my case student is pretrained on in_21k and teacher on liaon5b and fine tuned on in_1k. I'm using in_1k to distill knowledge into student. essentially distilling while finetuning after pretraining.