Deploying the model for Inference

[gitvicky]

Hi,

Would you be able to provide python script that can perform inference over one of the pre-trained models with one of the datasets that the code/model has been validated on ?

[mikemccabe210]

Hi,

Could you post the actual error/stack trace that you're getting? Those should be in the right format, but I remember we needed to make some edits to the text strings to make them work with the new instructions (the old process was much more labor intensive). It's possible something didn't make it over to the public code, but I'll need to look into where it's currently going wrong.

[gitvicky]

Hi,

So I have been able to setup the pre-training pipeline following the instructions that you had given. I was trying to set up the model under the finetune config and got this error:

(.venv) Vicky@Vigneshs-Macbook multiple_physics_pretraining % python train_basic.py --yaml_config config/mpp_avit_ti_config.yaml --config finetune

Initializing data on rank 0
Initializing model on rank 0
Model parameter count: 7285884
Starting from pretrained model at ~/ multiple_physics_pretraining /weights/MPP_AViT_Ti
Traceback (most recent call last):
  File "~/ multiple_physics_pretraining /train_basic.py", line 551, in <module>
    trainer = Trainer(params, global_rank, local_rank, device, sweep_id=args.sweep_id)
  File "~/ multiple_physics_pretraining /train_basic.py", line 80, in __init__
    self.restore_checkpoint(params.pretrained_ckpt_path)
  File "~/ multiple_physics_pretraining /train_basic.py", line 205, in restore_checkpoint
    self.model.module.unfreeze()
  File "~/ multiple_physics_pretraining /.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'AViT' object has no attribute 'module'

Could you check to see whether the shared model weights are in the right format ? Or is it that I might have missed something.

Thanks,
Vignesh

[gitvicky]

Just pre-trained a tiny model using the diffreact and swe datasets and tried to fine-tune over that model with the previously saved checkpoints and was shown this error again. So it wont be about the formatting of the model perhaps but more about something to do with the code hopefully.

[mikemccabe210]

Yes, the actual error is in the freezing/unfreezing logic which is written for DDP. If you're not using DDP:

self.model.module.____()

where ____ is freeze/unfreeze, would be:

self.model.____()

I'll merge the fix in the nearish future, but if you just want to get it working, you can make the change locally.

[gitvicky]

Thanks. That worked a charm. I have done a fix for this and you can find it here: https://github.com/gitvicky/multiple_physics_pretraining/blob/csd3/train_basic.py

[gitvicky]

Was working on setting up the model for inference initially and noticed this unusual artefacts popping up on the model outputs. I have provided the visualisation of the model output and the expected model target here:

I had only slightly modified your train_basic file to set it up for inference across trained models. The inference code can be found here: https://github.com/gitvicky/multiple_physics_pretraining/blob/b4f39c11c87df098bc3d79f767f6746a325daf83/test_basic.py#L379

[mikemccabe210]

Thanks for finding this! Since this discussion happened offline, for anyone reading this later the early patching was the result of a bug in the bias copy for finetuning. This was fixed in 462d299 so closing the issue.