Pythia model usage with VizWiz

[CCYChongyanChen]

❓ Questions and Help

I am trying to run the pretrained Pythia model on the VizWiz dataset. The pretrained pythia model I downloaded is from #697 (comment).

Overview

I can test the pretrained vqa2 model on vqa2 dataset; can train pythia model for the vizwiz dataset, but cannot test the pretrained vqa2 pythia model on the vizwiz dataset.

command line I run

mmf_predict dataset=vizwiz model=pythia config=projects/pythia/configs/vizwiz/defaults.yaml run_type=test checkpoint.resume_file=/data/vqa/mmf/data/models/pythia_train_val.pth

The pythia_train_val.pth is provided by #697 (comment)

error message

The first error I encounter is

To address it, I referred to issue #207 and changed mmf/mmf/configs/datasets/vizwiz/defaults.yaml' s answer_processor vocab_file from "vizwiz/v2019/extras/vocabs/answers_vizwiz_7k.txt" to "vqa2/defaults/extras/vocabs/answers_vqa.txt"

Then I encounter error

To address this, I referred #891 and removed "
- type: default
params:
model_data_dir: ${model_config.pythia.model_data_dir}" in pythia config.

Then I encounter

Thanks for the help in advance!

[vedanuj]

This will be difficult to load since the pretrained model was using two types of modal features fc6 as well as resnet152 and hence the image_text_multi_modal_combine_layer dimensions are not matching. I suggest better to retrain the pythia model on Vizwiz as that will also ensure the answer space is correct.

Otherwise you can omit the layers which don't match by using checkpoint.pretrained_state_mapping as described in this wiki : https://mmf.sh/docs/tutorials/checkpointing