Pythia model usage with VizWiz

[1144181135]

I want to convert my dataset to the 'vizwiz' format and use Pythia model. I found several difficulties.

1. I have extracted features as the intuition, and get a folder composed of "xxx.npy" and "xxx_info.npy". How to use "tools/scripts/features/lmdb_conversion.py" to convert npy feature to lmdb feature. What is the argument "--mode"? What enviroment it dpends?

2. I use the original vizwiz dataset, download every thing automatically, and run "mmf_run config=projects/pythia/configs/vizwiz/defaults.yaml run_type=train dataset=vizwiz model=pythia". Assert Error happens:
   Traceback (most recent call last):
   File "/home/wulonglyx/miniconda3/envs/mmf/bin/mmf_run", line 33, in
   sys.exit(load_entry_point('mmf', 'console_scripts', 'mmf_run')())
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf_cli/run.py", line 133, in run
   main(configuration, predict=predict)
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf_cli/run.py", line 56, in main
   trainer.train()
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/mmf_trainer.py", line 141, in train
   self.training_loop()
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 33, in training_loop
   self.run_training_epoch()
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 91, in run_training_epoch
   report = self.run_training_batch(batch, num_batches_for_this_update)
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 165, in run_training_batch
   report = self._forward(batch)
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/trainers/core/training_loop.py", line 182, in _forward
   model_output = self.model(prepared_batch)
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/models/base_model.py", line 236, in call
   model_output = super().call(sample_list, *args, **kwargs)
   File "/home/wulonglyx/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
   result = self.forward(*input, **kwargs)
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/models/pythia.py", line 294, in forward
   "image", sample_list, text_embedding_total
   File "/home/wulonglyx/competition/imagecaption/mmf/mmf/models/pythia.py", line 241, in process_feature_embedding
   "to number of features, {}.".format(len(feature_encoders), len(features))
   AssertionError: Number of feature encoders, 2 are not equal to number of features, 1.

[apsdehal]

Hi,
You can also directly use the npy features with MMF, no need to convert to LMDB. --mode=convert is what you are looking for when converting npy files to LMDB.

AssertionError: Number of feature encoders, 2 are not equal to number of features, 1.

Pythia model also requires ResNet152 grid features for which there is a script as well in tools. Alternatively, kick out the encoder from the model's config.

[apsdehal]

Closing this as resolved. If you have more questions, please open up a new issue.

[CCYChongyanChen]

@apsdehal
I encountered a similar issue.

Overview

I can test the pretrained vqa2 model on vqa2 dataset; can train pythia model for the vizwiz dataset, but cannot test the pretrained vqa2 model on the vizwiz dataset.

command line I run

mmf_predict dataset=vizwiz model=pythia config=projects/pythia/configs/vizwiz/defaults.yaml run_type=test checkpoint.resume_file=/data/vqa/mmf/data/models/pythia_train_val.pth

The pythia_train_val.pth is provided by #697 (comment)

error message

The first error I encounter is

To address it, I referred to issue #207 and changed mmf/mmf/configs/datasets/vizwiz/defaults.yaml' s answer_processor vocab_file from "vizwiz/v2019/extras/vocabs/answers_vizwiz_7k.txt" to "vqa2/defaults/extras/vocabs/answers_vqa.txt"

Then I encounter error

To address this, I referred this issue and removed "
- type: default
params:
model_data_dir: ${model_config.pythia.model_data_dir}" in pythia config.

Then I encounter

Thanks for the help in advance! You have resolved many issues posted by me hahaha, although I know many of the issues I posted are naive and silly. Thank you again. @apsdehal