Reproduce Flickr30k Evaluation results - DataSet problem

[JoanFM]

Hello again @dandelin ,

I was trying to reproduce the steps from https://github.com/dandelin/ViLT/blob/master/EVAL.md to get the results from Flickr30k T2IR.

First I did what is suggested in https://github.com/dandelin/ViLT/blob/master/DATA.md.

So I have in a folder /content/flickr30k this structure:

    /content/flickr30k
     ├── flickr30k_images            
     │   ├── ....jpg
     |   ├── ....jpg
     ├── karpathy          
         ├── dataset_flickr30k.json              

Then I do the transformation:

from vilt.utils.write_f30k_karpathy import make_arrow
make_arrow( '/content/flickr30k',  '/content/arrow')

But when I run:

python run.py with data_root='/content/arrow' num_gpus=1 num_nodes=1 per_gpu_batchsize=4 task_finetune_irtr_f30k_randaug test_only=True load_path="/content/TFM_Sparse_Embeddings/vilt_irtr_f30k.ckpt"

I get the error:

ERROR - ViLT - Failed after 0:00:06!
Traceback (most recent calls WITHOUT Sacred internals):
  File "run.py", line 73, in main
    trainer.test(model, datamodule=dm)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 755, in test
    results = self.__test_given_model(model, test_dataloaders)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 820, in __test_given_model
    results = self.fit(model)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 473, in fit
    results = self.accelerator_backend.train()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 152, in train
    results = self.ddp_train(process_idx=self.task_idx, model=model)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 305, in ddp_train
    results = self.train_or_test()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 67, in train_or_test
    results = self.trainer.run_test()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 662, in run_test
    eval_loop_results, _ = self.run_evaluation(test_mode=True)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 566, in run_evaluation
    dataloaders, max_batches = self.evaluation_loop.get_evaluation_dataloaders(max_batches)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 56, in get_evaluation_dataloaders
    self.trainer.reset_test_dataloader(model)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/data_loading.py", line 299, in reset_test_dataloader
    self._reset_eval_dataloader(model, 'test')
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/data_loading.py", line 249, in _reset_eval_dataloader
    num_batches = len(dataloader) if has_len(dataloader) else float('inf')
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/data.py", line 33, in has_len
    raise ValueError('`Dataloader` returned 0 length.'
ValueError: `Dataloader` returned 0 length. Please make sure that your Dataloader at least returns 1 batch

[dandelin]

Hi @JoanFM

Could you confirm that files named f30k_caption_karpathy_train.arrow, f30k_caption_karpathy_val.arrow, f30k_caption_karpathy_test.arrow exist under the /content/arrow directory?

[JoanFM]

Hey @dandelin ,

sorry I forgot to update but I did manage to reproduce.

Thanks