Index Error in Evaluation on COCO test set

[Yifanfanfanfan]

Hi Ruotian,

I tried to run the evaluation on COCO test set and encountered the following problem:

python tools/eval.py --input_json cocotest.json --input_fc_dir data/cocotest_bu_fc --input_att_dir data/cocotest_bu_att --input_label_h5 none --num_images -1 --model /home/zzgyf/github_yifan/neural_talk_pytorch/models/model-best-3.pth --infos_path /home/zzgyf/github_yifan/neural_talk_pytorch/models/infos_tdl3_nsc_best.pkl --language_eval 0
Hugginface transformers not installed; please visit https://github.com/huggingface/transformers
Warning: coco-caption not available
DataLoader loading json file:  cocotest.json
vocab size is  9487
DataLoader loading h5 file:  data/cocotest_bu_fc data/cocotest_bu_att data/cocotalk_box none
read 40775 image features
assigned 0 images to split train
assigned 0 images to split val
assigned 40775 images to split test
Traceback (most recent call last):
  File "tools/eval.py", line 102, in <module>
    loader = DataLoader(opt)
  File "/home/zzgyf/github_yifan/neural_talk_pytorch/captioning/data/dataloader.py", line 322, in __init__
    self.iters[split] = iter(self.loaders[split])
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 746, in __init__
    self._try_put_index()
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 861, in _try_put_index
    index = self._next_index()
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 339, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 200, in __iter__
    for idx in self.sampler:
  File "/home/zzgyf/github_yifan/neural_talk_pytorch/captioning/data/dataloader.py", line 389, in __next__
    elem = (self._index_list[self.iter_counter], self.iter_counter+1, wrapped)
IndexError: list index out of range
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/home/zzgyf/anaconda3/envs/ntp/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 14309) is killed by signal: Terminated.

I also have the problem that the evaluation runs ok for the first time but then throws a "ZeroDivisionError: division by zero", I saw the same issue from isssues, but haven't solved it yet.

I wonder if you have any solution? Thanks for your help.

[ruotianluo]

It is discussed here. ruotianluo/self-critical.pytorch#176

I guess if you add:

if len(self._index_list) == 0:
  return None

before
https://github.com/ruotianluo/self-critical.pytorch/blob/master/captioning/data/dataloader.py#L389

may fix it.

[ruotianluo]

Does that fix the problem?

[Yifanfanfanfan]

Thanks so much for your reply. It solves my problem.