Error about finetune_qa_msvd task (Miss key 'desc' or 'caption' in descs_qa_trainval.json)

[BinzheLi95]

03/26/2024 19:23:18 - INFO - main - load_from_pretrained: ./output/vast/pretrain_vast/ckpt/model_step_204994.pt
03/26/2024 19:23:18 - INFO - main - Load from pretrained dir ./output/vast/pretrain_vast
03/26/2024 19:23:19 - INFO - main - Unexpected keys ['vision_encoder.text.logit_scale']
03/26/2024 19:23:19 - INFO - main - missing_keys ['vision_encoder.logit_scale']
03/26/2024 19:23:20 - INFO - main - ==================learning_rate_settings==================

03/26/2024 19:23:20 - INFO - main - basic_lr : 1e-05
03/26/2024 19:23:20 - INFO - main - clip_lr_visual : 5e-07
03/26/2024 19:23:20 - INFO - main - clip_lr_visual_len : 245
03/26/2024 19:23:20 - INFO - main - new_lr : 0
03/26/2024 19:23:20 - INFO - main - new_params_name: []
0%| | 0/5670 [00:00<?, ?it/s]Traceback (most recent call last):
File "/mnt/workspace/Project/VideoLargeModel/VAST/./run.py", line 63, in
main()
File "/mnt/workspace/Project/VideoLargeModel/VAST/./run.py", line 46, in main
train(model, optimizer, train_loader, val_loaders, args.run_cfg, start_step = start_step, verbose_time=False)
File "/mnt/workspace/Project/VideoLargeModel/VAST/utils/pipeline.py", line 35, in train
for step, (name, batch) in enumerate(train_loader):
File "/mnt/workspace/Project/VideoLargeModel/VAST/data/loader.py", line 101, in iter
self.preload(loader_it)
File "/mnt/workspace/Project/VideoLargeModel/VAST/data/loader.py", line 112, in preload
self.batch = next(it)
File "/mnt/workspace/Project/VideoLargeModel/VAST/data/loader.py", line 48, in iter
batch = next(iter_)
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 633, in next
data = self._next_data()
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/_utils.py", line 644, in reraise
raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/pai/envs/vast/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/mnt/workspace/Project/VideoLargeModel/VAST/data/IndexAnno.py", line 69, in getitem
raw_captions = anno['desc'] if 'desc' in anno else anno['caption']
KeyError: 'caption'

I am trying the VQA task on MSVD-QA dataset.
I use the "python3 -m torch.distributed.launch
--nnodes 1
--node_rank 0
--nproc_per_node 4
--master_port 9834
./run.py
--learning_rate 1e-5
--checkpointing true
--first_eval false
--config ./config/vast/finetune_cfg/VQA-msvd.json
--pretrain_dir $output_dir
--save_best true
--output_dir $output_dir/downstream/VQA-msvd " command line and meet above error.

I notice the AnnoIndexedDataset(Dataset) require 'desc' or 'caption' in anno, but the msvd/descs_cap_train.json do not have these info. I want to ask how to fix thie error. Thank you.

[BinzheLi95]

I solve the problem by modifying the line 70 in IndexAnno.py
From raw_captions = anno['desc'] if 'desc' in anno else anno['caption']
to
if 'desc' in anno or 'caption' in anno :
raw_captions = anno['desc'] if 'desc' in anno else anno['caption']
else
raw_captions = " "

[wonzin]

Hi, is your solution returns the right result? I am curious to leave the raw_captions empty is a visible solution or not

[BinzheLi95]

Hi, is your solution returns the right result? I am curious to leave the raw_captions empty is a visible solution or not

Yes, I obtain the right QA result. Setting raw_captions = " " for QA task is ok because the raw_captions is not necessary for QA task.

[wonzin]

Thank you for your reply :) It really helps me!