SROIE-VQA task performance is not as expected

[8652]

Thank you for your open source code, but I encountered problems when reproducing it, and failed to achieve the results in the paper in the SROIE-VQA data set. Could you please help me adjust the code configuration?
The parameters are set to:

task=vqa_gen
arch=ofa_base
criterion=adjust_label_smoothed_cross_entropy
label_smoothing=0.1
batch_size=8
update_freq=8
resnet_drop_path_rate=0.0
encoder_drop_path_rate=0.1
decoder_drop_path_rate=0.1
dropout=0.1
attention_dropout=0.0
max_src_length=8192
max_tgt_length=4096
num_bins=1000


max_epoch 50
warmup_updates 
lr 5e-5
patch_image_size 480

The log information is intercepted as follows:

2024-01-07 15:18:57 - train.py[line:111] - INFO: task: VqaGenTask
2024-01-07 15:18:57 - train.py[line:112] - INFO: model: OFAModel
2024-01-07 15:18:57 - train.py[line:113] - INFO: criterion: AdjustLabelSmoothedCrossEntropyCriterion
2024-01-07 15:18:57 - train.py[line:117] - INFO: num. shared model params: 182,765,896 (num. trained: 137,101,384)
2024-01-07 15:18:57 - train.py[line:124] - INFO: num. expert model params: 0 (num. trained: 0)
local datafile /home/zrb/SROIE-SIMVLM/test/ofa/val.csv slice_id 0 begin to initialize row_count and line_idx-to-offset mapping
local datafile /home/zrb/SROIE-SIMVLM/test/ofa/val.csv slice_id 0 finished initializing row_count and line_idx-to-offset mapping
file /home/zrb/SROIE-SIMVLM/test/ofa/val.csv slice_id 0 row count 1320 total row count 1320

The result of best is:

2024-01-07 18:51:31 - progress_bar.py[line:282] - INFO: epoch 050 | valid on 'valid' subset | loss 3.56 | loss_v1 0 | loss_v2 0 | nll_loss 2.49 | ntokens 98.952 | nsentences 8 | sample_size 98.952 | sample_size_v1 0 | sample_size_v2 0 | total 98.9515 | n_correct 74.7879 | ppl 5.62 | accuracy 75.58 | wps 124.7 | wpb 99 | bsz 8 | ner_accuracy 0.453647 | num_updates 1846 | best_ner_accuracy 0.454361
2024-01-07 18:51:31 - train.py[line:547] - INFO: Best checkpoint metrics:0.4536465981399902
2024-01-07 18:51:31 - train.py[line:548] - INFO: Class ago validation: P: 0.44449418084153985 R: 0.46318384753781566 F1: 0.4536465981399902
2024-01-07 18:51:31 - checkpoint_utils.py[line:64] - INFO: Preparing to save checkpoint for epoch 50 @ 1846 updates
2024-01-07 18:51:31 - trainer.py[line:434] - INFO: Saving checkpoint to /home/zrb/GenKIE/save-path//genie_base_sroie_fullshot/checkpoint50.pt
2024-01-07 18:51:34 - trainer.py[line:444] - INFO: Finished saving checkpoint to /home/zrb/GenKIE/save-path//genie_base_sroie_fullshot/checkpoint50.pt
2024-01-07 18:51:35 - checkpoint_utils.py[line:136] - INFO: Saved checkpoint /home/zrb/GenKIE/save-path//genie_base_sroie_fullshot/checkpoint50.pt (epoch 50 @ 1846 updates, score 0.4536465981399902) (writing took 3.7790093910880387 seconds)
2024-01-07 18:51:35 - train.py[line:333] - INFO: end of epoch 50 (average epoch stats below)
2024-01-07 18:51:35 - progress_bar.py[line:282] - INFO: epoch 050 | loss 2.594 | loss_v1 0 | loss_v2 0 | nll_loss 1.475 | ntokens 805.108 | nsentences 63.973 | sample_size 805.108 | sample_size_v1 0 | sample_size_v2 0 | total 805.108 | n_correct 707.459 | ppl 2.78 | accuracy 87.871 | wps 91.7 | ups 0.11 | wpb 805.1 | bsz 64 | num_updates 1846 | lr 1.12613e-07 | gnorm 6.435 | clip 100 | loss_scale 16 | train_wall 139 | gb_free 34.1 | wall 12756

[kenneys-bot]

你好，请问你遇到的这个问题解决了吗？我现在在尝试跑通该项目的代码，但是发现项目文件有缺失，导致复现失败。可以帮我看看您是否拥有这个文件吗？
File "/home/workspace/GenKIE-main/fairseq/fairseq/tasks/fairseq_task.py", line 13 , in <module>
from fairseq import metrics, search, tokenizer, utils
ImportError: cannot import name 'metrics' from 'fairseq'(unkown location)
我在仔细查询项目文件位置的时候找不到名为metrics的文件或者函数，希望能得到你的解答帮助。谢谢

[8652]

你好，请问你遇到的这个问题解决了吗？我现在在尝试跑通该项目的代码，但是发现项目文件有缺失，导致复现失败。可以帮我看看您是否拥有这个文件吗？ File "/home/workspace/GenKIE-main/fairseq/fairseq/tasks/fairseq_task.py", line 13 , in <module> from fairseq import metrics, search, tokenizer, utils ImportError: cannot import name 'metrics' from 'fairseq'(unkown location) 我在仔细查询项目文件位置的时候找不到名为metrics的文件或者函数，希望能得到你的解答帮助。谢谢

你可以参考 OFA-Sys/OFA#305 来解决
我的问题已经解决了，是load ofa base的时候路径设置不正确导致的