Can not reproduce report result about MultiView BART

[Ricardokevins]

I want to reproduce the result in paper. But result still lower than paper's result.
Any suggestion or solution greatly appreciated~~~~~~~~

And i do with follow step

1. Clone this repo
2. Conda create and pip install using the command in README
3. download data from the link in README. (the data.tar.gz in Google Drive)
4. using bpe.sh and binary.sh in train_sh
5. using train_multi_view.sh in train_sh ( I have downloaded BART pretrain model and specifiyed in training script)
6. observed result remain a distance from paper result

Following is my train.log
`2021-05-30 16:31:37 | INFO | fairseq_cli.train | model bart_large, criterion LabelSmoothedCrossEntropyCriterion
2021-05-30 16:31:37 | INFO | fairseq_cli.train | num. model params: 416791552 (num. trained: 416791552)
2021-05-30 16:31:44 | INFO | fairseq_cli.train | training on 1 GPUs
2021-05-30 16:31:44 | INFO | fairseq_cli.train | max tokens per GPU = 800 and max sentences per GPU = None
2021-05-30 16:31:47 | INFO | fairseq.trainer | loaded checkpoint /home/data_ti4_c/gengx/PGN/DialogueSum/bart.large/bart.large/model.pt (epoch 41 @ 0 updates)
group1:
511
group2:
12
2021-05-30 16:31:47 | INFO | fairseq.trainer | NOTE: your device may support faster training with --fp16
here schedule!
2021-05-30 16:31:47 | INFO | fairseq.trainer | loading train data for epoch 0
2021-05-30 16:31:47 | INFO | fairseq.data.data_utils | loaded 14731 examples from: cnn_dm-bin_2/train.source-target.source
2021-05-30 16:31:47 | INFO | fairseq.data.data_utils | loaded 14731 examples from: cnn_dm-bin/train.source-target.source
2021-05-30 16:31:47 | INFO | fairseq.data.data_utils | loaded 14731 examples from: cnn_dm-bin_2/train.source-target.target
2021-05-30 16:31:47 | INFO | fairseq.tasks.translation | cnn_dm-bin_2 train source-target 14731 examples
!!! 14731 14731
2021-05-30 16:31:48 | WARNING | fairseq.data.data_utils | 5 samples have invalid sizes and will be skipped, max_positions=(800, 800), first few sample ids=[6248, 12799, 12502, 9490, 4269]
True
2021-05-30 16:43:49 | INFO | train | epoch 001 | loss 5.35 | nll_loss 3.418 | ppl 10.686 | wps 549.2 | ups 0.13 | wpb 4165.4 | bsz 158.3 | num_updates 93 | lr 1.395e-05 | gnorm 30.101 | clip 100 | oom 0 | train_wall 706 | wall 725
2021-05-30 16:44:02 | INFO | valid | epoch 001 | valid on 'valid' subset | loss 4.067 | nll_loss 2.182 | ppl 4.537 | wps 1721.8 | wpb 132.8 | bsz 5 | num_updates 93
100%|██████████| 817/817 [02:32<00:00, 5.35it/s]here bpe NONE
here!
Test on val set:
Val {'rouge-1': {'f': 0.466633180309513, 'p': 0.49140138382586446, 'r': 0.48556837413794035}, 'rouge-2': {'f': 0.2283604408486965, 'p': 0.23967396780627975, 'r': 0.2406360296133875}, 'rouge-l': {'f': 0.45239921360854707, 'p': 0.4768419669949866, 'r': 0.46298214107054253}}
2021-05-30 16:46:51 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints/checkpoint_best.pt (epoch 1 @ 93 updates, score 4.067) (writing took 13.230378480977379 seconds)

100%|██████████| 818/818 [02:34<00:00, 5.30it/s]Test on testing set:
Test {'rouge-1': {'f': 0.46401575684701973, 'p': 0.4876149230960775, 'r': 0.4856753031382108}, 'rouge-2': {'f': 0.22558266819086983, 'p': 0.23804809718663697, 'r': 0.2380510356102369}, 'rouge-l': {'f': 0.4507830089574146, 'p': 0.47369243895761404, 'r': 0.463735231898608}}

2021-05-30 17:01:12 | INFO | train | epoch 002 | loss 4.071 | nll_loss 2.233 | ppl 4.702 | wps 371.3 | ups 0.09 | wpb 4165.4 | bsz 158.3 | num_updates 186 | lr 2.79e-05 | gnorm 3.805 | clip 100 | oom 0 | train_wall 690 | wall 1768
2021-05-30 17:01:25 | INFO | valid | epoch 002 | valid on 'valid' subset | loss 3.943 | nll_loss 2.093 | ppl 4.267 | wps 1714.4 | wpb 132.8 | bsz 5 | num_updates 186 | best_loss 3.943
100%|██████████| 817/817 [02:40<00:00, 5.10it/s]here bpe NONE
here!
Test on val set:
Val {'rouge-1': {'f': 0.48628543166111304, 'p': 0.4849894455764677, 'r': 0.5314782969117535}, 'rouge-2': {'f': 0.24927341750101978, 'p': 0.24701375251072502, 'r': 0.2760416390474299}, 'rouge-l': {'f': 0.47186839505679423, 'p': 0.47350846043003486, 'r': 0.5050869486974158}}
2021-05-30 17:04:31 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints/checkpoint_best.pt (epoch 2 @ 186 updates, score 3.943) (writing took 23.186852996994276 seconds)

100%|██████████| 818/818 [02:43<00:00, 5.00it/s]Test on testing set:
Test {'rouge-1': {'f': 0.4786341588084106, 'p': 0.4786291670320602, 'r': 0.5265071766157988}, 'rouge-2': {'f': 0.2388174902966539, 'p': 0.23853517761149043, 'r': 0.2657237453293394}, 'rouge-l': {'f': 0.46179904611032535, 'p': 0.4642844043532549, 'r': 0.496457510321212}}

2021-05-30 17:18:59 | INFO | train | epoch 003 | loss 3.863 | nll_loss 2.03 | ppl 4.083 | wps 363.2 | ups 0.09 | wpb 4165.4 | bsz 158.3 | num_updates 279 | lr 2.95062e-05 | gnorm 3.972 | clip 100 | oom 0 | train_wall 684 | wall 2835
2021-05-30 17:19:12 | INFO | valid | epoch 003 | valid on 'valid' subset | loss 3.886 | nll_loss 2.05 | ppl 4.141 | wps 1659.8 | wpb 132.8 | bsz 5 | num_updates 279 | best_loss 3.886
100%|██████████| 817/817 [02:32<00:00, 5.36it/s]here bpe NONE
here!
Test on val set:
Val {'rouge-1': {'f': 0.48681990701428274, 'p': 0.5021639425012622, 'r': 0.514617894002924}, 'rouge-2': {'f': 0.25267840837027383, 'p': 0.2601730865438312, 'r': 0.2694340002654152}, 'rouge-l': {'f': 0.47263410942856693, 'p': 0.48723857579759916, 'r': 0.4928951658275421}}
2021-05-30 17:22:09 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints/checkpoint_best.pt (epoch 3 @ 279 updates, score 3.886) (writing took 21.174986565019935 seconds)

100%|██████████| 818/818 [02:33<00:00, 5.32it/s]Test on testing set:
Test {'rouge-1': {'f': 0.48538106530910036, 'p': 0.5043341635928553, 'r': 0.5140378691028713}, 'rouge-2': {'f': 0.24471431210883865, 'p': 0.2551209404134376, 'r': 0.2601614283339945}, 'rouge-l': {'f': 0.468263423938775, 'p': 0.4845770907657205, 'r': 0.4894307801054096}}

2021-05-30 17:36:32 | INFO | train | epoch 004 | loss 3.672 | nll_loss 1.826 | ppl 3.545 | wps 367.9 | ups 0.09 | wpb 4165.4 | bsz 158.3 | num_updates 372 | lr 2.8925e-05 | gnorm 2.403 | clip 100 | oom 0 | train_wall 692 | wall 3888
2021-05-30 17:36:45 | INFO | valid | epoch 004 | valid on 'valid' subset | loss 3.866 | nll_loss 2.047 | ppl 4.133 | wps 1719.5 | wpb 132.8 | bsz 5 | num_updates 372 | best_loss 3.866
100%|██████████| 817/817 [02:36<00:00, 5.23it/s]here bpe NONE
here!
Test on val set:
Val {'rouge-1': {'f': 0.4827315494753739, 'p': 0.50526373048346, 'r': 0.5041134859110057}, 'rouge-2': {'f': 0.24840728492515798, 'p': 0.2592307191193046, 'r': 0.2622010621533662}, 'rouge-l': {'f': 0.4675137179138125, 'p': 0.4863900933656205, 'r': 0.4838663279210453}}
2021-05-30 17:39:46 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints/checkpoint_best.pt (epoch 4 @ 372 updates, score 3.866) (writing took 22.199564302980434 seconds)

100%|██████████| 818/818 [02:34<00:00, 5.31it/s]Test on testing set:
Test {'rouge-1': {'f': 0.47986946672852465, 'p': 0.505675068662463, 'r': 0.5010302542324921}, 'rouge-2': {'f': 0.24667273240725318, 'p': 0.2601824293731443, 'r': 0.2595407303752472}, 'rouge-l': {'f': 0.4649056998973506, 'p': 0.4862129586100383, 'r': 0.4809156933510188}}

2021-05-30 17:54:06 | INFO | train | epoch 005 | loss 3.507 | nll_loss 1.646 | ppl 3.131 | wps 367.7 | ups 0.09 | wpb 4165.4 | bsz 158.3 | num_updates 465 | lr 2.83438e-05 | gnorm 2.009 | clip 100 | oom 0 | train_wall 687 | wall 4941
2021-05-30 17:54:18 | INFO | valid | epoch 005 | valid on 'valid' subset | loss 3.882 | nll_loss 2.059 | ppl 4.167 | wps 1719.8 | wpb 132.8 | bsz 5 | num_updates 465 | best_loss 3.866
100%|██████████| 817/817 [02:58<00:00, 4.59it/s]here bpe NONE
here!
Test on val set:
Val {'rouge-1': {'f': 0.4843439334064069, 'p': 0.4676299223115565, 'r': 0.5524586002963356}, 'rouge-2': {'f': 0.24974488509752005, 'p': 0.24019083936995303, 'r': 0.28883689957033615}, 'rouge-l': {'f': 0.46841726367851844, 'p': 0.4529545727532749, 'r': 0.5246822708277491}}
2021-05-30 17:57:33 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints/checkpoint_last.pt (epoch 5 @ 465 updates, score 3.882) (writing took 13.954715799016412 seconds)

100%|██████████| 818/818 [02:54<00:00, 4.68it/s]Test on testing set:
Test {'rouge-1': {'f': 0.4831880715673854, 'p': 0.4698315545550697, 'r': 0.5481711003208287}, 'rouge-2': {'f': 0.24967258791161379, 'p': 0.24298097108018568, 'r': 0.28566565132721744}, 'rouge-l': {'f': 0.47041083526959915, 'p': 0.45832861492381066, 'r': 0.5230912021841242}}

(base) gengx@v100-13:~/Multi-View-Seq2Seq-master/Multi-View-Seq2Seq-master/train_sh$
03752472}, 'rouge-l': {'f': 0.4649056998973506, 'p': 0.4862129586100383, 'r': 0.4809156933510188}}
`

[jiaaoc]

1. There might be sth related to the update of fairseq.
2. Can you try more epochs? From my past experiments, the best models usually appeared after 6 or 7 epochs.
3. Can you try BART baseline/single-view models as well?

[Ricardokevins]

1. There might be sth related to the update of fairseq.
2. Can you try more epochs? From my past experiments, the best models usually appeared after 6 or 7 epochs.
3. Can you try BART baseline/single-view models as well?

**Thank a lot for your in-time reply. **
For the first Point, The only modified I did to Code is here #4 ( I think if i pip install from this repo, the version of fairseq should be same as yours?)

And I try more epoch, but Eval Loss increase and increase which looks like already overfit So I stop training

And I try train_single_view by exec train_single_view.sh in train_sh. And the result seems still below the reported result in paper
following is the train log.

It seems Hit lowest val loss in Epoch4. According test ROUGE is
Test {'rouge-1': {'f': 0.4806754387783283, 'p': 0.47575689443993296, 'r': 0.533090841327234}, 'rouge-2': {'f': 0.24723642740645052, 'p': 0.24485542975184837, 'r': 0.27718088869081886}, 'rouge-l': {'f': 0.4685603118461953, 'p': 0.4639976287785791, 'r': 0.5105758908770055}}

2021-05-31 10:21:37 | INFO | fairseq_cli.train | model bart_large, criterion LabelSmoothedCrossEntropyCriterion
2021-05-31 10:21:37 | INFO | fairseq_cli.train | num. model params: 416791552 (num. trained: 416791552)
2021-05-31 10:21:41 | INFO | fairseq_cli.train | training on 1 GPUs
2021-05-31 10:21:41 | INFO | fairseq_cli.train | max tokens per GPU = 800 and max sentences per GPU = None
2021-05-31 10:21:44 | INFO | fairseq.trainer | loaded checkpoint /home/data_ti4_c/gengx/PGN/DialogueSum/bart.large/bart.large/model.pt (epoch 41 @ 0 updates)
group1:
511
group2:
12
2021-05-31 10:21:44 | INFO | fairseq.trainer | NOTE: your device may support faster training with --fp16
here schedule!
2021-05-31 10:21:44 | INFO | fairseq.trainer | loading train data for epoch 0
2021-05-31 10:21:44 | INFO | fairseq.data.data_utils | loaded 14731 examples from: cnn_dm-bin/train.source-target.source
2021-05-31 10:21:44 | INFO | fairseq.data.data_utils | loaded 14731 examples from: cnn_dm-bin/train.source-target.target
2021-05-31 10:21:44 | INFO | fairseq.tasks.translation | cnn_dm-bin train source-target 14731 examples
2021-05-31 10:21:44 | WARNING | fairseq.data.data_utils | 6 samples have invalid sizes and will be skipped, max_positions=(800, 800), first few sample ids=[6248, 12799, 12502, 9490, 4269, 8197]
False
epoch 001 | loss 5.513 | nll_loss 3.596 | ppl 12.092 | wps 916.2 | ups 0.23 | wpb 4077.1 | bsz 155 | num_updates 95 | lr 1.425e-05 | gnorm 32.314 | clip 100 | oom 0 | train_wall 414 | wall 429
epoch 001 | valid on 'valid' subset | loss 4.094 | nll_loss 2.2 | ppl 4.595 | wps 2698.9 | wpb 130.4 | bsz 5 | num_updates 95
here bpe NONE
here!
Test on val set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [02:02<00:00, 6.67it/s]
Val {'rouge-1': {'f': 0.46480339065997567, 'p': 0.47049869812919654, 'r': 0.5024883727698615}, 'rouge-2': {'f': 0.2276876538373641, 'p': 0.22955358188022437, 'r': 0.24868065849103466}, 'rouge-l': {'f': 0.4542692837269408, 'p': 0.46116078309275904, 'r': 0.48214810071155845}}
2021-05-31 10:31:16 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints_stage/checkpoint_best.pt (epoch 1 @ 95 updates, score 4.094) (writing took 12.353126897010952 seconds)
Test on testing set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 818/818 [02:05<00:00, 6.50it/s]
Test {'rouge-1': {'f': 0.46114517047708026, 'p': 0.46537271323141755, 'r': 0.5042302000143742}, 'rouge-2': {'f': 0.2224237363836951, 'p': 0.22493083103419967, 'r': 0.24544451649543284}, 'rouge-l': {'f': 0.44474361373637405, 'p': 0.448927478999051, 'r': 0.47784909879262455}}
epoch 002 | loss 4.109 | nll_loss 2.276 | ppl 4.843 | wps 555.9 | ups 0.14 | wpb 4077.1 | bsz 155 | num_updates 190 | lr 2.85e-05 | gnorm 3.147 | clip 100 | oom 0 | train_wall 411 | wall 1126
epoch 002 | valid on 'valid' subset | loss 3.962 | nll_loss 2.089 | ppl 4.256 | wps 2811.3 | wpb 130.4 | bsz 5 | num_updates 190 | best_loss 3.962
here bpe NONE
here!
Test on val set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [02:11<00:00, 6.21it/s]
Val {'rouge-1': {'f': 0.4725457817939638, 'p': 0.4475755100804326, 'r': 0.5487082350417644}, 'rouge-2': {'f': 0.24120891668239897, 'p': 0.22704731361123773, 'r': 0.2844809692235128}, 'rouge-l': {'f': 0.4663167405970829, 'p': 0.4482708376545491, 'r': 0.5233676583405283}}
2021-05-31 10:43:12 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints_stage/checkpoint_best.pt (epoch 2 @ 190 updates, score 3.962) (writing took 22.749575738969725 seconds)
Test on testing set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 818/818 [02:17<00:00, 5.93it/s]
Test {'rouge-1': {'f': 0.46377397032285694, 'p': 0.4401052932465904, 'r': 0.5403512633724206}, 'rouge-2': {'f': 0.227645627774666, 'p': 0.21571026664497425, 'r': 0.26880516383329606}, 'rouge-l': {'f': 0.4566739235923508, 'p': 0.43997051958006705, 'r': 0.5136008923034153}}
epoch 003 | loss 3.891 | nll_loss 2.058 | ppl 4.165 | wps 532.3 | ups 0.13 | wpb 4077.1 | bsz 155 | num_updates 285 | lr 2.94688e-05 | gnorm 4.792 | clip 100 | oom 0 | train_wall 411 | wall 1854
epoch 003 | valid on 'valid' subset | loss 3.907 | nll_loss 2.06 | ppl 4.17 | wps 2804.9 | wpb 130.4 | bsz 5 | num_updates 285 | best_loss 3.907
here bpe NONE
here!
Test on val set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [02:03<00:00, 6.63it/s]
Val {'rouge-1': {'f': 0.4898078401784239, 'p': 0.49122354401662094, 'r': 0.5326126375037953}, 'rouge-2': {'f': 0.2511270631508343, 'p': 0.25084270093658784, 'r': 0.2765290715978279}, 'rouge-l': {'f': 0.47323360177962054, 'p': 0.47368732278776043, 'r': 0.5075501058044481}}
2021-05-31 10:55:09 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints_stage/checkpoint_best.pt (epoch 3 @ 285 updates, score 3.907) (writing took 20.81832773098722 seconds)
Test on testing set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 818/818 [02:04<00:00, 6.57it/s]
Test {'rouge-1': {'f': 0.4805272898101375, 'p': 0.4894492247983221, 'r': 0.5191923582925742}, 'rouge-2': {'f': 0.24698109742466243, 'p': 0.25318572010412027, 'r': 0.26818822523077385}, 'rouge-l': {'f': 0.4670266945740137, 'p': 0.47376982129121814, 'r': 0.49816406780975997}}
epoch 004 | loss 3.695 | nll_loss 1.851 | ppl 3.608 | wps 550.5 | ups 0.14 | wpb 4077.1 | bsz 155 | num_updates 380 | lr 2.8875e-05 | gnorm 1.693 | clip 100 | oom 0 | train_wall 411 | wall 2557
epoch 004 | valid on 'valid' subset | loss 3.868 | nll_loss 2.032 | ppl 4.09 | wps 2825.3 | wpb 130.4 | bsz 5 | num_updates 380 | best_loss 3.868
here bpe NONE
here!
Test on val set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [02:05<00:00, 6.52it/s]
Val {'rouge-1': {'f': 0.4932441558906935, 'p': 0.4849630708516529, 'r': 0.5470369285867579}, 'rouge-2': {'f': 0.257195820352599, 'p': 0.2512144258115529, 'r': 0.2890727559145565}, 'rouge-l': {'f': 0.4809091476362043, 'p': 0.4733958728540178, 'r': 0.5243670254679919}}
2021-05-31 11:06:56 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints_stage/checkpoint_best.pt (epoch 4 @ 380 updates, score 3.868) (writing took 22.06681061600102 seconds)
Test on testing set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 818/818 [02:05<00:00, 6.51it/s]
Test {'rouge-1': {'f': 0.4806754387783283, 'p': 0.47575689443993296, 'r': 0.533090841327234}, 'rouge-2': {'f': 0.24723642740645052, 'p': 0.24485542975184837, 'r': 0.27718088869081886}, 'rouge-l': {'f': 0.4685603118461953, 'p': 0.4639976287785791, 'r': 0.5105758908770055}}
epoch 005 | loss 3.531 | nll_loss 1.672 | ppl 3.186 | wps 546.2 | ups 0.13 | wpb 4077.1 | bsz 155 | num_updates 475 | lr 2.82813e-05 | gnorm 1.635 | clip 100 | oom 0 | train_wall 411 | wall 3266
epoch 005 | valid on 'valid' subset | loss 3.886 | nll_loss 2.054 | ppl 4.153 | wps 2806.1 | wpb 130.4 | bsz 5 | num_updates 475 | best_loss 3.868
here bpe NONE
here!
Test on val set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [02:04<00:00, 6.56it/s]
Val {'rouge-1': {'f': 0.4944719510449904, 'p': 0.48697511330010795, 'r': 0.5466309284512514}, 'rouge-2': {'f': 0.2574232380298895, 'p': 0.25247220336568593, 'r': 0.2882837122736105}, 'rouge-l': {'f': 0.4798093107727924, 'p': 0.4732569365674383, 'r': 0.5224374168296736}}
2021-05-31 11:18:37 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints_stage/checkpoint_last.pt (epoch 5 @ 475 updates, score 3.886) (writing took 14.390713212022092 seconds)
Test on testing set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 818/818 [02:05<00:00, 6.51it/s]
Test {'rouge-1': {'f': 0.48509137071148967, 'p': 0.48264626996202653, 'r': 0.5366850271571914}, 'rouge-2': {'f': 0.2525567410158395, 'p': 0.2534527679918857, 'r': 0.2804807867555358}, 'rouge-l': {'f': 0.47183828333752353, 'p': 0.47041232293825985, 'r': 0.5118757042998228}}
epoch 006 | loss 3.39 | nll_loss 1.516 | ppl 2.859 | wps 552.4 | ups 0.14 | wpb 4077.1 | bsz 155 | num_updates 570 | lr 2.76875e-05 | gnorm 1.943 | clip 100 | oom 0 | train_wall 412 | wall 3967
epoch 006 | valid on 'valid' subset | loss 3.909 | nll_loss 2.08 | ppl 4.23 | wps 2779.9 | wpb 130.4 | bsz 5 | num_updates 570 | best_loss 3.868
here bpe NONE
here!
Test on val set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [01:55<00:00, 7.07it/s]
Val {'rouge-1': {'f': 0.4910794409859213, 'p': 0.4974831279301767, 'r': 0.5273234137514343}, 'rouge-2': {'f': 0.2531795545509134, 'p': 0.2557374222451652, 'r': 0.27467339364892834}, 'rouge-l': {'f': 0.4749285189223979, 'p': 0.4792720164512937, 'r': 0.5046984304455332}}
2021-05-31 11:30:08 | INFO | fairseq.checkpoint_utils | saved checkpoint checkpoints_stage/checkpoint_last.pt (epoch 6 @ 570 updates, score 3.909) (writing took 12.884504603978712 seconds)
Test on testing set:
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 818/818 [01:56<00:00, 7.05it/s]
Test {'rouge-1': {'f': 0.4831205514829332, 'p': 0.4938850069205325, 'r': 0.5181878116116928}, 'rouge-2': {'f': 0.24977383043483103, 'p': 0.2561533148681936, 'r': 0.2691522722501041}, 'rouge-l': {'f': 0.47026294118429235, 'p': 0.4798468138122066, 'r': 0.49714213160146487}}

[jiaaoc]

1. For the fairseq, it seems that they updated the bart-large model (the vocab size (including encoder.json', /vocab.bpe' and 'dict.txt') is changed as well). There might exist some discrepancies.

2. For single-view models, you might need to change some codes in 'fairseq_cli/train.py' as well to match the test and validation files. Did you also try the BART baseline?

[Ricardokevins]

1. For the fairseq, it seems that they updated the bart-large model (the vocab size (including encoder.json', /vocab.bpe' and 'dict.txt') is changed as well). There might exist some discrepancies.
2. For single-view models, you might need to change some codes in 'fairseq_cli/train.py' as well to match the test and validation files. Did you also try the BART baseline?

Thank a lot for reply

During these days I tried to fetch the previous version of BART. But get nothing, the DownLoad Link in Fairseq Repo seems unchange since it was created. Do you keep the BART model.pt and other file(like vocab) you used in paper? Thanks a lot!

For the BART baseline, I can not found the version of Baseline you implement in this Repo. The baseline implemented myself is differ from yours ( because of different data preprocessing or some other things)

[jiaaoc]

I did not keep the version of BART I was using. I noticed this issue because when I was preparing this repo and trying to load the trained model (https://drive.google.com/file/d/1Rhzxk1B7oaKi85Gsxr_8WcqTRx23HO-y/view) I saved, I got an error saying vocab mismatch. (I guessed they updated data pre-processing configs like encoder.json', /vocab.bpe' and 'dict.txt').

[jiaaoc]

For the BART baseline, i think the easiest way is to change the input files (remove all the segmentations).