Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I failed to replicate the results of the abs mission #22

Closed
shellycsy opened this issue Sep 10, 2019 · 2 comments
Closed

I failed to replicate the results of the abs mission #22

shellycsy opened this issue Sep 10, 2019 · 2 comments

Comments

@shellycsy
Copy link

Thank you for your sharing!
when i run the code
"python train.py -task abs -mode validate -test_all -bert_data_path /home/PreSumm/src/bert_data/ -ext_dropout 0.1 -model_path abs -lr 2e-3 -visible_gpus 1,2,3 -report_every 50 -save_checkpoint_steps 1000 -batch_size 500 -train_steps 50000 -accum_count 2 -log_file ../logs/ext_bert_cnndm -use_interval true -warmup_steps 10000 -max_pos 512" ,
and then i got the result:

1 ROUGE-1 Average_R: 0.33535 (95%-conf.int. 0.33294 - 0.33767)
1 ROUGE-1 Average_P: 0.43293 (95%-conf.int. 0.43023 - 0.43579)
1 ROUGE-1 Average_F: 0.36442 (95%-conf.int. 0.36217 - 0.36671)

1 ROUGE-2 Average_R: 0.14269 (95%-conf.int. 0.14078 - 0.14462)
1 ROUGE-2 Average_P: 0.18674 (95%-conf.int. 0.18429 - 0.18927)
1 ROUGE-2 Average_F: 0.15570 (95%-conf.int. 0.15358 - 0.15776)

1 ROUGE-L Average_R: 0.31230 (95%-conf.int. 0.30999 - 0.31456)
1 ROUGE-L Average_P: 0.40353 (95%-conf.int. 0.40070 - 0.40642)
1 ROUGE-L Average_F: 0.33950 (95%-conf.int. 0.33733 - 0.34176)

[2019-09-09 21:34:50,109 INFO] Rouges at step 46000

ROUGE-F(1/2/3/l): 36.44/15.57/33.95
ROUGE-R(1/2/3/l): 33.53/14.27/31.23

Why is this result so much lower than the one given in the paper ? But the results I got on the ext task were the same as in the paper.

@nlpyang
Copy link
Owner

nlpyang commented Sep 10, 2019

please read readme carefully and use the command specifically for abstractive setting...

@shellycsy
Copy link
Author

please read readme carefully and use the command specifically for abstractive setting...

I‘m sorry ,I expected the abs task to execute the same command as ext

@nlpyang nlpyang closed this as completed Sep 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants