why i can't reproduce the result of paper?

[SHUSHENGQIGUI]

hi, i am very confused, I am reproducing opt-1.3b sparsity. the fact is that i can get the same dense model preplexity at 14.62 of wikitext2, but the preplexity after sparsing 50% is 26.71, which is higher than 17.46 in paper. And I didn't modify any code in this repo. I wonder if the hyperparams in paper's experiments are different?
look forward to your reply.

[SHUSHENGQIGUI]

hi, i am very confused, I am reproducing opt-1.3b sparsity. the fact is that i can get the same dense model preplexity at 14.62 of wikitext2, but the preplexity after sparsing 50% is 26.71, which is higher than 17.46 in paper. And I didn't modify any code in this repo. I wonder if the hyperparams in paper's experiments are different? look forward to your reply.

@ilmarkov hi, could you reply to me in your spare time ? i am stuck in this problem. I don't know how i reproduce the result as showed in paper. please

[efrantar]

Hi, what command are you running? Generally, the code has been running fine for many people; however, it is possible that some recent HuggingFace update broke things.

[SHUSHENGQIGUI]

Hi, what command are you running? Generally, the code has been running fine for many people; however, it is possible that some recent HuggingFace update broke things.

Thank you for your reply. The command I run is the one in the readme. as follows:
python opt.py facebook/opt-1.3b wikitext2 --sparsity .5 --save ./out/opt-1.3b/sparse0.5/

and huggingface version i am using is 0.23.0
I can run your code succuessfully, but the results are slightly different from the results in paper:

look forward to your reply, Thanks

[efrantar]

Hi, in the paper we run with calibration data from c4 and not wikitext2; the former is not only more general but I think the latter can even sometimes lead to overfitting and correspondingly bad results (possibly this is what you are seeing?).

python opt.py facebook/opt-1.3b c4 --sparsity .5 --save ./out/opt-1.3b/sparse0.5/

[SHUSHENGQIGUI]

Hi, in the paper we run with calibration data from c4 and not wikitext2; the former is not only more general but I think the latter can even sometimes lead to overfitting and correspondingly bad results (possibly this is what you are seeing?).

python opt.py facebook/opt-1.3b c4 --sparsity .5 --save ./out/opt-1.3b/sparse0.5/

Thank you. To verify if the code is wrong, I test more opt models:

as you can see, Apart from opt-1.3b, the other model results differ only slightly from the paper, So I think the code is right and most reproduced results looks normal too .
That is weird. And I can confirm there is not any modify when i test opt-1.3b.
Besides, The table in your paper shows the result on wikitext2:

[efrantar]

We evaluate on wikitext but always use c4 as calibration data (this should be noted in the paper). Yes, this is a bit weird, however there can sometimes be outlier models which do not follow the trend.

[SHUSHENGQIGUI]

We evaluate on wikitext but always use c4 as calibration data (this should be noted in the paper). Yes, this is a bit weird, however there can sometimes be outlier models which do not follow the trend.

yes. i realize i made a mistake. i didn't use c4 as calibration data. Now I solve the problem, Thank you!!!!