How to reproduce the experimental results

[ygan]

Hi, I hope you are doing well.
Your work is amazing!
I ran your code three times but only got a maximum of 70.5% ACC on dev.
Then I download your model from https://drive.google.com/u/0/uc?id=1ALf5ycxMViHrT5WGuFO3g9eT7R2S1rgy&export=download
I run it by run/run_evaluation.sh command.
But I can only get 71.5% ACC on dev. The output SQL looks different from that in the results_electra.txt file.
I was wondering how to reach the 75% ACC on dev.

[rhythmcao]

This is little weird. My collaborator can re-train and reproduce the experiment~(even a little higher), and the open-sourced Electra-large model should achieve 75% EM acc on dev set.

1. Please check the version of libraries in requirements.txt first and strictly follow the environmet setup. For example the stanza lib, in the follow-up work, we found that stanza==1.3.0 does not perform well.
2. Leave the hyper-parameters in the run directory unchanged, those are suggested arguments.

[ygan]

Thanks for your quick reply.
The problem happen because I use preprocessed data run by myself.
I download your preprocessed data, then my evaluation reached 75%.
I think it is my problem in the data preprocessing.
Thank you again!

[persistforever]

Hi, I download your preprocessed data and keep the run directory unchanged. However, I just obtain 70% in first 50 epoches.

Does your training log like this? If I want to obtain 74% (I use bert model), what should I do? Is there some wrong or just wait the training continousely and finally the accuracy will reach 74%?

Thanks for your reply!