New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evaluation during training BARTforConditionalGeneration pre-training on English corpora #55

Open

Sreyan88 opened this issue Dec 26, 2022 · 1 comment

Sreyan88 commented Dec 26, 2022

Hello,

Great repo! It's of great help to me. I just had 2 questions:

How do you do evaluation for pre-training?
Does the pre-training involve both mask infill and sentence permutation? If it does both can I just do mask infill? My main motive is to fine-tune a pre-trained BART with mask infill on an English corpus.

Thank You so much!

Owner

prajdabre commented Dec 26, 2022

Hi,

Thanks for trying the repo.

I don't assume evaluation during pretraining. Pretraining for a fixed number of iterations is what I follow usually. However if you want to do some validation then you can use the flag to save intermediate checkpoints and validate them separately via the decoding script in the repo.
By default it's only mask infilling. Sentence permutation hasn't led to major benifits in my experience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment