-
Notifications
You must be signed in to change notification settings - Fork 364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Text-to-Code] Error during evaluation #60
Comments
Hi, It seems I couldn't reproduce your error. I guess it related to multi-GPU training. Did you use multi-GPU training with DataParallel instead of DistributedDataParallel? It's recommended to run I have committed a change for this to only calculate ppl when evaluating during training. You can evaluate BLEU scores after the training finished. |
Hi, Yes, I was training with multi-GPU. I pulled the latest version and your change fixes the issues. |
Hi @celbree, I'm reopening this because the issue is still present during evaluation (not training), even on a single GPU. If I run the evaluation script, with
The |
Would you please provide your version number of python, pytorch and transformers? |
python: 3.6.9 |
It seems in transformers 4.3.2, the format of outputs in GPT-2 model has been changed. You could try downgrade it to 3.3.0 and it should work. |
Hello, (cc @celbree)
When I try to replicate the results for
Text-Code/text-to-code
, I get an error on the evaluation step.File:
CodeXGLUE/Text-Code/text-to-code/code/run.py
The text was updated successfully, but these errors were encountered: