New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conll perl script refusing to score because of "too many repeated mentions (>10) in the response" #37
Comments
UPDATE: I ran the bash command for each metric
I got 86, 79, and 76 f1 for muc, bcub, and ceafe respectively. Average = 80.3 which is ~ 81 claimed in the paper But still I am not able to get why |
Please, see this solution |
@vdobrovolskii I replaced the loop mechanism as suggested here. And I commented out the error throwing condition in the perl script as suggested here The perl script runs fine through bash:
But the python file still shows error:
Is there any other way to fix |
I believe the output is a bit different than expected for at least one of the perl scripts. Can you send me the outputs (just the last two lines) for the perl script with "muc", "ceafe" and "bcub" as metrics? |
This comment was marked as off-topic.
This comment was marked as off-topic.
I mean, can you send me the outputs of the perl script? The |
Here is the output of Output
|
Hmm. Each line that is supposed to be fed to the script is matched correctly: Then it might be the case that when calling each metric separately the output is different... I could investigate it further. Could you kindly modify the extract_f1 function as follows as run the script again? Then send me the output.
|
The issue was in the way you were converting bytes to string. As stated here; simply typecasting bytes to string using Changing the for loop Output:
|
I ran the preparation scripts successfully.
Downloaded the roberta checkpoint from dropbox link, and placed it in data folder.
Ran the command:
python calculate_conll.py roberta test 20
I noticed some errors due to subprocess because I was using python3.6 instead of python3.7.
Error was:
unexpected keyword argument 'capture_output'
Fixed the issue with this
But then I got an error:
'NoneType' object has no attribute 'group'
origin of error --> line 15I ran the perl script directly in bash:
perl reference-coreference-scorers/scorer.pl all data/conll_logs/roberta_test_e20.gold.conll data/conll_logs/roberta_test_e20.pred.conll none
MUC came out to be 86 (f1) but while calculating b3, I got this error:
Found too many repeated mentions (> 10) in the response, so refusing to score. Please fix the output
I think it is because of this error only that the line 15 above was throwing that error (because output was empty).
How to proceed forward now? How to evaluate the results?
The text was updated successfully, but these errors were encountered: