-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] The result of SUMBT model #56
Comments
This code is in |
@function2-llx please look at this |
How do you get the above result? by running |
Yes, but the results I report are not using the pre training model provided by the project. However, using the pre training model provided by the project, I also got similar results. |
update SUMBT & test results #69 |
Hi! When I tried to evaluate the translation training SUMBT model, I found that the
eval mode
was not set, which had a certain impact on the results. According to the results of my local test, I found that there is a difference of two points on theMutliWOZ-zh human-val
dataset. I think it may be necessary to re-evaluate the SUMBT model after modifying the code. The current resultsare not real model performance.
My Local Result on MultiWOZ-zh
not set eval mode
{'Joint Acc': 0.4821722435545804, 'Turn Acc': 0.9738983360760534, 'Joint F1': 0.8826705748001639}
set eval mode
{'Joint Acc': 0.49972572682391664, 'Turn Acc': 0.9751935149631128, 'Joint F1': 0.8885012208542876}
The text was updated successfully, but these errors were encountered: