New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poor performance of theorem predictor #10
Comments
Hi, Thank you for your interest in our work! This evaluation result is not normal. Would you mind sharing the script you were running and the log it printed? It could help me narrow down the reasons. Thanks! Best, |
Thank you for your reply! The script is same as yours. I only change the file name of output. #!/usr/bin/env python coding: utf-8import json import torch def evaluate(diagram_logic_file, text_logic_file, tokenizer_name, model_name, check_point, seq_num):
if name == 'main':
The log: D:\Anaconda\envs\intergps\python.exe D:/WorkSpace/InterGPS-main/theorem_predict/eval_transformer.py Process finished with exit code 0 Thanks! |
Hi, Below is my script:
And the running log is here: https://github.com/lupantech/InterGPS/blob/main/symbolic_solver/logs/final/log-1612098244-predict_low-first_1.log. The executed result is here: https://github.com/lupantech/InterGPS/blob/main/symbolic_solver/pred_results/final/logic_1612098244-predict_low-first_1.json. |
Thank you Pan! I can run your script to get the corresponding results. But I am focus on theorem predictor. Thanks! Best, |
Hi Fucheng, For the theorem predictor, you can follow the instructions at https://github.com/lupantech/InterGPS#theorem-predictor. For the second question, yes. As we discussed in the paper, one of the main functions of the theorem predictor is to improve the search efficiency and thus improve the final accuracy, which is verified in Table 7 and Figure 5. Best, |
Thanks, Pan! I follow the instruction at https://github.com/lupantech/InterGPS#theorem-predictor. Thanks! Best, |
Hi Fucheng, I see. Would you mind if I checked your issue a few days later? I am working on some emergent deadlines and I need more time to figure your problem out. For now, I think it is not a big problem to ignore the theorem predictor if you just want to reproduce our results. I appreciate your understanding! Best, |
Thanks, Pan. Sure. Best, |
Hi Fucheng, Thanks! I am happy to help with your project as well! Yours sincerely, |
Hello, Pan. Thank you for your open source.
I download checkpoint model from https://acl2021-intergps.s3.us-west-1.amazonaws.com/tp_model_best.pt
But the evaluation results are empty. How can I get it back to normal? Thanks.
The text was updated successfully, but these errors were encountered: