-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ground truth predictions don't yield 100% success rate #7
Comments
Thanks for trying this out @shuyanzhou. Can you email me the results JSON files? During the data generation, we replayed every trajectory with the masks and GT actions, and evaluated the success rate (here). We also did multiple replay checks per trajectory, so technically this shouldn't happen. |
@MohitShridhar Thanks for your prompt reply! If there is no problem on your side, the difference might come from my own revisions to the code/data. I will check. If I could not solve the problem, I will reopen the issue. Thanks! |
Yeah let me how it goes. We put in extra effort during the generation phase to make sure this doesn't happen. We discarded any trajectory that failed the evaluation. But something could have changed since then. |
@shuyanzhou looks like there was something faulty with the evaluation metrics. Can you please pull from |
@MohitShridhar thanks for pointing this. I did another evaluation pass, now the results are correct -- GTs yield 100% SR. |
Hi,
I recently find that feeding ground truth actions and masks does not yield a 100% success rate on valid_seen.
Over 817 valid_seen samples (first 3 removed for personal reason), the result is:
SR: 674/817 = 0.825
PC: 1946/2097 = 0.928
Any thoughts about the possible cause?
The text was updated successfully, but these errors were encountered: