Ground truth predictions don't yield 100% success rate

[shuyanzhou]

Hi,

I recently find that feeding ground truth actions and masks does not yield a 100% success rate on valid_seen.

Over 817 valid_seen samples (first 3 removed for personal reason), the result is:
SR: 674/817 = 0.825
PC: 1946/2097 = 0.928

Any thoughts about the possible cause?

[MohitShridhar]

Thanks for trying this out @shuyanzhou. Can you email me the results JSON files?

During the data generation, we replayed every trajectory with the masks and GT actions, and evaluated the success rate (here). We also did multiple replay checks per trajectory, so technically this shouldn't happen.

[shuyanzhou]

@MohitShridhar Thanks for your prompt reply! If there is no problem on your side, the difference might come from my own revisions to the code/data. I will check. If I could not solve the problem, I will reopen the issue. Thanks!

[MohitShridhar]

Yeah let me how it goes. We put in extra effort during the generation phase to make sure this doesn't happen. We discarded any trajectory that failed the evaluation. But something could have changed since then.

[MohitShridhar]

@shuyanzhou looks like there was something faulty with the evaluation metrics. Can you please pull from master and re-run your GT actions+masks test? Let me know how it goes.

[shuyanzhou]

@MohitShridhar thanks for pointing this. I did another evaluation pass, now the results are correct -- GTs yield 100% SR.