Submission failed on test standard

[devendrachaplot]

Hi,
Our submission keeps failing on test standard with "standard_error", which I am assuming is because of time limit. The same docker images takes 14mins for remote evaluation on val_mini. Assuming test standard is 33-66 times val_mini (1000-2000 episodes vs 30 episodes), our submission should take 8-16 hrs on test standard. But the submission ran for 42 hrs and failed. Is there any way to get more information such as how many episodes were completed and left, average length of episodes, the intermediate scores when the submission timed out, etc?

Any other suggestions on how to figure out the problem would be helpful.

Thanks

[abhiskk]

Hi Devendra,

Can you try resubmitting again, I would suggest submitting on minival, the logs for minival will show you how long the submission is taking. There is an overhead in the evaluation setting as we are using gRPC to communicate between your agent docker and environment docker, this requires sending frames from environment docker to agent docker which slows down the process of evaluation, although I don't expect the overhead to be this significant.

Once you have verified that the logs on minival work well you can try and submit to test-standard and you should get a record of how many episodes your agent finished successfully.

[abhiskk]

Do rebuild the your docker before resubmitting, to be safe run a docker system prune -a before rebuilding.

[abhiskk]

@devendrachaplot can you try submitting again after running the system prune and rebuilding.

[devendrachaplot]

@abhiskk I submitted again on minival and it finished. I do not see a record of how many episodes were completed. Am I still supposed to see that if the submission finished successfully?

I also submitted to test-std again which will take a while to complete. What is the time limit for test-std for ObjectNav? Just for reference, I submitted a forward only agent to the ObjectNav test-std which does not do any computation, and it took around 34 hrs to complete.