Questions about imagining the future

[jingtianweidizzf]

Dear MILE project authors,

thank you so much for publishing such an excellent work! I tried with the evaluate.py script and it seems that the model drives quite well in the Carla environment, bravo!

However, I'm more interested in the imagination of the model. Do you know would happen if the model imagines for a longer future period, e.g. 30 seconds. And will the imagined future state and the generated video still be reasonable?

Thus, I want to play with the imagining capability of the model. But I didn't found any example that can let the model imagine the future and generate the corresponding videos. From this line of the code, it seems that imagining is deactivated by default.

Also, it is a bit confusing for me that, the functions defined here and here seems to do some imagining work and they both have the comment of """ This is only used for visualisation of future prediction""". But neither of these functions has ever been called in the repository.

I wonder if you could please provide some example code snippets or instructions to show me how to let the model use its imagination. I would appreciate it for that.

Best,
Zifan

[anthonyhu]

Thank you for your kind words!

We did try to generate futures for up to 60 seconds and they were reasonable (visualised in the bird's-eye view space). If you wish to make the model predict far in the future, you will have to use the imagine() function of the model to predict future state and action, and repeatedly loop it. The resulting states state can then be decoded in the bird's-eye view space using self.bev_decoder(state)