Get the future position of all actors to simulate perfect predictions

[manolotis]

Hi!

TLDR: With the Python API, can I somehow get the positions of all actors in the simulation for some time in the future? E.g. where every actor is expected to be 10 seconds from the current timestep.

Longer explanation:
I'm am currently investigating the impact of predictions (trajectory prediction of surrounding road users) on the overall behavior of an automated vehicle (AV). I could use actual prediction models, but I am interested in a different aspect: suppose we had perfect predictions, then what would be the point at which improving predictions does not present any additional benefits. In other words: how far ahead should we aim to predict? Is 5 seconds enough? 15 seconds? Does 30 seconds present any benefit w.r.t. 15? Once I find this out, I can also artificially induce errors in my perfect "predictions" to additionally find accuracy requirements for different prediction horizons.

I have done some preliminary work on this topic, and it is challenging for several reasons, but the main problem is that if my AV is interacting with a certain road user, let's call it X, then when the AV knows X's future, it will react to the intended motion of X. As a consequence to the new behavior of the AV, X will also react differently, therefore making the initial prediction of X no longer be realized.

In the work I have done, I implemented the following workaround:

• Using scenario_runner, I create scenarios making sure non-AV actors do not react to the AV
• I let one of these scenarios run, recording and saving the positions of all non-AV actors the simulation. I discard this run
• I run the same scenario again, feeding to the AV the previously recorded positions of surrounding actors as "predictions".

This works, but as you can imagine, if any of the actors reacts to the behavior of the AV, then the "predictions" are no longer valid, which limits quite a lot the type of scenarios I can test.

So my question is: using the Python API, can I somehow find where all actors will be some time in the future? I see with the carla.TrafficManager I can call get_all_actions(self, actor) to "Returns all known actions (i.e. road options and waypoints) that an actor controlled by the Traffic Manager will perform in its next steps.". This would be applicable to all vehicles, although restricting in that it must comply with road geometry. In a similar fashion, I could potentially implement a custom function in the carla.WalkerAIController to get the walker's path, as in #4333. The problem with this is that it would just get me a list of positions, but no information on at what time the actors will be at those waypoints. From the architecture of the traffic manager, I gather there is not a straightforward way of implementing a function like get_future_position(actor, seconds), returning the position of an actor some seconds into the future (if there is, any pointers are appreciated). So I was wondering if it would be possible to forward simulate the world for a given number of seconds, get the current positions of all actors, and then restore the previous state of the world before forward simulation. A procedure like that would follow this logic:

1. Copy the current state of the world, or create some sort of checkpoint. (Is that even possible? I see a remark in the carla.WorldSnapshot) stating that every tick, the client receives this snapshot which cannot be stored.
2. Tick the copied world to "fast forward" the desired number of seconds
3. Get the position of all actors in the fast-forwarded world.
4. Restore the simulation to the previous checkpoint before fast-forwarding the desired number of seconds.

In this way, I would still not eliminate the fact that if the AV reacts to the predicted motion of other actors, this could ultimately lead to the other actors behaving differently than initially predicted. However, it would allow me to incorporate these predictions in various interacting scenarios, and be confident that every prediction I obtain is correct given the current state of the world (including the currently planned AV trajectory).

Any ideas on how I could go about implementing such a feature? Or if it is even possible? Any help is highly appreciated.

[SExpert12]

have you done this work? I would like to do the same.

[manolotis]

Unfortunately not. I had to go with the workaround I described above

[SExpert12]

Okay. Do you know any source or github repo where I can learn to predict the trajectory of multiple vehicles in carla using reinforcement learning?

[manolotis]

I am not aware of any repository with plug-and-play prediction models for Carla. If you find one, let me know.

[SExpert12]

Sure.