Cannot use WASD controls in scenario_runner

[aureliuszi]

Hello again friends!

I am having trouble using WASD (manual) controls in scenario_runner. I use the following commands below:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>cd %CARLA_ROOT%

C:\...\Documents\G\carla9.12\carla\Build\UE4Carla\0.9.13-1-g8854804f4-dirty\WindowsNoEditor>CarlaUE4.exe

C:\...\Documents\G\carla9.12\carla\Build\UE4Carla\0.9.13-1-g8854804f4-dirty\WindowsNoEditor>cd %SCENARIO_RUNNER_ROOT%

C:\...\Documents\G\carla9.12\scenario_runner>python run_experiment.py --title "dreyevr_experiment" --route %SCENARIO_RUNNER_ROOT%\srunner\data\routes_training.xml %SCENARIO_RUNNER_ROOT%\srunner\data\all_towns_traffic_scenarios1_3_4.json 3
Starting scenario runner
Recording on file: C:/Users/zhuang/AppData/Local/CarlaUE4/Saved/exp_dreyevr_experiment_05_04_2022_15_23_55.rec
INFO:  Found the required file in cache!  Carla/Maps/TM/Town01.bin
Preparing scenario: RouteScenario_3
ScenarioManager: Running scenario RouteScenario_3

And when I try to manually control the vehicle, I am moving at speeds of 1-2.

A few things of note:

• No Logitech wheel, only keyboard here (we built the most basic DReyeVR)
• The vehicle just slightly moving in the direction of WASD but not really moving (as in the video). This is with me holding down the keys.
• When I run normal CarlaUE4.exe I am able to WASD the vehicle without any issues.

Thanks!

[aureliuszi]

Also, when I run this code, I typically only complete 1-10% of the scenarios. I tend to freeze at traffic lights/ intersections. Is this normal?

[GustavoSilvera]

Hi @aureliuszi thanks for the details, some more questions come to mind:

• Are you using the DReyeVR_AI.py function simultaneously?
• When you say "normal CarlaUE4.exe" do you mean compared to CarlaUE4.exe -vr in VR mode?
• So the WASD controls work fine until you approach a traffic light/intersection? (no, this is not normal)

My first thoughts are that there is some input conflict happening, such as between the AI driver and your WASD inputs, but I'm not convinced.

[aureliuszi]

Hi @GustavoSilvera !

1. No, it seems that I don't have activate DReyeVR_AI.py in order for a (faulty) autopilot to come on. It's just automatically there.
2. By normal CARLAUE4.exe, I mean running that code without scenario runner. The autopilot doesn't have a problem there.
3. Yes, only for scenario_runner. We break down right at intersections without fail every time.

[GustavoSilvera]

Okay, since this sounds like a scenario-runner only problem, have you tried rebuilding the DReyeVR PythonAPI (make clean && make PythonAPI)? This directly interfaces with the scenario_runner API.

We will try to reproduce this bug and gather more information. Also, is this on DReyeVR [main] or DReyeVR [dev] branch? Just so we can tune our testing accordingly.

[GustavoSilvera]

Also, we think there might by some backend autopilot stuff happening with scenario_runner itself (like, turning on the EgoVehicle's Autopilot without explicitly running DReyeVR_AI.py). So are you ever pressing 3 to enable the Autopilot driving (even without starting DReyeVR_AI.py?), or is this just fully manual driving?

[aureliuszi]

dev branch, will go back to the lab to test again shortly before rebuilding. Thank you so much for your help on this!

[aureliuszi]

Just tested again.

C:\...\Documents\G\carla9.12\carla\Build\UE4Carla\0.9.13-1-g8854804f4-dirty\WindowsNoEditor>CarlaUE4.exe

Carla pulls up and I am able to WASD

C:\...\Documents\G\carla9.12\scenario_runner>python run_experiment.py --title "dreyevr_experiment" --route %SCENARIO_RUNNER_ROOT%\srunner\data\routes_training.xml %SCENARIO_RUNNER_ROOT%\srunner\data\all_towns_traffic_scenarios1_3_4.json 3
Starting scenario runner
Recording on file: C:/Users/zhuang/AppData/Local/CarlaUE4/Saved/exp_dreyevr_experiment_05_04_2022_15_23_55.rec
INFO:  Found the required file in cache!  Carla/Maps/TM/Town01.bin
Preparing scenario: RouteScenario_3
ScenarioManager: Running scenario RouteScenario_3 

I am automatically advancing forward (without launching DReyeVR_AI.py or pressing any keys). To be honest, I am unsure about whether or not I am able to press 3 to enable autopilot because I start in what appears to be autopilot, and even when I try to activate manual control the response is minimal

[GustavoSilvera]

Gotcha, sounds like the autopilot is automatically being activated here (bug). When you are in this "ghost autopilot" mode can you press 1 to return EgoVehicle possession to the player?

[GustavoSilvera]

Pressing 1, 2, 3 should change who gets control of the EgoVehicle by toggling who is in possession of the EgoVehicle. Pressing 1 (should) return the control to the player, pressing 2 should allow the player to fly around as a spectator, and pressing 3 should provide control to the AI

[aureliuszi]

So scenario 2 is the only one that the car somewhat completes (about 40%) because there are no intersections or traffic lights during the first part.

In scenario 2, which I activated by running the code below, I was unable to press 1, 2, or 3. The car had no response and proceeded to drive itself.

python run_experiment.py --title "dreyevr_experiment" --route %SCENARIO_RUNNER_ROOT%\srunner\data\routes_training.xml %SCENARIO_RUNNER_ROOT%\srunner\data\all_towns_traffic_scenarios.json 2
Starting scenario runner
Recording on file: C:/Users/zhuang/AppData/Local/CarlaUE4/Saved/exp_dreyevr_experiment_05_06_2022_16_45_37.rec
INFO:  Found the required file in cache!  Carla/Maps/TM/Town01.bin
Preparing scenario: RouteScenario_2
ScenarioManager: Running scenario RouteScenario_2

[GustavoSilvera]

Gotcha, okay this is not expected behaviour and we will be investigating further.

Are you testing the dev branch or main branch currently?

[aureliuszi]

dev

I have yet to rebuild, will probably do that tomorrow.

[aureliuszi]

Hi @GustavoSilvera !

We have rebuilt everything using make clean && make PythonAPI; make launch; and make package. Here is where we stand after launching scenario_runner:

The autopilot starts first. We are able to enter manual control by pressing 1 as long as we are not at an intersection.

When we reach an intersection, we stop moving. Trying to use 1 to enter into manual mode (and start moving again) doesn't work and results in the unstable motion as shown in the video I sent you.

There seems to be a problem with the intersection traffic logic. Note in the images below that the traffic lights that are perpendicular to each other appear to be the same color at the intersection at which we are unable to move. This happens across all intersections in all scenarios.

[GustavoSilvera]

Thanks for all the detailed info @aureliuszi !

Development is a bit slow at the moment since I am traveling but I will investigate this problem more this week.

It looks like the routes have some kind of specific intersection logic to override all vehicle inputs and play some specific scenario (hence the multiple adjacent green lights).

[GustavoSilvera]

Hi @aureliuszi just wanted to let you know that I have been able to reproduce the bug and will have a patch out on dev soon.

[GustavoSilvera]

Okay, @aureliuszi this is a mini-patch for now (upstream incoming) I think this will work for you:

Go to srunner/scenariomanager/carla_data_provider.py:694 and change line:

if spawn_point:

with

if spawn_point and "dreyevr" not in blueprint.id:

[GustavoSilvera]

(Also, dev should be updated now, so you should be able to simply git pull and re-install make sr SR=../scenario_runner) for scenario runner (no CarlaUE4 changes necessary)

[aureliuszi]

@GustavoSilvera Just followed your instructions above and it works! Thank you so much for all of your help :D

[GustavoSilvera]

Hey, @aureliuszi please check out the upgrading-DReyeVR section of the documentation to pull from main to get all the latest bugfixes. This issue should be fixed with those changes.