Unusual behaviour of DReyeVR_vehicle after autpoilot is disabled

[shh1v]

Once the traffic manager's autopilot is set to True for the ego vehicle, it runs autonomously perfectly. However, when it is disabled afterward, the vehicle still has a throttle and a steer. Why is that so?
I tried to apply manual control with everything to default settings (like steer = 0, brake = 0.5, etc), however, it does not work. If you require any additional information/logs, please let me know.

[GustavoSilvera]

That should not be the case, since the autopilot control should be gracefully handed back to the user. Though honestly I don't remember testing this particular case well so maybe we missed it and there is a bug.

In either case, you should at any time be able to override the autopilot controls by pressing 1 to return to "manual driving" (which can reset the lingering throttle/brake/steering) while the server instance is running. Does this work?

[shh1v]

Yes! Pressing 1 does work. However, it's kind of weird that set_autopilot() does not work. Moreover, pressing 1 would not suit my study implementation as I want to enact a perfect manual hand-over upon Take-Over-Request (TOR).
Are you planning to fix this (potential) bug anytime soon?
Do you think the steps you mentioned in #40 with False parameter successfully give manual control? I am asking this before implementing this by myself because it is going to take significant time for me to code this (lol)

[GustavoSilvera]

Hmm. Pressing 1 should work so this does sound like a bug. I will also be running a study in the future with TOR so I'll be adding this to the featureset (and fixing this bug) sometime soon-ish.

Yes, calling SetAutopilot(false) (in c++) should work, but this is the same function that pressing one (1) is supposed to do.

What I could recommend you try (which shouldn't take too long) is so toggle SetAutopilot every ~5s in the EgoVehicle, so you can verify that the autopilot vs manual control can be toggled on and off repeatedly.

Code would look something like this:

// in EgoVehicle tick

if (timeToToggle > 0) { // using timeToToggle as a timer
  timeToToggle -= DeltaSeconds;
}
else {
  SetAutopilot(!bAutopilotEnabled);
  timeToToggle = 5.0; // restart the timer to 5s
}

[GustavoSilvera]

Also, I'm now testing again and the behaviour seems to be working for me. So just to verify, can you try this:

1. run DReyeVR_AI.py -n 0 to start the EgoVehicle in autopilot mode.
2. press 1: returns all control to user (WASD or logitech)
3. press 3: returns all control back to the autopilot
4. while autopilot running, press WASD/logitech to manually take over temporarily. As soon as you release your inputs the autopilot will take over again (you can change this logic so it doesn't that'd be effectively a TOR)
5. press 1: again to return control to the user and press the brakes to stop the car to a halt.

This seems to work for me.

[GustavoSilvera]

Also are you sending controls via keyboard (WASD) or logitech?

[shh1v]

Yes! Pressing 1 does work. However, it's kind of weird that set_autopilot() does not work. Moreover, pressing 1 would not suit my study implementation as I want to enact a perfect manual hand-over upon Take-Over-Request (TOR). Are you planning to fix this (potential) bug anytime soon? Do you think the steps you mentioned in #40 with False parameter successfully give manual control? I am asking this before implementing this by myself because it is going to take significant time for me to code this (lol)

Yes, as I mentioned earlier, pressing 1 does fully give control to the user (i.e., it can now ONLY be controlled by WASD/Logitech), but this does not serve well for my study implementation. I want to use set_autopilot() in python API to get the same behavior as pressing 1 (to give manual control)

[shh1v]

Moreover, after setting set_autopilot(False, args.tm_port), it's NOT that the vehicle continues on autopilot, it has a constant throttle and sometimes a constant steer which I defined as "unusual". I think we have had a miscommunication, did I do a good job explaining the problem?

[shh1v]

Try running this:

import time
import carla
import numpy as np
from DReyeVR_utils import find_ego_vehicle


def main():
  client = carla.Client('127.0.0.1', 2000)
  client.set_timeout(10.0)
  world = client.get_world()

  traffic_manager = client.get_trafficmanager(8000)
  traffic_manager.set_global_distance_to_leading_vehicle(2.5)
  settings = world.get_settings()
  traffic_manager.set_synchronous_mode(True)
  if not settings.synchronous_mode:   
      settings.synchronous_mode = True
      settings.fixed_delta_seconds = 1.0/80
  world.tick()
  DReyeVR_vehicle = find_ego_vehicle(world)

  # Increase the speed of the spawned and ego vehicle (in autopilot mode)
  traffic_manager.global_percentage_speed_difference(-200.0)

  DReyeVR_vehicle.set_autopilot(True, 8000)
  wait(world, 5) # wait 5 seconds
  print("Autopilot: True")

  DReyeVR_vehicle.set_autopilot(False, 8000)
  print("Autopilot: False")
  while True:
    world.tick()

def wait(world, duration):
  target_time = time.time() + duration
  while time.time() < target_time:
    world.tick()

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        pass
    finally:
        print('\ndone.')

You may be able to reproduce the problem. The autopilot isn't disabled after 5 secs.

[GustavoSilvera]

Okay gotcha, yeah this makes more sense now. So everything is actually working as intended I think.

By calling set_autopilot(False, args.tm_port) you are correctly disabling the autopilot but there are lingering commands (like if the throttle/steer was on and you turned off autopilot in the middle of some maneuver) that in my case get refreshed by updating the controls with manual input.

I agree this is unusual since you might expect instead of lingering inputs the car sets all its inputs to 0 and comes to a halt. Is this more like what you want to do?

[shh1v]

Yes! That's exactly what I want!

[shh1v]

Wait! Not exactly come to a halt. But the car continues to move forward at a decreasing speed (as it normally does) since autopilot is disabled. Moreover, it should ONLY be responsive to WASD/Logitech's steer/brake/throttle.

What I experienced is that once I disable autopilot, the vehicle continues to move with a constant velocity/steer which shouldn't happen.

[GustavoSilvera]

Right, as if the user were to let go of all the inputs (until they press WASD/Logitech again)

[GustavoSilvera]

Ok I think you can try this:

in SetAutopilot replace it with this:

void AEgoVehicle::SetAutopilot(const bool AutopilotOn)
{
    bAutopilotEnabled = AutopilotOn;
    AI_Player->SetAutopilot(bAutopilotEnabled);
    AI_Player->SetStickyControl(bAutopilotEnabled);
    if (!bAutopilotEnabled) {
        this->SetSteering(0);
        this->SetThrottle(0);
        this->SetBrake(0);
    }
}

Then rebuild and you should get the behaviour you want.

[shh1v]

Unfortunately, this still doesn't change anything. The vehicle still has a constant velocity after autopilot is disabled. Moreover, if the brakes are applied by using WASD/ Logitech and released, the vehicle accelerates to reach a certain speed (based on the global speed difference percentage). Have you tried running the script? Does it work for you?
It seems that setting controls (this->SetSteering(0), etc.) are applied however, the autopilot is still running and changing these values instantaneously afterward.

[GustavoSilvera]

Yes, you are right. You can ignore that suggestion and undo those changes.

I did a bit of digging and found the solution. Arguably this is more of a Carla bug than a DReyeVR bug but here you go:

You are gonna want to add this line here in Vehicle::SetAutopilot (in LibCarla/Source/carla/client/Vehicle.cpp)

on line ~39 (before the TM stuff)

GetEpisode().Lock()->SetVehicleAutopilot(*this, enabled); 

Then rebuild everything make PythonAPI && make launch (including the PythonAPI!)

We will include this Carla patch in the next version of DReyeVR

[shh1v]

This doesn't seem to work. Sorry! The vehicle is still moving with a constant velocity after the autopilot is disabled.

[GustavoSilvera]

Hmm it worked for me. I'll investigate more tmrw but for now verify that you built everything successfully (maybe make clean first)

[shh1v]

This is so infuriating that it's not working for me. I did do make clean && make PythonAPI && make package. I will redo it and let you know.
Just so that I am following you, this is what I changed:

void Vehicle::SetAutopilot(bool enabled, uint16_t tm_port) {
  GetEpisode().Lock()->SetVehicleAutopilot(*this, enabled);
  TM tm(GetEpisode(), tm_port);
  if (enabled) {
    tm.RegisterVehicles({shared_from_this()});
  } else {
    tm.UnregisterVehicles({shared_from_this()});
  }
}

[GustavoSilvera]

Ok just so we're on the same page, can you try this:

Go to EgoVehicle tick (in C++: EgoVehicle.cpp) and add a debug message such as:

UE4_LOG(LogTemp, Log, TEXT("Autopilot enabled: %d"), AI_Player-> IsAutopilotEnabled());

After rebuilding with these changes go to the Editor (make launch) and press play. Notice how the logs should all initially say "Autopilot enabled: 0" since you haven't started the autopilot yet.

Next, run the script you posted above and watch the car's autopilot turn on, when this happens you should see Autopilot Enabled: 1 and as soon as your script waits 5s and calls set_autopilot(false) you should then see it go back to Autopilot Enabled: 0 (and the car should go to rest)

Is this consistent with what you're seeing?

[shh1v]

I really appreciate you following up with me. So, Instead of using UE_LOG, I created a Log text file because I am not able to see the logs in the output log window (and I have no incentive to fix this atm).
I updated the void AEgoVehicle::Tick(float DeltaSeconds) method and added:

// DEBUG: Autopilot Enabled/Disabled
	const FString SignalFilePath = FPaths::ProjectContentDir() / TEXT("ConfigFiles/AutopilotLog.txt");
	FString LogString = AI_Player->IsAutopilotEnabled() ? TEXT("True\n") : TEXT("False\n");
	FFileHelper::SaveStringToFile(LogString, *SignalFilePath, FFileHelper::EEncodingOptions::AutoDetect, &IFileManager::Get(), EFileWrite::FILEWRITE_Append);

The surprising thing is that it only contains "FALSE" in the text log. I have no clue why this is happening.
AutopilotLog.txt

[GustavoSilvera]

Yeah, mine contains both 1's (true) and 0's (false) so there might be something funky going on with your addition of GetEpisode().Lock()->SetVehicleAutopilot(*this, enabled);. Just to be sure, you are modifying this in the lowest carla/Libcarla directory right? not in any "dependency" directories. I ask this because in building the PythonAPI and UE4 server the build scripts copy LibCarla to be used elsewhere, so those are only temporary directories.

So just to verify, after running make clean you added the line to SetVehicleAutopilot then ran make PythonAPI again?

[shh1v]

I first edited LibCarla/source/carla/client/Vehicle.cpp
Then I did mingw32-make clean && mingw32-make PythonAPI && mingw32-make launch && mingw32-make package
Whould this give out different result? If yes, how can I fix it now?

[GustavoSilvera]

Can you print out the whole path (relative to the base Carla/DReyeVR root) of this LibCarla file?

[shh1v]

carla\LibCarla\source\carla\client\Vehicle.cpp Is this what you asked for?

[GustavoSilvera]

Ok well everything looks okay here. Just to verify things are correctly getting built can you add a debug log in the same code for void Vehicle::SetAutopilot(bool enabled, uint16_t tm_port). Just to make sure that function is getting called.

Then make sure the log is updated when you call set_autopilot.

Sorry for all the hacky tests but I really thought this would fix it (as it did for me) so I just want to sanity check.

[shh1v]

While I do that, I just want to make sure that I don't have to execute mingw32-make LibCarla, right? I read in a previous discussion that make PythonAPI already executes make LibCarla, right?

[shh1v]

The log was not displayed. I think that The updated file is not being compiled. Do you have any suggestions on what I should do now?

[GustavoSilvera]

Sure, there may be something very wrong with the build procedure (or caching) so doing a clean build might be the way to go.

You could also try this by deleting the Build directory inside Carla and running make PythonAPI && make launch again.

[GustavoSilvera]

I'm pretty sure LibCarla is rebuilt with PythonAPI because it is a dependency as denoted in the makefile. However I'm not sure if it checks if there were any changes in LibCarla source or for just existing binaries (which would explain not rebuilding)... So you can try it as well:

make clean && make PythonAPI && make LibCarla && make launch

[shh1v]

I deleted the Build directory inside Carla. Then I executed make clean && make PythonAPI && make LibCarla && make launch.
Still, nothing has changed. I still have "FALSE"s in the file. I really don't know what to do now.

I can see the following in the terminal.

  -- Installing: E:/DReyeVR/carla/PythonAPI/carla/dependencies/include/carla/client/Vehicle.cpp
  -- Installing: E:/DReyeVR/carla/PythonAPI/carla/dependencies/include/carla/client/Vehicle.h

Doesn't this mean that there is no error in the building stage? And maybe the implementation not working?
Here is the full terminal output log: Output.txt

Just so that we are on the same page, this is the behavior I am getting. Video (You will have to skip a bit in the starting).

[GustavoSilvera]

Okay this makes sense now. Your PythonAPI build is broken and hence not actually building. In that output log you just sent there are many linker errors in the form of:

libcarla.obj : error LNK2001: unresolved external symbol _png_destroy_write_struct

So this explains why your LibCarla changes are not getting applied.

Can you investigate more why this is happening and try to fix it. I'll also do some research on my end but I'm away from my computer at the moment.

[GustavoSilvera]

Upon a quick search, this linker issue may be related to this, this, or this.

[shh1v]

Thank you for the resources. I will investigate further into this.

[shh1v]

Hey! I have tried all the things the posts you mentioned asked me to do. I am not sure why this I happening since I do not have expertise in CPP. But, It worked the first time when I installed DReyeVR, I don't know what happened between then and now.
If you could please look into it whenever you have time, I would really appreciate it! Thanks.

[GustavoSilvera]

Hm ok this will be difficult to debug since I don't have a windows 11 machine and don't have your system configuration.

But a couple things to point out:

• Are you using x64 version of python? What is the output of python3.exe in command prompt?
• Do you have python2 installed? (this might cause conflict)
• In the Build/ directory, is there a libPNG folder somewhere? can you check out the contents of that folder (eg. with tree or a similar operation)
• what version of MSVC compiler are you using (visual studio version?)

[shh1v]

Hey! Sorry for taking a long time to respond. So, I downgraded my system to Windows 10 and reinstalled carla and DReyeVR, and made the changes to Vehicle.cpp as suggested earlier, However, autopilot still seems to not turn off. Here is the whole terminal output of make clean && make PythonAPI && make LibCarla && make launch. You will find that there are some linker errors when I first run make PythonAPI as highlighted in this discussion. And, someone suggested that running make PythonAPI again will fix it. When I did that, the errors were gone however, I don't know if the errors were still fixed or not. In any case, I would love your input

[GustavoSilvera]

Well since you made a fresh install of CARLA a clear sign of the make PythonAPI working or not is whether you have the python .egg file in pythonapi/dist.

I would be very surprised if the changes to Vehicle.cpp still do not solve it and that hints to me that they were not properly built. Can you add the debug prints mentioned earlier to ensure the file was compiled correctly?

[shh1v]

The .egg file does exists.
Secondly, I just added the log code in Vehicle.cpp and Tick() method of EgoVehicle.cpp. When I did make PythonAPI, I got the following error. Do you knw what could be the reason?

-[BuildOSM2ODR]: OSM2ODR has been successfully installed in "D:\DReyeVROFR\carla\PythonAPI\carla\dependencies\"
-[BuildPythonAPI]: [Batch params]: --py3
Building Python API for Python 3.
Traceback (most recent call last):
  File "setup.py", line 168, in <module>
    ext_modules=get_libcarla_extensions(),
  File "setup.py", line 121, in get_libcarla_extensions
    libs = [x for x in os.listdir('dependencies/lib') if any(d in x for d in required_libs)]
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'dependencies/lib'

-[BuildPythonAPI]: Carla lib for python has been successfully installed in "D:\DReyeVROFR\carla\PythonAPI\carla\dist"!

log.txt

[GustavoSilvera]

First off, in your logs (around line ~8000) there are some compile errors with your logging mechanism, so you should first check that out. They look like this:

D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(40,19): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(40,19): error C2146: syntax error: missing ';' before identifier 'SignalFilePath' [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(40,19): error C2065: 'SignalFilePath': undeclared identifier [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(40,36): error C2653: 'FPaths': is not a class or namespace name [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(40,44): error C3861: 'ProjectContentDir': identifier not found [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,4): error C2653: 'FFileHelper': is not a class or namespace name [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,58): error C2065: 'SignalFilePath': undeclared identifier [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,74): error C2653: 'FFileHelper': is not a class or namespace name [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,105): error C2065: 'AutoDetect': undeclared identifier [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,118): error C2653: 'IFileManager': is not a class or namespace name [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,132): error C3861: 'Get': identifier not found [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,139): error C2653: 'EFileWrite': is not a class or namespace name [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,151): error C2065: 'FILEWRITE_Append': undeclared identifier [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]
D:\DReyeVROFR\carla\LibCarla\source\carla\client\Vehicle.cpp(41,17): error C3861: 'SaveStringToFile': identifier not found [D:\DReyeVROFR\carla\Build\libcarla-visualstudio\LibCarla\cmake\client\carla_client.vcxproj]

My guess is that since LibCarla (hence PythonAPI) is not building, the dependencies are not getting copied over to dependencies/lib, so you are seeing this FileNotFoundError later on. Fixing these build errors should help.

[shh1v]

I am not sure what I should do at this point. I was somewhat unsure if my logging method:

    const FString SignalFilePath = FPaths::ProjectContentDir() + TEXT("MethodCall.txt");
	  FFileHelper::SaveStringToFile(TEXT("Method called"), *SignalFilePath, FFileHelper::EEncodingOptions::AutoDetect, &IFileManager::Get(), EFileWrite::FILEWRITE_Append);

would work or not because I didn't know whether FFileHelper is imported or not. I am using FFileHelper to log the data because the normal UE_LOG() method doesn't work.
Let me take some time and look into this

[GustavoSilvera]

Well I can't say for sure since I haven't used this library before but you are probably missing a header of some sort because there are lots of identifier not found errors which usually happen due to that.

[shh1v]

Okay so the logging problem is fixed and egg file is also being generated but the log is not being displayed. At this point, I am not sure what to do. Edit: You may be able to see LNK errors but, that is only when I run make PythonAPI the first time. When I run it again, they are resolved.
log.txt

[GustavoSilvera]

Once again, there are linker errors but this time with xercesc which is a known and common CARLA issue. So there are several solutions available.

[shh1v]

Okay, @GustavoSilvera and I finally figured out what the problem was. I installed the whl file (This was recommended in CARLA doc) that was first generated by running make PythonAPI i.e., without making the necessary changes to the Vehicle.cpp file . When the necessary changes were made to the Vehicle.cpp file and make PythonAPI was executed, a new .egg and .whl were generated. And, since I did not import the new egg file in the PythonAPI script I used to test the autopilot, the carla module was imported from the old .whl file which was installed using pip3 earlier. So, the autopilot was not turning off.

Thank you so much @GustavoSilvera for your assistance throughout this issue.

[shh1v]

A note for the future readers:
I just noticed that the solution I posted above was not the complete solution. I had linker errors previously, and I thought that the solution mentioned in this discussion solved the problem, i.e., make clean & make PythonAPI && make PythonAPI. This does not solve the problem. Here is the solution to resolve the linker errors:

cd %CARLA_ROOT%/Util/InstallersWin
install_sqlite3.bat
install_xercesc.bat
install_proj.bat
:: Copy all the folders generated to the Build folder in carla root
xcopy %CARLA_ROOT%/Util/InstallersWin/FOLDER_NAMES %CARLA_ROOT%/Build
cd %CARLA_ROOT%
make osm2odr
make PythonAPI

After this, the autopilot should be disabled.