How to create a windshield Display in the Ego Vehicle

[shh1v]

I know! I know! This isn't an issue.
But, I have fiddled with an unreal engine project for 15 hours straight, and I can't figure out anything. The simulator keeps on crashing.
I am trying to create a windshield display in the ego vehicle to display text. I tried to create a custom socket in the DReyeVR's ego vehicle blueprint and added an elongated sphere for display which would be spawned at the socket on the keyboard input toggle. But, when I run the simulator and give keyboard input, the simulator just crashes. I figure that when I edit the ego vehicle's blueprint class and add something through the event graph, it crashes.

I do not have enough experience with an unreal engine, so I would really appreciate it if you could help me out. If you have any alternate solution of how I could add a windshield display to add text, I would really appreciate it. Thank you.

[shh1v]

If you have any questions in order to answer my questions, I will respond ASAP. Thanks!

[GustavoSilvera]

Hi @shh1v this is interesting since we haven't experimented much with the blueprints as nearly all our work lies in the c++ implementation. I would be interested to see your CarlaUE4.log logfiles for the instances that crash to get a better idea of how/why the simulator is crashing.

As for your main question we already have an implementation that is similar in the form of the dash-text. This draws 2D text in 3D space as a function in the vehicle's update (showing speed, gear, turn signals, etc.) all of the logic lies in the EgoVehicle tick. Could this be something you could use for your windshield?

[shh1v]

Thank you so much for the reply @GustavoSilvera. I just need some time to understand the codebase and implement what you suggested. I will get back to you very soon. I nothing works, I will create a detailed response with the logs and everything I have done so far.

[shh1v]

Hello @GustavoSilvera, I don't think editing the blueprint is crashing the game. I can't figure out the bug in the game. I created a UTextRenderComponent subobject in the EgoVehicle.cpp just to test if displaying text like that would work for my research.

Code snippet added in EgoVehicle.cpp
`
// Creating a Heads-Up Display for reading task

HeadsUpDisplay = CreateDefaultSubobject<UTextRenderComponent>(TEXT("HeadsUpDisplay"));
HeadsUpDisplay->AttachToComponent(GetRootComponent(), FAttachmentTransformRules::KeepRelativeTransform);
HeadsUpDisplay->SetRelativeLocation(DashboardLocnInVehicle + FVector(0, 50.f, 5.f));
HeadsUpDisplay->SetRelativeRotation(FRotator(0.f, 180.f, 0.f)); // need to flip it to get the text in driver POV
HeadsUpDisplay->SetTextRenderColor(FColor::Black);
HeadsUpDisplay->SetText(FText::FromString("This is a sample text."));
HeadsUpDisplay->SetXScale(1.f);
HeadsUpDisplay->SetYScale(1.f);
HeadsUpDisplay->SetWorldSize(10); // scale the font with this
HeadsUpDisplay->SetVerticalAlignment(EVerticalTextAligment::EVRTA_TextCenter);
HeadsUpDisplay->SetHorizontalAlignment(EHorizTextAligment::EHTA_Center);

`
You can see the sample text being displayed on the right (Yeah, I know the orientation is crap):

However, whenever there is a cursor movement or a keyboard input, the game crashes, saying "Fatal Error". Note that this does not happen when input is given through the Logitech steering wheel and pedals. Can you help me out with this bug?

I have attached the CarlaUE4.log file below. Thanks!
CarlaUE4.log

[GustavoSilvera]

Hmm. Unfortunately there is not much interesting content in the logfile regarding the crash (everything looks normal ... then silence). But I am curious of a few things:

• Is there a particular reason that you are building this in the editor? (I can tell by the debug lines) I would recommend trying it in standalone package mode (make package) to see if the problem persists. Sometimes the editor is more finicky with these kinds of fatal-error messages.
• What are your computer specs? Not sure if this may be a hardware issue and something is bottlenecking DReyeVR.
• Can you use the HeadsUpDisplay text you created for your windshield display? It doesn't have to be static text like you made it, see how we did the speedometer/gear/signals for reference.

Unfortunately I'm not really able to say much more about how/why your system might be crashing... We have seen similar problems in the past that seemingly 'went away' after a reboot or clean build (make clean && make PythonAPI && make launch && make package).

Happy to discuss more!

[shh1v]

1. I was building the program in the editor just for ease I was not aware that it would cause some issues. I will try packaging and see how it goes. I will let you know if that was the reason for the crash.
2. Currently, I am working on my personal laptop: ASUS ROG Strix G17 17.3 (AMD Ryzen 7 5800H/1TB SSD/16GB RAM/RTX 3060)
3. Yeah, I looked at your implementation of speedometer/gear/signal implementation. I was just trying to figure out first why the crash was occurring. I have not implemented the Heads-Up display at the moment. But, I figured that I would also implement UBoxComponent behind the text (text overlapping the box component) to seem as if the text is being displayed on the box component(the display). I am not sure how it's going to work out. But, I will keep you posted!

[shh1v]

You were right! The "Fatal Error" only occurs when I play the game through UE4. But, the error does not occur after packaging and running the game. I was able to implement the Heads-Up display by creating a UStaticMeshComponent, and I displayed text by creating UTextRenderComponent. Here is what I was able to implement (although, this is the initial implementation):

Thank you for all the help you provided!

[StijnOosterlinck]

Hi @shh1v, I am currently working on a kind of similar thing where I want to show instructions to a participant, using the same idea of using an overlay which has some text in it.

I was wondering, for the result you posted here, did you manually align everything so that the text aligns nicely with the overlay? Because I am doing it this way but it does not seem like the most developer-friendly way to do it.

Or did you manage to implement it in a way where the text automatically aligns with the overlay?

Thx!

[shh1v]

Hey @StijnOosterlinck,
Apologies for the delayed response. I was on leave.

I somewhat recall the process, as it's been quite a while since I last delved into this project. However, you can review the source code responsible for aligning the text. While the code isn't perfect, it serves its purpose effectively.

The fundamental approach I used involved storing the desired text as a string. I then split this string into individual words and stored them in an array. During each update cycle, I determined the phrases that could fit within each of the five lines displayed. These five string phrases were then stored in a separate array. Periodically, the first string/phrase in this array is removed, and a new phrase is generated and added, producing a scrolling effect. This real-time generation of the next phrase to display ensures that the text does not overflow from the display area.

Feel free to reach out if you have further questions.

More resources:
GitHub repository
Published paper

[StijnOosterlinck]

Hi @shh1v ,
Thanks for you response!

Since I do not need dynamic (scrolling text) I could simply use an overlay with the text (hardcoded) displayed on it, so it kind of works. By looking at your repository (and paper), it seems like you conducted some behavioral experiments using DReyeVR. Since I'm also trying to set up an (easier) experiment, I wanted to ask you some advice if you don't mind.

Essentially, the first type of experiments I want to conduct are time production tasks. In such a task, the driver would just drive around for some time, and then the game is paused, and an instruction is shown which explains the task (time production task = 'predicting' a time interval). After the instruction disappears, the game continuous and the participant has to press a button when he/she thinks (e.g.) 30 seconds have passed. After the button press, the car could for example respawn, and the experiment is repeated but with different conditions (e.g. the weather changes).

I'm not quite sure how I would approach the setup of these experiments. Most of the technical aspects (such as pausing the game, showing the instruction, and measuring the time until the button press, I was able to implement in the C++ source code of DReyeVR), so a very simple implementation of the experiment would already work.

However, I want to minimize the amount of manual interruptions I would need to do when conducting an experiment for a participant, and there are some complexities which I need to deal with. The first one is that I have some python scripts which extract physiological data from a bracelet the participant is wearing, and I think it's weird to start executing these from within the DReyeVR C++ source code. Apart from this, I want to repeat the same experiment several times, but with different conditions as I mentioned (such as very rainy weather, or foggy weather, or sudden oddballs in the world). I already encounter problems here since changing the weather in the C++ code does not seem to be possible while it is very easy to do using the PythonAPI.

By looking at your repository, it seems like you also used python scripts to execute the experiments, so I guess trying to do everything from within the C++ source code is just not a good approach.

So the other approach I could take is to also use python scripts to execute the experiments, since I'm quite sure I can run my physiological data extraction scripts (also .py) from here, read config variables (such as weather conditions) from a certain file and then execute the experiment. However, I'm not sure about the logical implementation here, and since your python files are quite complex for me, I couldn't find answers either. I'll explain myself: the first thing I would do is start the simulator, but this would mean that the source code is already running, so I must be able to trigger a certain function from my python script such that a function in the C++ code is executed which starts my experiment, with the necessary conditions. So from a logical point of view, I would for example use PythonAPI in the script to set the weather, and then set a boolean in the C++ code, and then in the next tick the C++ code would read the boolean and execute the corresponding function, which starts the experiment. Then, at the end of the experiment, I would need to reset everything, change some conditions such as the weather and then execute the experiment again, so here the python script must be able to read variables which are set from within the C++ code I guess?

So, based on your experience, do you think it would be possible to set up my experiments using a python script? Or maybe you could tell me what the approach in your python scripts is?

It's probably not totally clear what I'm trying to explain so please ask if you would have questions, help will be much appreciated!

I'm quite sure my experiment is way simpler than the things you did so I'm convinced it should be possible to do it (totally) automatically in some way.

Thanks in advance!

[shh1v]

Okay so, correct me if I am wrong, but you essentially want to know how to give a signal from PythonAPI [script] to CARLA for executing a certain function. Well, one way to do this is to extend PythonAPI to include another method which CARLA recognizes and can execute certain behavior; however, this would be quite challenging. One easy way is to use ZMQ library to create a communication channel between CARLA and your python script, which is how we have implemented our current study. CARLA would continuously check if there is a message/signal received from the PythonAPI, and PythonAPI will be responsible for sending that signal. This signal could be a boolean value, which CARLA can then read and execute any behavior.

In our study, we run a Runnable Thread (RetrieveDataRunnable.h, RetrieveDataRunnable.cpp) which calls a method responsible for receiving messages from a network port. Please take a look at ExtHardware.cpp to see how the signal retrieval was implemented on the CARLA side. The function defined in this file were called in the parallel thread.

On the Python side, all you have to do in your python script is create the respective connection to the port for sending data. We defined HardwareSuite utility class in our experiment. Please see it for reference on how to implement it. If you are confused, just google how to implement ZMQ PUB-SUB protocol.

Also, I am curious what other items are there in the configuration file? If there are other items you would ideally want to set before launch, you can just create a set-reload-read mechanism, which we have done in our experiment. I can elaborate on this if you want.

I wrote this response in a hurry, please feel free to ask more questions, if you are confused!