Weather rendering issue (rain and fog)

[MaWel8]

Hi,
We're having some issues with the weather in CARLA (using the pre-packaged CARLA version 0.9.10.1 for Ubuntu).
Specifically, it seems that rain and fog are not properly rendered.
(Tested with the CARLA server window, as well as, with the CARLA internal camera object using the official manual_control.py example client.)
We want to have some heavy rain or fog situation which is limiting the visual range of the vehicles (or that is at least noticeably observable).
Unfortunately, we are not able to achieve this.

Whether using the presets provided by the carla.WeatherParameters object nor setting the precipitation value manually resulted in visible rendered rain in the air (visual range is identical as well).
It is only possible to observe that it is actually raining by checking the visible drops in the precipitation deposits (see image below).

This may be related to #3530 (similar needs but using CARLA version 0.9.9.4 instead).

Further, we're not able to create fog that is observable. When setting up a weather condition including fog, e.g. carla.WeatherParameters(precipitation=95.0, precipitation_deposits=50, sun_altitude_angle=45, cloudiness=80, wetness=80, fog_density=80, fog_distance=1, fog_falloff=1), it seams that the sky box breaks.
The sky becomes completely black, but the lighting of the town still works.
In #3494 the same issue is reported (the issue is indicating that it is a problem with the latest CARLA version), but no solution was given so far.

Any help how to have a more realistic raining (and fog) situation is appreciated.
Thanks!

[XGodina]

Hi, @MaWel8

I recommend to use the environment.py to control the weather in manual_controls.py. It's the script we're currently working with to change the whole weather theme
When you open the manual_control.py and change the weather (environment.py) you can see that changes in the camera spectator too.

[MaWel8]

Hi, @XGodina
Thank you for your reply!

I tried changing the weather using the environment.py but unfortunately the behavior seems to be the same.
The rain is significantly visible in the puddles on the road but there is still no rain visible in the air.

When setting fog with the environment.py the sky box still turns black.

(Using the pre-packaged CARLA 0.9.10.1 release (for Ubuntu) from GitHub, running with ./CarlaUE4.sh. Tried with Python 3.6 and Python 3.7 for the manual_control.py and the environment.py script.)

[germanros1987]

@Axel1092 @marcgpuig could you please take a look?

[Axel1092]

Hi @MaWel8, sorry for the delay. I believe that you are using OpenGL for the rendering is this right? I haven't been able to reproduce the problem with vulkan (the default renderer in CARLA). Is it not an option to use Vulkan for you?

[MaWel8]

Hi @Axel1092 ,
Thank you for your reply!

I think we are using Vulkan as well for rendering.
For me neither launching the server using ./CarlaUE4.sh nor using ./CarlaUE4.sh -opengl nor using ./CarlaUE4.sh -vulkan (probably the "vulkan" flag doesn't exist, does it?) makes any difference regarding the fog and the rain.

Is there a way to check whether CARLA is running using Vulkan?

Information to my machine and my Vulkan installation:
apt list --installed *vulkan* results in:

Listing... Done
libvulkan1/bionic,now 1.1.126.0-2~gpu18.04.1 amd64 [installed]
mesa-vulkan-drivers/bionic-updates,now 20.0.8-0ubuntu1~18.04.1 amd64 [installed,automatic]
vulkan-tools/bionic,now 1.1.126.0+dfsg1-1~gpu18.04.1 amd64 [installed,automatic]
vulkan-utils/bionic,bionic,now 1.1.126.0+dfsg1-1~gpu18.04.1 all [installed]

I am running Ubuntu 18.04.5 with a GTX 1080 TI with the nvidia-driver-450 (installed with the Ubuntu Software&Updates tool).

[Axel1092]

I can confirm that this issue comes from the OpenGL renderer in the Unreal Engine (the missing water drops, the black sky with the fog...). I tested it with a setup almost identical to yours but with the nvidia-driver-440 and in vulkan the weather works fine. My guess is that somehow unreal is not detecting vulkan in your machine and goes back to opengl. Maybe try reinstalling your drivers or checking another version. The -vulkan flag does exists and does what it should so maybe try that as well.

[MaWel8]

Hi,
thank you for the help so far!

I changed graphics driver from 450 to 455 and re-installed the vulkan packages using apt.
Unfortunately neither did help.

But Unreal in general seems to find the vulkan drivers.
When I try to run the server using the Unreal Editor with the "Play" button (GitHub repo master branch, respectively the 0.9.10.1 tag in your repository) it seems to work fine. (See images below.)
But the packaged versions do not work... (I tried to package CARLA myself, but again, same issues.)
(Also, I tried running the Unreal Editor while having the vulkan packages uninstalled and it gave me correctly an error message that "vulkan is missing" on startup -> when installed Unreal is able to work with vulkan.)

Do you have any further ideas what could solve this issue?

[Axel1092]

This is weird but I still have the feeling that the issue are the drivers. But definitely the issue is that unreal is not able to find vulkan in your machine in the package. I suggest trying the 440 nvidia drivers, I tried the same package as you did with that driver version and it works fine with the same gpu and OS as yours.
Another suggestion would be to make the package yourself, just run make package and test that (it will take quite a while).

[MaWel8]

Packaging myself (using make package) didn't change anything with this behavior.

We will check with some other machines whether vulkan works on them properly.
I will report back then.

[MaWel8]

Hi,
we could reproduce the results from my reply above (working fog and rain weather conditions when launching from UnrealEditor) on another machine using the pre-packaged GitHub release.

So, the vulkan/opengl issue is definitely with my machine (could not find out the reason yet).

Just to confirm if it looks the same for you:
In our case it just looks like the rain is visible on surfaces and for the visual range fog is added.
We could not observe raindrops in mid-air as in the release video of CARLA 0.9.8:
YouTube
(second 40 onwards)

Does the simulation look for you the same as in the release video or are the drops in the air missing for you as well?

Thanks!

[XinkaiJi]

@MaWel8 I have the same problems (the missing water drops, the black sky with the fog...).I think my CARLA is running using Vulkan. ubuntu18.04,carla 0.9.10.1,nvidia 2060 gpu, Driver Version: 450.80.02,Another computer still has the same problems.(ubuntu18.04,carla 0.9.10.1,nvidia 3080 gpu, Driver Version: 455.23.04). Could not find out the reason yet.

[ARSekkat]

I have the same issue using both Vulkan and opengl. @XinkaiJi @MaWel8 did you manage to solve this problem?

[jimmyw404]

Running into this issue also.

How do I confirm I am using Vulcan?

wetness, precipitation, precipitation_deposits and cloudiness are set to 100 in the below image.

Using a package built from source using this git revision:
j@rainbow:~/carla/source/carla$ git show
commit 993f440 (HEAD -> dev, origin/dev, origin/HEAD)
Author: Marc Garcia Puig marcgpuig@gmail.com
Date: Wed Nov 4 12:57:41 2020 +0100

Running Ubuntu 18.04 with the below drive info

j@rainbow:~/carla/source/carla$ nvidia-smi 
Fri Jan  1 06:12:35 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57       Driver Version: 450.57       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2060    Off  | 00000000:01:00.0  On |                  N/A |
| 40%   41C    P0    38W / 160W |   2075MiB /  5931MiB |      6%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       985      G   /usr/lib/xorg/Xorg                588MiB |
|    0   N/A  N/A      1259      G   /usr/bin/gnome-shell              137MiB |
|    0   N/A  N/A      3779      G   ...token=7160779015186545274       48MiB |
|    0   N/A  N/A      4473      G   ...AAAAAAAA== --shared-files      265MiB |
|    0   N/A  N/A     10942      C   mainboard                        1007MiB |
+-----------------------------------------------------------------------------+

[XinkaiJi]

I have the same issue using both Vulkan and opengl. @XinkaiJi @MaWel8 did you manage to solve this problem?

Sorry, I don't know how to solve the problem. I tried Carla 0.9.11. But this version still has the problem.

[germanros1987]

@MaWel8 @XinkaiJi just want to follow up to check if this problem occurs using the binary versions of CARLA and/or building from source.

[XinkaiJi]

@MaWel8 @XinkaiJi just want to follow up to check if this problem occurs using the binary versions of CARLA and/or building from source.

I used the binary versions of CARLA and didn't build from source.

[jimmyw404]

Thanks for looking into this. I tried to reproduce this just now and have some new information. All the precipitation values in the below images are set to 100.

I tried to reproduce this issue with my compiled 0.9.10 source code and with the manual_control.py script:

But you can see there are rain drops being rendered on this screen.

I then reproduced this issue with my own custom configuration that uses a PyGame GUI built on manual_control.py.

It's a bit hard to see but what is interesting about this image is that the rain drops are visible in the image produced by the simulated camera (shown in the PyGame window on the right) but not visible in the CarlaUE4 window.

I believe this is because the rain particles that aren't near the vehicle don't render. So when I set the spectator too far away from the vehicle the spectator can't see it. When I move the spectator near the vehicle it can, as shown in the below image.

So, I think the issue I had with this can be worked around by keeping my camera close to the vehicle. I don't know if this helps anyone else. Note that there are two related issues:

• The density of rain is fairly low (maybe solved by Is there no very hard raining like CARLA 0.8 in CARLA 0.9? #3530 )?
• The rain drops themselves are dark gray, I remember the rain being more of a light gray color.

[Axel1092]

Hi all, I have tested and the rain dropplets and fog are working. The issue that can cause it to not work is the use of OpenGL so please confirm that you are running the vulkan renderer.
Second, in the spectator camera (the server window) you will not see the rain, it is only rendered appears for the RGB and related sensors.

[jimmyw404]

Hi all, I have tested and the rain dropplets and fog are working. The issue that can cause it to not work is the use of OpenGL so please confirm that you are running the vulkan renderer.
Second, in the spectator camera (the server window) you will not see the rain, it is only rendered appears for the RGB and related sensors.

1. How can I confirm that I am actually running the Vulkan renderer and not running OpenGL?
2. I see the same rain drops in the spectator camera / server window (shown in the above images), are these the correct rain drops or something else?

[Madecu]

Hi, the raindrops don't have color, is the refraction from the water drop. So it is hard to spot them depending on the camera angle and the lighting.

To emphasize the contrast of the raindrops you can add some color as we explained at #3530

Another thing; the rain particles are only spawned were we spawn an rgb sensor like the one we create with manual_control.py. It is a square rendering raindrops and it follows the rgb sensor. That is why you don't see rain particles all over the scene.

[XinkaiJi]

Hi all, I have tested and the rain dropplets and fog are working. The issue that can cause it to not work is the use of OpenGL so please confirm that you are running the vulkan renderer.
Second, in the spectator camera (the server window) you will not see the rain, it is only rendered appears for the RGB and related sensor

I have the same issue using both Vulkan and opengl. @XinkaiJi @MaWel8 did you manage to solve this problem?

I see the rain drop in the rgb camera(sensor). I can not see the rain In the spectator view(the server window). So the problem is solved.