Depth information from camera

[Dokans]

Is there any plan for providing depth information from the camera? Or if there is right now any possible setting in "settings.json", which enables it?

Thank you

[SijmenHuizenga]

Hi! Thanks for your interest in the project 👋

Currently depth camera's are not supported. To get depth information you can use a Lidar or two cameras in stereo configuration. This simulation is aimed to provide as close to a real life experience as possible. And as far as I know, depth cameras do not exist. So at this moment there are no plans to support depth cameras.

However, I can imagine depth camera's to be useful during testing and training. For that reason I would love to add depth cameras in the future, but I don't expect to find time for this during this summer. But if you wanna build it yourself, we always accept pr's!

[widlarizer]

Infrared-based depth cameras do exist, have been around since 2010 in commercial products (Xbox kinect), are popular among FS Driverless teams, and are integrated into ROS.
Here's a Stereoloabs ZED mounted on an FS driverless car., and here's an Intel RealSense mounted on another

[smnschfr]

Both the RealSense and the ZED rely on stereo technology. Their main advantage is that they include the necessary data processing to calculate depth information with an onboard chip. You could get yourself the same thing by adding two cameras in the simulation (even with a nicer baseline) and doing the calculations yourself / using an appropriate library.

Whether the simulation should include such an end-to-end product is something that can be discussed.

[widlarizer]

If I understand the devices correctly, depth cameras have more data to work with than the simulator currently provides, that is, an extra layer with some IR geometry projected onto it, from which they can much less ambiguously estimate depth without requiring defined edges in the scene. I think that a possible way of simulating this would be doing some sort of raycasting using LIDAR data, if LIDAR data was sufficiently accurate to replace the depth cameras. Either way, that would be requiring teams to recreate proprietary algorithms, that in competition are accepted to be ok to use. I understand that replicating real races is not the goal or even possible, but asking teams to recreate the DSP in available tools seems about as odd as requiring them to only use self-developed LIDARs at non-virtual races.
Since the data, which depth cameras use to estimate depth, aren't present, and cannot be replicated from LIDAR, since it is much coarser than available depth cameras, which can have at close range centimeters of accuracy in testing, would it be acceptable for the FS Online team, if such an implementaiton, emitting pointcloud2 or RGBD, with an accuracy comparable to commonly used commercial products, was developed and pulled into the simulator? That way, teams which count on having depth data available for their stereo cameras, could use their existing code and models. It would only provide sensor data which are available in real races

[SijmenHuizenga]

Good point! Thanks for teaching me something about depth cameras 😄 Yes, adding depth camera's seems like a good idea.

This simulator is based on AirSim which already supports depth and infrared cameras. But because we didn't think these were needed we disabled them. So we don't have to start from scratch. Most of the heavy lifting on the game engine side is already in this codebase. But because we didn't think other camera types were needed we broke support for the other camera types in an effort to improve the normal cameras. To add the other cameras back in we would need to:

• Update the ros camera wrapper to support multiple image types.
• Get the airsim plugin to work again with multiple image types. This will be a lot of debugging.
• Test it for correctness, transforms and performance
• Document it

This is quite some work. Unfortunately, I don't expect the development team (which I am part of) to be able to implement this before the start of FSOnline. However if you have the opportunity to build it, I would love to accept a pull request!

[lussekatter]

Dear teams! Please keep it in mind that in April-May we asked all teams about the desired sensor suite. We provided multiple platforms for providing answers. Based on the incoming answers we selected a sensor suite that helped us provide the most fair way the competition can go. Also based on that we made a quite strict development plan for ourselves so that the DV simulator can exist. Since we have a couple of weeks for the competition, our highest prio is to provide to most stable framework for you to enjoy the game. This means lots of work but we are happy to do this because we think it helps the event to be the most enjoyable. This sadly means that at this point feature requests like this are highly unlikely to happen.

[Plavit]

@lussekatter Hello. Thank you for your input regarding the FS Online Driverless Comp design survey.

We do keep the survey in mind, and incidentally we did ask for sensor input from cameras with depth information in the form that we submitted in your survey from May 11th 2020:

Information_collection_for_simulation_environment_development.pdf

So this is certainly not something we only thought of now.

So far we had maintained this discussion on a constructive level, no need to start being passively aggressive about it being someone's fault, so I propose we return to the task at hand without pointing fingers. Keep in mind we were not hostile towards the development team and we do appreciate the work being done. We also understand the time and resource constraints of FS Online, and we are simply also just trying to make it work.

Of course we are also trying to create solutions, but some teams that use only some sensors seem to be better prepared and have to deal with less integration problems than other teams, which is certainly a pity and I think there is no shame in pointing it out and trying to fix it.

In order to have an interesting and fair race it has to work also for the teams, not just organizers. It will not be much fun if our models cannot work/we will have to bake something quick just for the competition that will not be tested and we end up with a "check these stationary 3d models" type of competition.

I hope you will agree we are on the same boat here.

[SijmenHuizenga]

so I propose we return to the task at hand without pointing fingers

Yes, I agree. Let's talk about what you need in technical terms. This would help me better estimate how much time it would take and figure out if (and when) we can do this. So, some questions:

• What ros message type do you need? Would with receiving sensor_msgs/Image where every pixel contains a depth value suffice?
• Can you explain how accuracy works with depth camera? Like, do objects further away have worse accuracy? How does noise work? I'm asking because providing a perfect depth image might be over powered. Are there downsides to depth cameras? I have never used a depth camera, can you give me a quick introduction to their workings?
• What would be a realistic competition rules? Like limiting framerate, field of view and resolution?

[Plavit]

@SijmenHuizenga Thank You for Your understanding!

We also understand it is not easy to develop the FSO Sim and appreciate all your hard work efforts! 😉

Regarding your questions:
1) ROS messages
This setup sounds good to me, but maybe @Dokans will have better feedback

2) Sensor accuracy
A perfect depth map would definitely be overpowered, this is not what we want. Accuracy in real hardware works in complicated ways, but I think it would be simple to simulate the general gist.

In general, your assumption is correct - objects further away have worse accuracy. The accuracy itself varies from camera to camera depends mainly on the baseline of the stereocamera (how far apart are the two lenses). The larger the baseline, the better accuracy you get and the further the camera can reliably detect depth.

The camera usually has several feeds, some are visible light, some infrared, etc. Realsense for example gets the infrared datapoints, which are limited in number (upper right screen on pic below) and it seems to extrapolate it to produce the depth map (lower left picture).

Here is some practical footage we got from testing inside last summer just so you have an idea - Intel Realsense above, Stereolabs ZED below:

As you can see, this extra/interpolation may produce some static errors. Another issue are dynamic errors, which may occur due to quantization for instance - the camera may see some point to be at a static distance even though it is moving, and only if it moves enough, it may suddenly jump by a more significant distance.

Here is a compressed gif showing the Intel Realsense feed so you have an idea about the dynamic accuracy errors:

@Dokans did some testing of both cameras in outside conditions as well. Intel Realsense is usable cca. up to 14 meters and the Stereolabs ZED up to 24 meters if I remember correctly, with the margin of error being ca. 1 cm for 1 meter range and goes up to cca. 10cm beyond the 10m range. Maybe @Dokans will comment further to clarify, I did not find the precise testing report.

Overall, I think the depth information can be simulated relatively simply by some transformation, random offset static, value update delays, competition limits (below), range limits etc. It will not be perfect but it should do the job.

3) Realistic competition rules
I agree with limiting the variables you proposed:

• Field of view: Can be static, I think Realsense has 91.2°x65.5°x100.6°
  
  Source: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-D400-Series-Datasheet.pdf

• Framerate and resolution: I would propose a dynamic matrix allowing us to choose between framerate and resolution oriented approaches (more framerate means lower resolution and vice versa). This reflects real conditions of Realsense as well:
  
  Source: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-D400-Series-Datasheet.pdf

Hope this helps, let us know if you have further questions 🚀

[kaadam92]

Dear Plavit,

Thank you for the comments and ideas! I'm also very happy to have you and your team among the first testers of our new simulator platform! However, I have to say sorry for the inconvenience, due to our limited resources at the moment we cannot commit ourselves to enable this feature in the simulator. Of course, if the stability of the simulator during the event will be not compromised, we will do our best to support the depth cameras as well. Please also keep in mind, we have plans with the simulator-based driverless discipline in the future, so in the long run, your system will be covered as well!

Dávid

[SijmenHuizenga]

@Plavit thank you so much for this amazing writeup. Honestly, you should get an award for the best comment on this project.

In my off-time I've been playing around with the depth camera's from airsim. This is an image generated from such a camera:

capture information: DepthPerspective, matplotlib.imshow(vmin=1, vmax=10), "Width": 1280, "Height": 720, "FOV_Degrees": 92, "PixelsAsFloat": true

What you see here is a perfect depth image. Each pixel is a float value containing the distance in meters to the object using some raycast. It is captured using the DepthPerspective as described here. Surprisingly, this camera still works within our this project as well.

To get this competition ready, we would have to adapt the ros wrapper to support these images. This is not too difficult but, including testing, reviewing and the expected unexpected bugs I think it will still take more time than I want.

We should also add some noise, lower the resolution or lower the accuracy because this cm accuracy for each pixel is just way too powerful. I agree with your suggestion, to do some transformation, random offset and value update delays. It would be amazing if we could model some accuracy based on distance.

Some idea on how to balance accuracy: round all pixels based on the distance it represents. We find some curve / formula that does rounding. For example, any pixel within the 2-4 meter is rounded up to the next 3-cm point. So a pixel with value 224 cm would be rounded to 225. Between 4-6 meters rounded to 6-cm point. So pixel with value 511 cm would become 516 cm.

Framerate and resolution: everything in that table sounds reasonable, although I am not sure any camera will be able to do more then 20 fps (see #43).

About field of view: limiting the field of view to 91.2° horizontal sounds good. The vertical field of view is automatically calculated by AirSim using the following formula: vertical FoV = image height / image width * horizontal FoV. I think it is out of scope to change this.

Now the real problem: I am busy preparing the competition maps, implementing weather forecasts and integration testing on google cloud. The rest of the development team is either on holiday or busy working on implementing their own autonomous system (which, surprisingly, still doesn't work). As Dávid mentioned, there not really someone who can do this at the moment. Maybe (no promises) I can find some time to implement the ros wrapper, but I need your help with testing and adding noise. If I get the wrapper to work, would you @Dokans and @Plavit or someone else be able to write some code that can add the noise?

[Plavit]

@SijmenHuizenga @kaadam92 Thanks for the compliment and the feedback, I understand Your limitations. 👍

Just a quick response for now:
Thanks Sijmen for a detailed specification on how the depth cameras should work, I think your ideas are great and should definitely be revisited later 😉 Regarding the noise work - we would love to help, but as a first-year Driverless team, I highly anticipate us to have a lot of trouble just making something that can at least somewhat go around the virtual track, so I honestly do not see it as feasible for us to find time for this in the coming two weeks.

However, we are super glad that this project exists and do plan to use it as our primary simulator for testing in the coming years, so I think we are very open to participating on the impementation you propose after the competition is over if that would work for you 😎

[SijmenHuizenga]

So I just spend 40 minutes implementing the ros bridge adaptation for the depth cameras. You can see the progress in #181 It looks like this:

https://www.youtube.com/watch?v=CmHY8ng7UgM

Video was recorded using this settings.json file.

The average FPS of the depth camera was 6 fps compared to 20 of the rgb camera that was running at the same time. Improving performance is probably impossible without major changes, see #43.

[SijmenHuizenga]

Thanks to some resources shared by @Plavit I managed to add some noise in the depth images. It now looks like this:

https://www.youtube.com/watch?v=T0m2-ZLvZU0

[Plavit]

@SijmenHuizenga Wow, very cool, thank you so much for looking into it and dedicating so much energy!

I think you got it pretty much exactly where I thought we could get this weekend, great job! Would you consider keeping it in the project for the release? I would say that we would probably not use it due to the FPS, but we would be very interested to test it and work on it further later 😉

[SijmenHuizenga]

@lussekatter (the FSOnline official for driverless simulator event) and I just talked and we agreed that the depth camera's in the current state are not ready to be used during competition. They have not been tested thoroughly enough. It is more important to have a stable system then to have more features.

Amazing to hear that you are eager to test it! I will merge the depth-sensors so that you can do so. And in turn it will be included in the v1.0.0 release. But that doesn't change the fact that nobody is allowed these depth cameras during competition. This statement will also be added to the simulator handbook.

[Plavit]

@SijmenHuizenga @lussekatter Hi, sure, we are fine with this solution. Overall, we did not expect to have anything considering the state of events by the start of this week, so having something to build on now is amazing - again, I reiterate that we are very interested in improving this feature for our own usage in the future, so this has significant added value on its own.

Again, thank you for your efforts!

[Plavit]

Further development ideas now continuing here: #187