How to visualize data from rviz to Unity?

[linhdoan8]

Issue Template © Siemens AG, 2017-2018
Author: Dr. Martin Bischoff (martin.bischoff@siemens.com)
-->

• I am at the right place and my issue is directly related to ROS#. General technical questions I would post e.g. at ROS Answers or Stackoverflow. For library-specific questions I would look for help in the corresponding library forums.
• I have throughly read the Contributing Guideline and writing this issue is the right thing to do in my case.

---

I have a question!

• I searched the Wiki, open and closed issues for an answer. I tried my best in finding the answer by myself without success. I believe that the discussion we will have in this issue, and the solutions we might find, will help me, and likely other community members who have a similar problem.

Here is my question:
Hi all,
I am completely new to Ros and Unity and was searching for a way to visualize rviz sensor data like laser scan or mapping data in Unity to deploy to the Hololens. Fortunately I found ROS# which makes that possible. I would like to ask which are the approaches to visualize data from rviz especially the laser scan data. I would subscribe to the topic /base_scan but obviously I have to find a way to convert the data from that topic to a format to appropriately visualize in Unity (so that the Unity scene looks like Rviz). I saw the video 'ROS#: Robot Teleoperation via Unity ' - (https://www.youtube.com/watch?v=OytzagQirrk) from you and would like to achieve something similar. Is there any code available because in the wiki there is no part explaining that demo. I hope for your reply and many thanks in advance!
Best regards

[MartinBischoff]

Hi @lee4138 :
The turtlebot demo shown is this video is not publicly available. Please note that there is also no laser scan data illustrated in this video either.

When you go through the example applications provided in the Wiki, you will get familiar with ROS# and will be able to easily adjust this communication framework to your individual application.

@faizan-m is also currently working on LaserScan message import to Unity (see issue #78 ). Maybe you two want to exchange about it here.

[faizan-m]

Hi @lee4138!

I am actually in the process of creating an Augmented Reality Framework in Unity for ROS that would ideally be implementable on any devices supported by Unity, including Hololens. Unfortunately, it is still under development at the moment and not public.

I did not find any easy shortcuts to recreate rviz in Unity so I am taking the slow and steady approach of visualizing them myself.

[linhdoan8]

Hi @MartinBischoff and @faizan-m,

many thanks for your reply! So with slow and steady way you mean to convert the raw data into something that will fit into unity? If you have already implemented some converter code would you mind to share please?

I watched this video 'HoloLens + ROS + AR' (https://www.youtube.com/watch?v=TkorsuwkU1g), which shows a laserscan visualization similar to rviz into the hololens.

Unfortunately the author didnt reply to my request yet. Do you have any idea which approach he might have used? And do you have achieved something similar?
Hope for your reply. Thanks!

[faizan-m]

Yeah I am just creating graphics based off the values through ROS#. I don't really have any "converter code" in between, everything is wired up very directly the way I am doing it so there are no discrete pieces that might be useful to you at the moment.

I am not sure about the approach in the video but it has a lot of latency that I have not experienced.

[linhdoan8]

okay, thanks for your answer! I will try that too and report.

[linhdoan8]

Hi all,

i have now build up the connection and can receive PointCloud2 messages which I converted from the Laser Scan data. For that I have build a SensorReceiver.cs file and attached it to the Subscriber Component of the RosConnector, like done in the tutorials with the JoyReceiver. I extracted x y z values
out of the PointCloud2 messages.

I would like to ask what you would suggest for the next steps to visualize the data correctly in Unity. My idea is to use these xyz points to project circles into the Unity scene at that points but I am not sure about few things. I am not sure what the xyz values represent exactly, they have values ranging from (-5 to 2.5) someone have experince in working with PointClouds in Unity?

I think that I also need to do some kind of a coodrinate transformation from the RVIZ map to the Unity map in order to project the circles at the right spot. How can I do that? And where do I have to attach the scripts ? (sorry if the questions might seem basic , I am a complete beginner) .

[MartinBischoff]

Hi @lee4138 !

• You will have to do a coordinate transform. Unity is left-handed, Y-up. All ROS applications are right-handed Z-up.
  The scripts Ros2Unity and Unity2Ros in this class might be of help for you.
• I suggest not to instantiate / update individual Gameobjects per 3d point. That will cost forbiddingly much resources.
• instead update the vertices of one (or a set of) Mesh(es) in Unity with your 3d point coordinates. Here is a nice HowTo. Look for a simple high-performance shader.

Though 3D point cloud illustrations are really fun, please understand that I can't take the time to give further application-specific coding support.

For new technical problems please open a new issue in the future.

[linhdoan8]

Dear @MartinBischoff,
many thanks for your precious tipss! I will try them!

[kosenhitatchi]

@lee4138 Great initiative :D
I'm actually looking forward to display the point cloud of a RGBD kinect-like cam in unity for VR purpose.
How far are you with your implementation? Is it public?

Sincerely

[linhdoan8]

Hi @kosenhitatchi,
I ve implemented a visualization method which is working although not as efficient yet. It is not public yet but I will publish this as soon as some clean up is done and comments are added (latest till next week). Then I will inform you! :)

[linhdoan8]

Hi @kosenhitatchi,
I uploaded my Implementation for visualizing the laserscanner on my fork under PCL2 Handling.rar .
I couldnt find time to comment all and tidy it up though. Hope it still helps somehow!

[kosenhitatchi]

@lee4138 , what a wonderful new. I'll try your code right away and give you some feedback as soon as I can.

[kosenhitatchi]

--I'm sorry in advance if I'm spaming this thread a little bit--
@lee4138 I'm actually in the process of updating your code to work with the code refactorisation (see #59).

Could you tell me what each of your classes are doing.
I do not understand what SP2.cs is supposed to do.? thanks

[linhdoan8]

Hi @kosenhitatchi,
SP2 stands for SpawnPoints2 and is creating new gameobjects based on the incomming position data which have to be of PointCloud2 type.

Therefore the SensorReceiver.cs class, which is working like the Subscriber classes in the ROS# Examples, will receive the PCL2 Data from ROS. You have to drag a subscriber to your unity scene in the ROS Connector and as Message Receiver put in SensorReceiver.cs (SensorReceiver has to be a seperate empty Gameobject where you drag the sensorreceive.cs script into) , the topic to subscribe is the topic which produces the PointCLoud2 Data (in your case the kinect) and drag a sphere gameobject to the field 'point' of Sensorreceive and in the field youbot drag a reference gameobject from which the PCL Data will be displayed .(See ROS# Examples on how to subscribe to ROS topics).

DataPoint.cs is the class for the DataPoints which are created within SP2.cs.
SP2.cs is working as following: Incomming data will be checked if the position is in the dictionary dataPoints. If not, a new game object will be created and a new dataPoint object containing information about position and a timestamp (defined in the dataPoint.cs class) will be added to the dict. If yes, the timestamp of that dataPoint will be updated. When a specific time nextDeletionTime is passed, the class checks all dataPoints in the dictionary and deletes dataPoints with a timeStamp < cutoffAge. Thats it. This way new Points will be displayed and old points deleted.

I know this is a inefficient way of solving the problem and better solutions would be to use object pooling where you dont delete the points but make them invisible and reuse the gameobjects rather than delete them. Also the ideas of Martin Bischoff are more effective. But since I have to implement some other stuff I didnt tried that out yet. If you implement a more effective solution please let me know!

Ps.: I will have a full documentation about my project soon which also includes some flow charts, and diagramms on how the PCL Handling classes work. I can provide that to you as well.
Hope this helps!

[akashcastelino]

Hey @kosenhitatchi @lee4138 were you all able to visualise a pointcloud using ros sharp ?

I have used two subscribers that subscribe to the RGB and depth images from a realsense d435 and use the constructor in pointcloud.cs to create an object with the pointcloud data, but I'm receiving 'NullReferenceException: Object reference not set to an instance of an object'.

If you ll can share the work youll have done with the pointcloud visualision in unity and or any help would be appreciated.

[kosenhitatchi]

Good evening @akashcastelino,
I recently had to pause my development on this topic due to external factor, thus I did not advance much on this issue.
Keep me updated though :-)

[stevensu1838]

Hi buddy,
Have you done your documentation@lee4138? I need some help on the same topic. Help please @lee4138

[AlbertoGhiotto]

Hi @lee4138, @akashcastelino
I'm trying to visualize a point cloud map constructed using rtabmap which publishes this message:
topic: /rtabmap/mapData
type: rtabmap_ros/MapData
I successfully connected a linux pc running ROS and rviz which is visualizing the map with another pc running unity using this tutorial but I don't actually know how to visualize the point cloud map on unity.
Do I have to write a new c# script? If so where can I find some guidelines to do so?
Thanks

[linhdoan8]

Hi all,  For visualization you need additional scripts that work with the incomming ROS data.  I managed to visualize the pointclouds within the Unity Application and Hololens using different scripts and classes. Please refer to my code that I uploaded (as zip file) and posted in an answer above. There is also an explanation to that in the answers. I will try to make a git for the project of mine but dont know when exactly :/  Hope this helps and if there are specific questions to the code please dont hesitate to ask, when I have time I will answer!  Best regards On 12/02/2019 06:57:21, stevensu1838 <notifications@github.com> wrote: Hi @AlbertoGhiotto [https://github.com/AlbertoGhiotto], I am afraid you won't get any feedback on this topic from these experts here. I totally understand your problems and I tried it before. But it seems you have to be an expert of Unity to get the job done and no instructions online at all. Please just google and take a look at this thesis. Visualization of Robotic Sensor Data with Augmented Reality Hopefully, you can make it. Also, I sincerely hope the experts here can give some guidance on this topic. Coz it is really hard for our non-experts to really do something with ROS#. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [#79 (comment)], or mute the thread [https://github.com/notifications/unsubscribe-auth/ATYn-b1eZ2U7a6vrVcccnX5Hbyn5AUXlks5vMlfAgaJpZM4U5dWT].

[Dkrhub]

@linhdoan8 Hello, I am doing the same job as you recently, but I have no idea, based on your work, to visualize the point cloud in unity, can provide me with the code for reference, I would be very grateful

[Janebek]

where is your code ?