ROS Unity Videostream - half image

[pianita]

Hello,
I am sending a videostream (CompressedImage msgs) from ROS (Melodic) to Unity (2020.2.1f1). But when when applying the image as texture I can only see half the image (as if the data buffer was not loaded fully). The same happens if I send only one image - sometimes the whole image is applied as texture, sometime only half. Sometimes I also get the following errors:

Exception raised!! System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer.
Parameter name: bytes
at System.Text.UTF8Encoding.GetString (System.Byte[] bytes, System.Int32 index, System.Int32 count) [0x00057] in <9577ac7a62ef43179789031239ba8798>:0
at RosMessageGeneration.Message.DeserializeString (System.Byte[] data, System.Int32 offset, System.Int32 stringLength)

Exception raised!! System.ArgumentOutOfRangeException: Non-negative number required.
Parameter name: count
at System.Text.UTF8Encoding.GetString (System.Byte[] bytes, System.Int32 index, System.Int32 count) [0x0003a] in <9577ac7a62ef43179789031239ba8798>:0
at RosMessageGeneration.Message.DeserializeString (System.Byte[] data, System.Int32 offset, System.Int32 stringLength)

Maybe someone else had the same problem or any ideas on how I could improve this?
Thanks!

[mpinol]

Hi @pianita ,

What version of the TCP Connector are you using? You can find this in the Package Manager window or in the package file in the ROS-TCP-Connector directory.

Aside from the Exceptions, I ran into a similar issue a while ago where only the top half of an image was being applied to the texture. I believe I resolved the issue by setting the width and height of the image game object to match the resolution of the image coming from ROS. If you don't know the resolution of the images coming from ROS you should be able to find it by doing something like math.sqrt(len(CompressedImage.data) / 4) in Python. The 4 is for the RGBA values of the image.

Would you be able to share a simple project that reproduces this issue so that I can take a look?

[pianita]

Hi @mpinol,

The ROS TCP Connector version is 0.0.3.
I have to display the videostream on a non-image game object and do so by applying a 2Dtexture.
I attached a ROS workspace (rosrun server_endpoint.py, load params.yaml, rosrun image_publisher.py) and for the Unity Project I used the PickandPlace Project with a new scene and the attached Subscriber Script.

catkin_ws (copy).zip
CameraSubscriber.zip

Thanks for your reply! :)

[chirmy]

view it on GitHub

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "Unity-Technologies/Unity-Robotics-Hub" <notifications@github.com&gt;; 发送时间:&nbsp;2021年1月27日(星期三) 下午5:50 收件人:&nbsp;"Unity-Technologies/Unity-Robotics-Hub"<Unity-Robotics-Hub@noreply.github.com&gt;; 抄送:&nbsp;"Subscribed"<subscribed@noreply.github.com&gt;; 主题:&nbsp;Re: [Unity-Technologies/Unity-Robotics-Hub] ROS Unity Videostream - half image (#101) Hi @mpinol, The ROS TCP Connector version is 0.0.3. I have to display the videostream on a non-image game object and do so by applying a 2Dtexture. I attached a ROS workspace (rosrun server_endpoint.py, load params.yaml, rosrun image_publisher.py) and for the Unity Project I used the PickandPlace Project with a new scene and the attached Subscriber Script. catkin_ws (copy).zip CameraSubscriber.zip Thanks for your reply! :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mpinol]

Hey @pianita

Thank you for the files!

I tested everything using ROS TCP Connector version 0.1.2-preview and was able to get the full image displayed on a cube. Version 1.2 includes a variety of improvements and bug fixes that might resolve your issue.

Would you be able to update the TCP Connector and try again?

[pianita]

Hey @mpinol
I tried today with the TCP Connector in version 0.1.2.

If I publish the single compressed image in a loop it gets displayed correctly. But if I publish it once every time I run my image_publisher.py node I still get only half images sometimes. If instead of a single image I publish a stream of different compressed images (the video stream) I also still get half images. If I look at the content of the received message I can see that the lower half of the byte array are 0.

Do you get the full image even in a stream of e.g. two different images?

[at669]

Hi @pianita, I am able to get full images when toggling between two different images in a stream with the TCP Connector version mentioned, where the two images around the size of the provided jpeg. Would you mind sharing an updated version of your Unity project and ROS workspace so we can take another look?

[pianita]

Hi @at669!
Sorry my mistake - toggling images works fine. I am figuring it might really just be the resolution. If I print math.sqrt(len(CompressedImage.data) / 4) of the .jpeg image I get values around 63. For the images of the video stream I get more than double that value.
Do you know until which resolution this should work? I want of course highest quality but also the full image :D.
Thank you!

[at669]

Hi @pianita--sorry for the delay! After some investigation, it seems like more of a specific bottleneck issue, and we're in the process of making some fixes for it that will be added to the repositories soon. In the meantime, feel free to implement them on your end:

1. In ROS-TCP-Endpoint/src/ros_tcp_endpoint/tcp_sender.py's send_unity_message(), we increased s.settimeout to 10. The original value would cause larger images to time out on send.
2. Also in ROS-TCP-Endpoint/src/ros_tcp_endpoint/tcp_sender.py's send_unity_message(), we changed the call from s.send(serialized_message) to s.sendall(serialized_message). Using sendall ensures that the full message will try to be sent.
3. In the Unity ROS-TCP-Connector package's ROSConnection.cs (located in Runtime/TcpConnector), in ReadMessageContents, we changed while (networkStream.DataAvailable && bytesRemaining > 0) to just while (bytesRemaining > 0). It appears that networkStream.DataAvailable can be an unreliable verification for the TCP connection. Note of caution on this point: if the full message is not being sent from ROS, this while loop will hang indefinitely and will need to be force quit. You can implement a timeout on this end as well, or ensure the message is always being fully sent from ROS. We are working on a fix for this.
4. Also in the Unity ROS-TCP-Connector package's ROSConnection.cs ReadMessageContents, we attempt to read the incoming data in chunks instead of as a whole, changing the bytesRead line to int bytesRead = networkStream.Read(readBuffer, totalBytesRead, Math.Min(2048, bytesRemaining));.

You may need to test the exact timeouts and resolutions for your use case, depending on the rate of your stream and resolution of your images. Hopefully these points can help with your project! Let us know if these steps resolve this issue.

[pianita]

Hey @at669,

yes I get the full image now! Thank you so much :)
The videostream is still very jerky but I will do some testing by adjusting resolution, publish rate of my video and timeouts.
Of course if you have a setup you know runs smoothly I would be interested to know the values.
But for me this issue is solved and can be closed.

[at669]

Glad these points were able to help! Using jpegs that translated to a serialized message of below ~250k length byte array seemed to be working relatively well for my simple test case with the referenced PR changes to my Connector and Endpoint, but definitely feel free to test higher or lower values if they work for you.

I will close this issue, but feel free to re-open if you have any follow-ups. Thanks!