how can I select Freenect as source when i use ROS 2

[damavand1]

When I open rtabmap using "rtabmap" command in terminal I see full rtabmap and I can select freenect for xbox 360 kinect like this picture

but when i using this command
"ros2 launch rtabmap_launch rtabmap.launch.py rtabmapviz:=true rviz:=true"

both rtabmap and rviz opened but I can't select my camera, select source is not showing to select freenect

how can I tell rtabmap use freenect and xbox 360 kinect as input in second command and see data in rviz ?

tnx

[matlabbe]

I am not aware if there is a freenect package for ROS2. If there is one, you have to launch the freenect ros node at the same time. rtabmap_ros is not using built-in camera drivers in rtabmap library, but rely on ros camera topics.

I've found this package https://github.com/fadlio/kinect_ros2, based on the doc, you would have to do something like:

ros2 run kinect_ros2 kinect_ros2_node --ros-args -r __ns:=/kinect

ros2 launch rtabmap_launch rtabmap.launch.py \
    rtabmap_args:="--delete_db_on_start" \
    rgb_topic:=/kinect/image_raw \
    depth_topic:=/kinect/depth/image_raw \
    camera_info_topic:=/kinect/camera_info \
    frame_id:=kinect_rgb \
    approx_sync:=true \
    qos:=1 \
    rviz:=true

However, it should be FREENECT_DEPTH_REGISTERED on this line https://github.com/fadlio/kinect_ros2/blob/f1aafcfb36cae1f07f1a4654ebd3b70fb3b706db/src/kinect_ros2_component.cpp#L70 to be compatible with rtabmap.

UPDATE: added __ns:=/kinect instead of __ns:=kinect

[damavand1]

hi

I uploaded here : https://we.tl/t-fWcv00OCog
also i uploaded a bash file that i run commands step by step

thank you for your help

[matlabbe]

The header is not valid for /kinect/image_raw (null timestamp and no frame_id):

ros2 topic echo /kinect/image_raw --no-arr

header:
  stamp:
    sec: 0
    nanosec: 0
  frame_id: ''
height: 480
width: 640
encoding: rgb8
is_bigendian: 0
step: 1920
data: '<sequence type: uint8, length: 921600>'

The timestamp is not set in /kinect/camera_info too.

The depth image looks registered to rgb camera though, so if you fix the null timestamp it should work.

[damavand1]

thank you,
now, how can I solve this problem ?
how can I fix the null timestamp ?

if you see my bash file i do this step like your code:
However, it should be FREENECT_DEPTH_REGISTERED on this line https://github.com/fadlio/kinect_ros2/blob/f1aafcfb36cae1f07f1a4654ebd3b70fb3b706db/src/kinect_ros2_component.cpp#L70 to be compatible with rtabmap.

[matlabbe]

You will have to look at the camera node and see why the stamp is not set for the rgb image topic and for the camera info topic

[damavand1]

I'm so sorry
I'm new in ROS and Rtabmap
I don't know, how I can do "look at the camera node and see why the stamp is not set for the rgb image topic and for the camera info topic"

can you please help me ?

[Tjaart-Hobby]

I am another little fish in this pond attempting the same with the same issues.
My two cents: When the kinect_ros2 workspace is build, it builds with error: fadlio/kinect_ros2#5
There's mention of 'timestamp', perhaps the issue could be with the colcon build not resolving rosdep definition for [libfreenect]?

[matlabbe]

I looked at the kinect_ros2 package and fixed all issues to make it compatible with rtabmap in this fork:
https://github.com/matlabbe/kinect_ros2

I also added a launch file example:
https://github.com/introlab/rtabmap_ros/blob/ros2/rtabmap_examples/launch/kinect_xbox_360.launch.py

Tested on Ubuntu 20.04 with libfreenect-dev installed:

sudo apt install libfreenect-dev

[Tjaart-Hobby]

Thank you, I will be home from Tuesday. If it works I would like to buy you a pizza ;)

[Tjaart-Hobby]

I am sorry to report it does not work. ----scratch that---- IT WORKS 👍
Adding the xbox kinect sensor with ros2 is such a huge improvement for rtabmap.

Method used:

1. Ros2 humble installed on laptop,,, rtabmap installed... Libfreenect-dev installed.. freenect installed ..
2. create new workspace and clone package / mkdir -p newKinect_ws/src && cd src / git clone https://github.com/matlabbe/kinect_ros2 / cd .. / rosdep install --from-paths src --ignore-src -r -y (this has complains about freenect deps) / colcon build (has stedder out error but seems to work)

Test:

• new terminal source newKinect_ws/install/setup.bash / ros2 run kinect_ros2 kinect_ros2_node

• new terminal /ros2 topic list > camra topics NO namespace topics

• new terminal / ros2 run rqt_image_view rqt_image_view clear > check rgb output topic

• new terminal /ros2 run rviz2 rviz2 > no global base topic to point cloid ?

• new terminal / ros2 launch rtabmap_examples kinect_xbox_360.launch.py > empty rviz

• topics terminal recheck topiocs > Same camera topics + namespace topics published by xbox360.launch.py + rtabmap topics

• kinect ros2 node rerun with remapped topics /ros2 run kinect_ros2 kinect_ros2_node --ros-args --remap /rgb/image_raw:=/rtabmap/kinect/rgb/image_raw --remap /rgb/camera_info:=/kinect/rgb/camera_info --remap /depth/image_raw:=/kinect/depth_registered/image_raw

• Wht it WORKS!!
  It didn't yester day but now it does :)
  Here is how it worked for me.(video link to follow as soon as I have strong enough internet to upload to youtube)

update: link https://youtu.be/MBq0YxG41O8

[damavand1]

thanks
I do all step by step, after install Ros an RtamMapRos I do folowing steps:

sudo apt-get install freenect && sudo apt install libfreenect-dev
mkdir -p ~/newKinect_ws/src && cd ~/newKinect_ws/src
git clone https://github.com/matlabbe/kinect_ros2
cd ..
rosdep install --from-paths src --ignore-src -r -y
colcon build
source ~/newKinect_ws/install/setup.bash

ros2 run kinect_ros2 kinect_ros2_node
in new terminal -> ros2 topic list # too see runs topics
ros2 run rqt_image_view rqt_image_view
cd /opt/ros/humble/share/rtabmap_examples && sudo wget https://raw.githubusercontent.com/introlab/rtabmap_ros/ros2/rtabmap_examples/launch/kinect_xbox_360.launch.py
in new terminal -> ros2 launch rtabmap_examples kinect_xbox_360.launch.py
ros2 topic list # too see runs topics
in new termianl -> ros2 run kinect_ros2 kinect_ros2_node --ros-args --remap /rgb/image_raw:=/rtabmap/kinect/rgb/image_raw --remap /rgb/camera_info:=/kinect/rgb/camera_info --remap /depth/image_raw:=/kinect/depth_registered/image_raw

on run final line i get this error:
user1@user1-PC:/opt/ros/humble/share/rtabmap_examples$ ros2 run kinect_ros2 kinect_ros2_node --ros-args --remap /rgb/image_raw:=/rtabmap/kinect/rgb/image_raw --remap /rgb/camera_info:=/kinect/rgb/camera_info --remap /depth/image_raw:=/kinect/depth_registered/image_raw
[INFO] [1701439007.677182776] [kinect_ros2]: camera calibration URL: file:///home/user1/newKinect_ws/install/kinect_ros2/share/kinect_ros2/cfg/calibration_rgb.yaml
[WARN] [1701439007.677507885] [camera_calibration_parsers]: Camera calibration file did not specify distortion model, assuming plumb bob
[INFO] [1701439007.677562571] [kinect_ros2]: camera calibration URL: file:///home/user1/newKinect_ws/install/kinect_ros2/share/kinect_ros2/cfg/calibration_depth.yaml
[WARN] [1701439007.677689481] [camera_calibration_parsers]: Camera calibration file did not specify distortion model, assuming plumb bob
'[ERROR] [1701439013.107137091] [kinect_ros2]: FREENECT - ERROR OPEN'
[ros2run]: Segmentation fault

also
user1@user1-PC:~/newKinect_ws$ ros2 topic list
/depth/camera_info
/depth/image_raw
/parameter_events
/points
/rgb/camera_info
/rgb/image_raw
/rosout

i don't see
/depth/image_raw/compressed
/depth/image_raw/compressedDepth
/depth/image_raw/theora
/rgb/image_raw/compressed
/rgb/image_raw/compressedDepth
/rgb/image_raw/theora

why ?

Second Pizza is here ;-)

@matlabbe @Tjaart-Hobby is there any other connection to you, like Linkedin or ... to collaborate?
I'm traying to build a self driving delivery robot.
Tnx

[Tjaart-Hobby]

damavand1
for the error: [ERROR] [1701439013.107137091] [kinect_ros2]: FREENECT - ERROR OPEN'
[ros2run]: Segmentation fault

You will have to install ,,"call it camera driver" from openfreenect. https://github.com/OpenKinect/libfreenect This driver/library lives outside of ros and can be used in any code you write. You can control the kinect rgb and depth camera and the ir info or even use the kinect motor.
When you can run the example code outside of ros 'freenect-[example]' like open new terminal run: freenect-glview the error should go away.

for compressed topics not published:
"Specialized transports are provided by plugins. image_transport itself provides only "raw" transport so as not to impose unnecessary dependencies on client packages. Other transports will only be available if they are built on your system. On Ubuntu, the ros--base debians include the "compressed" and "theora" transports provided by the image_transport_plugins stack."

see: https://github.com/ros-perception/image_common/tree/humble and google some more about image transports and video encodings.
Your question covers a lot of subjects. In this specific case you would probable end up with something like "sudo apt install ros-humble-image-transport-plugins"
This command "sudo apt install ros-humble-((some package))" is the standard way of installing ros packages.
with "image-transport-plugins" the package you are going to install . When you typed "sudo apt install ros-humble-image" you can double 'tab' and you can see what options are available, you might need more that just the "image-transport-plugins".

To matlabbe
I still want to contribute, but other than thank you I don't see how.
Thank you's are good but can't put fuel in your tank or buy a pizza.

[damavand1]

@Tjaart-Hobby @matlabbe
As you see i installed openkinect using this line first of all

sudo apt-get install freenect && sudo apt install libfreenect-dev

also i can run '$ freenect-glview' successfully but freenect-tiltdemo, freenect-micview i think needs updating firmware using '$ sudo python3 ~/libfreenect/src/fwfetcher.py'

I'm using 1473 model

how can i solve this error:

'[ERROR] [1701439013.107137091] [kinect_ros2]: FREENECT - ERROR OPEN' [ros2run]: Segmentation fault

tnx

[matlabbe]

You may try the original kinect_ros2 repo just to check if it produces the same error or not. If it produces the same error, doublecheck the kinect power cable is connected... though freenect-glview looks working on your side. I tested on a 1414 model.

What is the version of libfreenect-dev? Maybe the version is different than the one on Ubuntu focal (which is 1:0.5.3-2).

apt show libfreenect-dev

[damavand1]

Wow, it solved, now i can see scanning map in Rtabmap

thanks your support @matlabbe @Tjaart-Hobby

[damavand1]

every things fine

But when I start using this command
ros2 launch rtabmap_examples kinect_xbox_360.launch.py

It work's but just a few seconds (usually 10-20 seconds), every things will be very slow and freeze
as you see in my uploaded video I move my camera very slowly, but it freeze very soon,
https://we.tl/t-vbUGYlZR0F

along this video I'm moving camera slowly but after some second nothings works and every things freeze.

also, when I use $ ratbmap , after some seconds every thins be very slow in may laptop (i9+16GB+SSD), and CPU usage from all cores be near 100%
It's killing my laptop.

how can I solve this problem ?

thanks

[matlabbe]

rtabmap itself would not use 100% of all your cores (maybe 1 or 2 at most). Can you do top in a terminal to see which processes are using all the CPU? If CPU usage is coming from rtabmap nodes, it could be related to OpenMP. From here:

High CPU Usage (100% all cores/threads): Most of the cases, this is related to OpenMP, try setting environment variable OMP_WAIT_POLICY to passive.

[damavand1]

I set OMP_WAIT_POLICY to passive.

now when i run rtabmap cpu usage normal and lower than before, but now speed is low yet, and rtabmap be very slow and freeze after some seconds after start, just 5 to 10 second in beginning is good and after that i have many problems in speed

[damavand1]

I tested on windows 11 on same laptop
on windows it's very very high speed and good experience

[matlabbe]

On windows, did you use the latest binary release?

On ubuntu, did you recompile rtabmap from source? which version are you using (git log)? Are some dependencies built from source? If so, make sure they are built in Release mode.

[damavand1]

yes
on windows speed of rtabmap is very very good, and I installed latest binary release from this page :
RTABMap-0.21.0-win64-cuda11.7.exe

on ubuntu, I installed Rtabmap-ros using this :
$ sudo apt install ros-humble-rtabmap-ros

the only compiled on my ubuntu is:
git clone https://github.com/matlabbe/kinect_ros2

every other things installed on my ubuntu, installed using $ apt install

[matlabbe]

Make sure you built the package in Release mode (I think humble builds in Debug by default):

colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release