Add hand pose estimation

[W567]

New version of #2600 and #2597.

Fixed image color problem in previous PRs, which is caused by wrongly setting the the cv_bridge image encoding format to rgb8 instead of bgr8 in cv2_to_imgmsg(frame, encoding='rgb8') and imgmsg_to_cv2(frame, encoding='rgb8') in d70f5aa.

Changed the color variable name sequence as in d70f5aa from r, g, b to b, g, r, as the previous sequence did cause misunderstanding, though the variable name didn't effect the detection functionality,

The rosbag file is also updated.

[knorth55]

@W567 thank you for your update.
But I cannot still get the detection result correctly with this PR, too.
Also, can you give me the url of the new rosbag file?

EDIT: I found the new rosbag file.

[W567]

@knorth55 I have sent you the gmail, please have a check, thank you very much.

It’s really weird that the detecting part failed, hope the new rosbag works well.

[knorth55]

@W567 I'm running with the latest rosbag, but it doesn't show detection results.
This is probably because I'm running with cpu.
can you try running with cpu in your environment?

[W567]

@knorth55 The latest rosbag is already the bag recorded when running with torch 1.4.0 cpu, and it did work on my computer...

[knorth55]

@W567 can you return the value of this command?

md5sum SRHandNet.pts

[W567]

@knorth55 It should be 9f39d3baa43cf1c962c8f752c009eb14

[knorth55]

I found out that we need to use __future__ division.
093b4d9

maybe your python is not python2.
can you do the following commands to check your python version?

$ which python
/usr/bin/python
$ python --version
Python 2.7.17

[W567]

@knorth55 Oh, yes, I'm running with python 3.7.10 for this program. Thank you for the correction.

[knorth55]

@W567 I tried with my local machine and it works great!
this model is cool! it works with cpu in not slow speed! Great!

Screencast.2021-05-25.20.43.07.mp4

[knorth55]

@knorth55 Oh, yes, I'm running with python 3.7.10 for this program. Thank you for the correction.

@W567 probably, you use conda and do source .anaconda/bin/activate in ~/.bashrc.
For ROS usage, we should not use conda environment, so it is better to remove conda activation line in ~/.bashrc.

[W567]

@knorth55 So nice to hear it works, and thank you so much for helping me with the debugging.

I once used conda and when I started using ros, it did output many errors...so I've already removed the conda activation line in ~/.bashrc. But after that, the python packages is located everywhere with different python versions, so I currently mainly try to export different PYTHONPATH to control the packages for different programs...Maybe I need to tidy up my packages now.

Anyway, thank you again for your advice and debugging.

[knorth55]

@W567 no problem. ROS depends on apt package manager not pip or conda, so it is a bit difficult.
Please make sure that python is python2 installed by apt if you are using Ubuntu18.04 and unset PYTHONPATH before using conda.

[W567]

@knorth55 I see, I'll pay attention, thanks for the instruction.

[knorth55]

@tohirose this PR is super cool, and it also runs on CPU!

[k-okada]

@W567 @knorth55 Great work!
I tried to run this node on my computer, and it works well on CPU mode,but could not run on GPU. Is there something missing?

Is there any reason that you did not use rosdep rule to install PyTorch?
k-okada@7dc2e39

k-okada@p51s:~/catkin_ws/ws_recognition/src/jsk_recognition/jsk_perception/sample$ nvidia-smi 
Thu Jul  8 17:47:10 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.143                Driver Version: 390.143                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P500         Off  | 00000000:02:00.0 Off |                  N/A |
| N/A   57C    P0    N/A /  N/A |    486MiB /  2002MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      6475      G   /usr/lib/xorg/Xorg                            93MiB |
|    0      7343      G   /usr/bin/gnome-shell                          52MiB |
|    0      8599      G   /usr/lib/xorg/Xorg                           199MiB |
|    0      8798      G   /usr/bin/gnome-shell                         126MiB |
|    0      9507      G   ...-token=E6518C091B9057DDB7FAC882C9CD65A3    12MiB |
|    0     11420      G   ...AAgAAAAAAAAACAAAAAAAAAA= --shared-files     1MiB |
+-----------------------------------------------------------------------------+
k-okada@p51s:~/catkin_ws/ws_recognition/src/jsk_recognition/jsk_perception/sample$ roslaunch sample_hand_pose_estimation_2d.launch gpu:=0 use_usb_cam:=true
... logging to /home/k-okada/.ros/log/1871de4c-dfc9-11eb-8fa6-00bb605aa0af/roslaunch-p51s-25132.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
WARNING: disk usage in log directory [/home/k-okada/.ros/log] is over 1GB.
It's recommended that you use the 'rosclean' command.

started roslaunch server http://p51s:37989/

SUMMARY
========

PARAMETERS
 * /hand_pose_estimation_2d/gpu: 0
 * /hand_pose_estimation_2d/model_file: /home/k-okada/cat...
 * /rosdistro: melodic
 * /rosversion: 1.14.11
 * /usb_cam/pixel_format: yuyv

NODES
  /
    hand_pose_estimation_2d (jsk_perception/hand_pose_estimation_2d.py)
    image_view_p51s_25132_639236277455545427 (image_view/image_view)
    usb_cam (usb_cam/usb_cam_node)

auto-starting new master
process[master]: started with pid [25149]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 1871de4c-dfc9-11eb-8fa6-00bb605aa0af
process[rosout-1]: started with pid [25169]
started core service [/rosout]
process[usb_cam-2]: started with pid [25177]
[ WARN] [1625734036.699279443]: Camera calibration file /home/k-okada/.ros/camera_info/head_camera.yaml not found.
[ WARN] [1625734036.876745356]: unknown control 'focus_auto'

process[hand_pose_estimation_2d-3]: started with pid [25280]
process[image_view_p51s_25132_639236277455545427-4]: started with pid [25281]
[INFO] [1625734038.693067]: Loading model
[INFO] [1625734038.905664]: Finished loading SRHandNet model to cpu

[knorth55]

for pytorch+gpu, we need to install specific version manually.
https://pytorch.org/get-started/previous-versions/

but if you install gpu, you can use gpu for this demo, I think, but I haven't tried yet.

[W567]

@k-okada Sorry for the late response.

I've switched to Driver Version 390.143 in 'Software & Updates' to test the program. And it did output the same result.

I think it is because in the hand_pose_estimation_2d.py, we check torch.cuda.is_available() to decide whether load model into cpu or gpu. And I tested as below:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.143                Driver Version: 390.143                   |
|-------------------------------+----------------------+----------------------+
wu@LT18-820:~/Desktop$ python
Python 2.7.17 (default, Feb 27 2021, 15:10:58) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False

It is likely that downloading nvidia-driver-390 from 'Software & Updates' won't download the cuda toolkit at the same time. So torch.cuda.is_available() is kept false, results in loading model into cpu even with the gpu argument set to 0.

As @knorth55 mentioned, we need to install specific version of pytorch, gpu driver and cuda. It is recommended to use pytorch>=1.4.0 to run the program. According to the official website of pytorch, CUDA 9.2 is the minial requirement. And according to the official website of nvidia, Driver Version >= 396.26 is the minimal requirement for CUDA9.2.

https://pytorch.org/get-started/previous-versions/
# CUDA 9.2
conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=9.2 -c pytorch

https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html
CUDA 9.2 (9.2.88) Linux x86_64 Driver Version >= 396.26

So, could you please update your driver and test the program again? I don't know if you are using 'Software & Updates' to download the gpu driver, but it works well when I use it to switch to the driver version of 465, in this case, the cuda toolkit is downloaded automatically.

Or if you are downloading the driver manually, please download

• Driver Version >= 396.26
• CUDA >= 9.2
• pytorch >= 1.4.0

Is there any reason that you did not use rosdep rule to install PyTorch?

I'm not familiar with rosdep before, and would like to use it from now on. Thank you for the tips.

[knorth55]

@k-okada @W567

I found a solution with nvidia-driver-390 + CUDA9.1 (default nvidia driver bionic)
We can use pytorch>=1.1.0 with this node.

First, download from torch-1.1.0-cp27-cp27mu-linux_x86_64.whl and torchvision-0.3.0-cp27-cp27mu-manylinux1_x86_64.whl https://download.pytorch.org/whl/cu90/torch_stable.html.

Then install them

pip install --user torch-1.1.0-cp27-cp27mu-linux_x86_64.whl
pip install --user torchvision-0.3.0-cp27-cp27mu-manylinux1_x86_64.whl

Finally, we can run this launch with usb_cam

roslaunch jsk_perception sample_hand_pose_estimation_2d.launch gpu:=0 use_usb_cam:=true

$ nvidia-smi
Fri Jul  9 15:24:40 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.143                Driver Version: 390.143                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro M1000M       Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   50C    P0    N/A /  N/A |   1001MiB /  2002MiB |     80%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2768      G   /usr/lib/xorg/Xorg                           164MiB |
|    0      2949      G   /usr/bin/gnome-shell                         125MiB |
|    0      6338      C   python                                       581MiB |
|    0      6352      C   /opt/ros/melodic/lib/image_view/image_view    24MiB |
|    0      9748      G   ...AAgAAAAAAAAACAAAAAAAAAA= --shared-files    91MiB |
+-----------------------------------------------------------------------------+

It works 12Hz with GPU.
and it is 4 times faster than CPU 3Hz in my lenovo thinkpad p-50.

$ rostopic hz /hand_pose_estimation_2d/output/vis
subscribed to [/hand_pose_estimation_2d/output/vis]
average rate: 12.269
        min: 0.074s max: 0.110s std dev: 0.01281s window: 12
average rate: 12.268
        min: 0.074s max: 0.110s std dev: 0.01255s window: 24

srhand_gpu.mp4

[k-okada]

@W567 @knorth55 thank you! I also successfully run hand pose estimation on my 18.04 machine and update the GPU information at #2612

I also update the code to print warning message when GPU is not available, see
W567#1