[jsk_perception] PeoplePoseEstimation2D #2115

iory · 2017-06-16T04:29:32Z

Add people_pose_estimation_in_2d.

wkentaro · 2017-06-17T06:10:44Z

jsk_perception/node_scripts/people_pose_estimation_2d.py

+
+        pose_estimated_img, people_joint_positions = self.pose_estimate(img)
+        pose_estimated_msg = br.cv2_to_imgmsg(
+            pose_estimated_img.astype(np.uint8))


Is br.cv2_to_imgmsg(..., encoding="bgr8"); fine?

Thanks! Fixed it

wkentaro · 2017-06-17T06:12:07Z

jsk_perception/node_scripts/people_pose_estimation_2d.py

+        img = br.imgmsg_to_cv2(img_msg, desired_encoding='bgr8')
+        pose_estimated_img, people_joint_positions = self.pose_estimate(img)
+        pose_estimated_msg = br.cv2_to_imgmsg(
+            pose_estimated_img.astype(np.uint8))


same as above.

wkentaro · 2017-06-17T06:17:10Z

jsk_recognition_msgs/msg/PeoplePose.msg

@@ -0,0 +1,2 @@
+string[] limb_names
+geometry_msgs/Point32[] points


Why not we use geometry_msgs/Pose with nan rotation values ((0, 0, 0, 0) for quaternion),
expecting the future work of estimating rotation of human limbs.

Good suggestion!
Changed Point32 to Pose

wkentaro · 2017-06-17T06:18:12Z

Anyway, this is a really nice feature 🎉

…e.msg

wkentaro · 2017-06-18T01:23:26Z

jsk_perception/node_scripts/people_pose_estimation_2d.py

@@ -149,7 +149,7 @@ def _cb_with_depth(self, img_msg, depth_msg, camera_info_msg):

        pose_estimated_img, people_joint_positions = self.pose_estimate(img)
        pose_estimated_msg = br.cv2_to_imgmsg(
-            pose_estimated_img.astype(np.uint8))
+            pose_estimated_img.astype(np.uint8), encoding='bgr8')
        pose_estimated_msg.header = img_msg.header
        pose_estimated_msg.encoding = "bgr8"


this can be removed.
and same below.

what indicates 'this'? astype(np.int32)? encoding='bgr8'?

pose_estimated_msg.encoding = "bgr8"

furushchev

Great feature! I'll use and would like to give you a feedback.
Please note my comment.

furushchev · 2017-06-21T07:21:24Z

jsk_perception/sample/sample_people_pose_estimation_2d.launch

+    </rosparam>
+  </node>
+
+  <include file="$(find opencv_apps)/launch/hsv_color_filter.launch" >


@iory I cannot see hsv_color_filter.launch in this directory. Will you create another pull request to opencv_apps?

In opencv_apps repository, hsv_color_filter.launch exists.
https://github.com/ros-perception/opencv_apps/blob/indigo/launch/hsv_color_filter.launch

Oh, sorry, thanks. It only took a lot of time to pull this repository...

furushchev · 2017-06-21T08:40:04Z

Traceback (most recent call last):
  File "/home/demo/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 504, in <module>
    PeoplePoseEstimation2D()
  File "/home/demo/ros/indigo/src/jsk-ros-pkg/jsk_common/jsk_topic_tools/src/jsk_topic_tools/transport.py", line 24, in __call__
    obj = type.__call__(cls, *args, **kwargs)
  File "/home/demo/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 101, in __init__
    self._load_model()
  File "/home/demo/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 109, in _load_model
    self._load_chainer_model()
  File "/home/demo/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 115, in _load_chainer_model
    self.func = pickle.load(open(model_file, 'rb'))
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.7/pickle.py", line 1126, in find_class
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute '_recover_parameter

Something is wrong in detection node...
@iory Which version of chainer do you use?
We have currently 1.24.0 (since chainer 2.X does not support TK1 (which is cuda 6.5) any more.)

iory · 2017-06-21T09:19:41Z

I am using chainer v2.0.0.
In chainer 1.24.0, caffe function's behavior is different.
We should change,

import chainer.links.caffe -> import chainer.functions.caffe

https://github.com/jsk-ros-pkg/jsk_recognition/pull/2115/files#diff-ecc75cd236b801b628c0107ef250fea5R11
Also, the pickled model file should be changed.
Please download following file, and change load file argument from sample launch file.
https://drive.google.com/open?id=0B_NiLAzvehC9Z0h1OGlIcDRtckU
Please try it and teach me whether you succeed.

furushchev · 2017-06-21T13:15:37Z

@iory Thanks a lot for advice.
I download the pretrained data for v1.24 and changed files:

diff --git a/jsk_perception/node_scripts/people_pose_estimation_2d.py b/jsk_perception/node_scripts/pe
index ed23542..660e9c9 100755
--- a/jsk_perception/node_scripts/people_pose_estimation_2d.py
+++ b/jsk_perception/node_scripts/people_pose_estimation_2d.py
@@ -8,7 +8,8 @@ import pickle
 
 import chainer
 import chainer.functions as F
-import chainer.links.caffe
+#    import chainer.links.caffe
+import chainer.functions.caffe
 from chainer import cuda
 import cv2
 import matplotlib
diff --git a/jsk_perception/sample/sample_people_pose_estimation_2d.launch b/jsk_perception/sample/sam
index 76eff5d..3ef7c2e 100644
--- a/jsk_perception/sample/sample_people_pose_estimation_2d.launch
+++ b/jsk_perception/sample/sample_people_pose_estimation_2d.launch
@@ -1,7 +1,7 @@
 <launch>
 
   <arg name="gui" default="true" />
-  <arg name="GPU" default="-1" />
+  <arg name="GPU" default="0" />
   <arg name="INPUT_IMAGE" default="/camera/rgb/image_rect_color" />
   <arg name="INPUT_DEPTH_IMAGE" default="/camera/depth_registered/hw_registered/image_rect_raw" />
   <arg name="INPUT_CAMERA_INFO" default="/camera/rgb/camera_info" />
@@ -22,7 +22,7 @@
     <remap from="~input/depth" to="$(arg INPUT_DEPTH_IMAGE)" />
     <rosparam subst_value="true">
       gpu: $(arg GPU)
-      model_file: $(find jsk_perception)/trained_data/pose_estimation_2d_chainermodel.pkl
+      model_file: $(find jsk_perception)/trained_data/pose_estimation_2d_chainermodel_v1.24.pkl
       with_depth: true
       scales: [0.38]
       stride: 8

but still no luck.

[ERROR] [WallTime: 1498050794.349546] [1497924863.451970] bad callback: <bound method Subscriber.callback of <message_filters.Subscriber object at 0x7f8f101a2290>>
Traceback (most recent call last):
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rospy/topics.py", line 720, in _invoke_callback
    cb(msg)
  File "/opt/ros/indigo/lib/python2.7/dist-packages/message_filters/__init__.py", line 74, in callback
    self.signalMessage(msg)
  File "/opt/ros/indigo/lib/python2.7/dist-packages/message_filters/__init__.py", line 56, in signalMessage
    cb(*(msg + args))
  File "/opt/ros/indigo/lib/python2.7/dist-packages/message_filters/__init__.py", line 229, in add
    self.signalMessage(*msgs)
  File "/opt/ros/indigo/lib/python2.7/dist-packages/message_filters/__init__.py", line 56, in signalMessage
    cb(*(msg + args))
  File "/home/furushchev/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 154, in _cb_with_depth
    pose_estimated_img, people_joint_positions = self.pose_estimate(img)
  File "/home/furushchev/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 223, in pose_estimate
    return self._pose_estimate_chainer_backend(bgr)
  File "/home/furushchev/ros/indigo/src/jsk-ros-pkg/jsk_recognition/jsk_perception/node_scripts/people_pose_estimation_2d.py", line 245, in _pose_estimate_chainer_backend
    outputs=['Mconv7_stage6_L2', 'Mconv7_stage6_L1'])
  File "/usr/local/lib/python2.7/dist-packages/chainer/links/caffe/caffe_function.py", line 193, in __call__
    output_vars = func(*input_vars)
  File "/usr/local/lib/python2.7/dist-packages/chainer/links/caffe/caffe_function.py", line 569, in __call__
    return self.caffe_func[self.name](*xs, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/chainer/links/connection/convolution_2d.py", line 109, in __call__
    deterministic=self.deterministic)
AttributeError: 'Convolution2D' object has no attribute 'deterministic'

furushchev · 2017-06-21T13:17:35Z

With pretrained data for v1.24, it may also try to search File "/usr/local/lib/python2.7/dist-packages/chainer/links/caffe/caffe_function.py", not at functions?

iory · 2017-06-21T13:48:28Z

It means that you cannot import chainer.functions.caffe?
In my environment,

In [1]: import chainer
In [2]: chainer.__version__
Out[2]: '1.24.0'
In [3]: import chainer.functions.caffe
In [4]:

furushchev · 2017-06-21T13:55:07Z

I can import chainer.functions.caffe, that's no problem! I mean, in self.func , chainer.links.caffe is referred. 2017年6月21日(水) 22:48 iory <notifications@github.com>:

It means that you cannot import chainer.functions.caffe? In my environment, In [1]: import chainer In [2]: chainer.__version__ Out[2]: '1.24.0' In [3]: import chainer.functions.caffe In [4]: — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#2115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB0B0LKordr1dN7Cj6AoVo-zkq1AbJ-Pks5sGR8tgaJpZM4N8AvY> .

-- -- ⌘ Yuki Furuta

iory · 2017-06-21T14:08:48Z

Sorry, try this pickle file.
https://drive.google.com/open?id=0B_NiLAzvehC9bGc4ZHRGTzRxZG8

iory · 2017-06-21T14:29:15Z

I try in chainer v1.24.0.
We need to change a little bit code.

+++ b/jsk_perception/node_scripts/people_pose_estimation_2d.py
@@ -283,7 +283,8 @@ class PeoplePoseEstimation2D(ConnectionBasedTransport):
         all_peaks[:, 1] = peaks[:, 0]
         all_peaks[:, 2] = heatmap_avg[peaks.T.tolist()]
         peaks_order = peaks[..., 2]
-        all_peaks = all_peaks[xp.argsort(peaks_order)]
+        peaks_order = chainer.cuda.to_cpu(peaks_order)
+        all_peaks = all_peaks[np.argsort(peaks_order)]
         all_peaks[:, 3] = xp.arange(peak_counter, dtype=np.float32)
         if self.gpu != -1:
             all_peaks = chainer.cuda.to_cpu(all_peaks)

k-okada · 2017-06-21T15:27:06Z

what is the recommended version of chainer ? can we install both v1 and v2 in same environment? #2107

…

-- ◉ Kei Okada 2017-06-21 23:29 GMT+09:00 iory <notifications@github.com>:

I try in chainer v1.24.0. We need to change a little bit code. +++ b/jsk_perception/node_scripts/people_pose_estimation_2d.py @@ -283,7 +283,8 @@ class PeoplePoseEstimation2D(ConnectionBasedTransport): all_peaks[:, 1] = peaks[:, 0] all_peaks[:, 2] = heatmap_avg[peaks.T.tolist()] peaks_order = peaks[..., 2] - all_peaks = all_peaks[xp.argsort(peaks_order)] + peaks_order = chainer.cuda.to_cpu(peaks_order) + all_peaks = all_peaks[np.argsort(peaks_order)] all_peaks[:, 3] = xp.arange(peak_counter, dtype=np.float32) if self.gpu != -1: all_peaks = chainer.cuda.to_cpu(all_peaks) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAeG3CGPiROeGygjZ3C5BiTN89OLn7gaks5sGSi8gaJpZM4N8AvY> .

iory · 2017-06-21T15:48:40Z

In my opinion, we should install chainer v2.0.0 if possible since backward compatibility is broken, it is necessary to write the corresponding code one by one.
https://chainer.org/announcement/2017/06/01/released-v2.html

can we install both v1 and v2 in same environment?

I do not know what 'same' means, but if we install a virtual environment such as pyenv or anaconda, we can install v1 and v2.
But those are incompatible with ROS.

furushchev · 2017-06-22T11:32:47Z

@iory Thanks a lot! It now works well on chainer v1.24.0

…iner v1

[jsk_perception][people_pose_estimation_2d.py] keep compatibility for chainer v1

iory · 2017-06-23T07:15:33Z

@wkentaro
This sample launch file is dependent on pcl and jsk_pcl_ros.
Can I write run_depend in package.xml?

wkentaro · 2017-06-23T07:24:08Z

This sample launch file is dependent on pcl and jsk_pcl_ros.
Can I write run_depend in package.xml?

I prefer to remain jsk_perception independent to jsk_pcl_ros considering compilation time, so cloud you please refine the 3d processing?

iory · 2017-06-23T13:30:20Z

Thanks! I removed pcl dependencies.

iory · 2017-06-23T14:09:15Z

@furushchev
Please review this PR.

furushchev · 2017-06-23T14:17:16Z

@iory Tested on my local machine. It works well. Great Job!

iory · 2017-06-23T16:13:23Z

Travis passed. Could you merge this?

iory added 2 commits June 16, 2017 13:18

[jsk_recognition_msgs] Add people_pose msgs

6ae5418

[jsk_perception/people_pose_estimation_2d] Add

2574c13

iory requested review from furushchev and wkentaro June 16, 2017 04:29

iory changed the title ~~Pose estim~~ PeoplePoseEstimation2D Jun 16, 2017

iory force-pushed the pose-estim branch 2 times, most recently from 8efc53e to ba459c8 Compare June 16, 2017 07:45

[jsk_perception/people_pose_estimation_2d] Add test

79a3fbc

iory force-pushed the pose-estim branch from ba459c8 to 79a3fbc Compare June 16, 2017 08:43

iory changed the title ~~PeoplePoseEstimation2D~~ [jsk_perception] PeoplePoseEstimation2D Jun 16, 2017

wkentaro requested changes Jun 17, 2017

View reviewed changes

wkentaro added feature pkg/jsk_perception labels Jun 17, 2017

[jsk_perception/people_pose_estimation_2d] Fiexed publish img encodings

b446854

iory force-pushed the pose-estim branch from 4f74a14 to 1a955ab Compare June 18, 2017 00:25

[jsk_perception/people_pose_estimation_2d] Modified type of PeoplePos…

166d7b6

…e.msg

iory force-pushed the pose-estim branch from 1a955ab to 166d7b6 Compare June 18, 2017 00:28

wkentaro reviewed Jun 18, 2017

View reviewed changes

iory added 4 commits June 19, 2017 15:17

[jsk_perception/people_pose_estimation_2d] Delete duplicated code

133c068

[jsk_perception/people_pose_estimation_2d] Fixed install sample bag

93fb047

[jsk_recogntion_msgs/PoseArray] Add score

68cdbb2

[jsk_perception/people_pose_estimation_2d] Publishe 2d image pose

63fafab

iory force-pushed the pose-estim branch 4 times, most recently from a17fe9f to 1057a8d Compare June 20, 2017 10:08

iory added 3 commits June 20, 2017 19:24

[jsk_perception/people_pose_estimation_2d] Add people_mask_publisher

8ece541

[jsk_perception/people_pose_estimation_2d] Changed sample bag file

fc912be

[jsk_perception/people_pose_estimation_2d] Fixed missed numpy/cupy type

ee6e9fc

iory force-pushed the pose-estim branch from 1057a8d to ee6e9fc Compare June 20, 2017 10:25

furushchev requested changes Jun 21, 2017

View reviewed changes

[jsk_perception][people_pose_estimation_2d.py] keep compatibility cha…

cd1e0ee

…iner v1

furushchev mentioned this pull request Jun 23, 2017

[jsk_perception][people_pose_estimation_2d.py] keep compatibility for chainer v1 iory/jsk_recognition#2

Merged

Merge pull request #2 from furushchev/pose-estim

cd47720

[jsk_perception][people_pose_estimation_2d.py] keep compatibility for chainer v1

[jsk_perception/people_pose] Delete pcl dependencies

8f247b7

wkentaro approved these changes Jun 23, 2017

View reviewed changes

furushchev approved these changes Jun 23, 2017

View reviewed changes

k-okada merged commit ef9e905 into jsk-ros-pkg:master Jun 24, 2017

furushchev mentioned this pull request Jul 12, 2017

Enhance PeoplePoseEstimation2D #2162

Merged

k-okada mentioned this pull request Apr 8, 2022

PeoplePoseArray visualizer #2673

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jsk_perception] PeoplePoseEstimation2D #2115

[jsk_perception] PeoplePoseEstimation2D #2115

iory commented Jun 16, 2017

wkentaro Jun 17, 2017

iory Jun 18, 2017

wkentaro Jun 17, 2017

iory Jun 18, 2017

wkentaro Jun 17, 2017

iory Jun 18, 2017

wkentaro commented Jun 17, 2017

wkentaro Jun 18, 2017

iory Jun 18, 2017

wkentaro Jun 19, 2017

iory Jun 19, 2017

furushchev left a comment

furushchev Jun 21, 2017

iory Jun 21, 2017

furushchev Jun 21, 2017

furushchev commented Jun 21, 2017

iory commented Jun 21, 2017

furushchev commented Jun 21, 2017

furushchev commented Jun 21, 2017

iory commented Jun 21, 2017

furushchev commented Jun 21, 2017 via email

iory commented Jun 21, 2017

iory commented Jun 21, 2017

k-okada commented Jun 21, 2017 via email

iory commented Jun 21, 2017

furushchev commented Jun 22, 2017

iory commented Jun 23, 2017

wkentaro commented Jun 23, 2017 •

edited

Loading

iory commented Jun 23, 2017

iory commented Jun 23, 2017

furushchev commented Jun 23, 2017

iory commented Jun 23, 2017

		@@ -0,0 +1,2 @@
		string[] limb_names
		geometry_msgs/Point32[] points

[jsk_perception] PeoplePoseEstimation2D #2115

[jsk_perception] PeoplePoseEstimation2D #2115

Conversation

iory commented Jun 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wkentaro commented Jun 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

furushchev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

furushchev commented Jun 21, 2017

iory commented Jun 21, 2017

furushchev commented Jun 21, 2017

furushchev commented Jun 21, 2017

iory commented Jun 21, 2017

furushchev commented Jun 21, 2017 via email

iory commented Jun 21, 2017

iory commented Jun 21, 2017

k-okada commented Jun 21, 2017 via email

iory commented Jun 21, 2017

furushchev commented Jun 22, 2017

iory commented Jun 23, 2017

wkentaro commented Jun 23, 2017 • edited Loading

iory commented Jun 23, 2017

iory commented Jun 23, 2017

furushchev commented Jun 23, 2017

iory commented Jun 23, 2017

wkentaro commented Jun 23, 2017 •

edited

Loading