Using data from other 2D pose estimation systems

[c-loob]

Thanks for your interest in our research!

Hi,
First off, thank you very much for sharing this great work! It has been incredibly helpful to have this available.

I have a quick question about OpenPose vs SH. I'm trying to estimate the pose based on the 16 feature point locations. I am using OpenPose for getting the estimation of pose in 2D. The features output by OpenPose can be found here:
https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/media/keypoints_pose.png

I noticed that stacked hourglass method outputs slightly different features and I am looking for ways to correct my data for that. The feature points that are missing/questionable in OpenPose are:

• hip center - not a huge problem, should be possible to just take the average from left/right hip
• spine - slightly more problematic as I don't know what sh means by "spine" and OpenPose does not provide a direct match
• thorax - could perhaps be replaced by base of neck?
• head - could be replaced by nose

My question is, have you tested or have you seen any results from using OpenPose and if you have, how were the results compared to sh after running it through your project(in your opinion, given that the model is trained on OP data)?
Is it even worth correcting my data for the current pre-trained model or should I just go ahead and retrain using OP data/set up SH since the probability of getting any reasonable results otherwise would be very low?

I would love to have a minimal system running asap and am trying to find some ways to cut corners :)

Again, thank you very much for the great work that you have done. I hope the question isn't too much of a waste of time as this is not necessarily what the project was designed to do.

[una-dinosauria]

Hi @c-loob!,

have you tested or have you seen any results from using OpenPose and if you have, how were the results compared to sh after running it through your project(in your opinion, given that the model is trained on OP data)?

No, we haven't. When we were writing this code, the best 2d pose detector out there was SH -- if I remember correctly, the code that eventually became OpenPose was lagging slightly behind SH on MPII. After submission we only worked on fine-tuning, and we have moved to other problems since.

Is it even worth correcting my data for the current pre-trained model or should I just go ahead and retrain using OP data/set up SH since the probability of getting any reasonable results otherwise would be very low?

I have never tried this, so I don't know for sure, but I would assume that our network has learned to exploit particular biases in the errors of SH, and it will likely be thrown off by the output of another detector.

I would love to have a minimal system running asap and am trying to find some ways to cut corners :)

You will probably get best results if you recompute 2d detections with OpenPose and re-train from there. It shouldn't take more than 1 day to write the script to get detections, and maybe a day or 2 for a good GPU to go do all the detections, and then you can train our network super fast! :)

Good luck on your research!

[c-loob]

Thank you very much for your answer :)

[larrypm]

Hi c-loob,

Good question. I have been thinking the same thing. Did you attempt to do this? If so, did you get the desired outcome?

Thanks in advance

[faj2007]

I tried one day on that, the open pose 2D detection seems doesn't improve the result. I gave up on it. (Missing key points is big problem. Also the different kind of key points calculation)

[una-dinosauria]

I'm too busy finishing my thesis and moving out to look at this right now, but if you can read Japanese here's a tutorial on how to use this code with OpenPose: http://akasuku.blog.jp/archives/73745862.html

[faj2007]

Thanks for the reference, unfortunately I don't understand Japanese(although close to my language). I guess my Openpose implementation is right. If the the person is faraway from the camera, the 2D estimation is not that good and will have a bad 3d result.

[larrypm]

Hi fa2007,

Do you mind explaining what way you tired to use open pose?

Did you (as suggested by una-dinosauria) get 2-D detection's from Open Pose, and then retrain the 2D-to-3D model?

I am very interested to hear what approach you took,
Thanks

[faj2007]

I only convert the openpose detection to SH style and then get the 3d prediction by the networks. I didn't retrain the network. I guess you need to do some modification of the network and keypoints if you want to retrain the network by openpose detection. If you manage to retrain the networks finally, tell us you result if possible.

[larrypm]

Thanks faj2007. I guess, as una-dinosauria says, you should check that japanese blog. The appear to get pretty good results in the demo video. You can compare how they convert the joints to your implementation.

I will report back if I get a chance to re-train the network.

Cheers,
Larry