Feet prediction quality

[kallivad]

Dear @mkocabas Thank you for the great work!
I have watched and analysed a number of indoor videos which illustrate the approach, and it looks like while the reconstruction results overall are impressive,
there is still a number of leg positioning drawbacks, especially noticeable in feet pose detection.

In some frames feet are not detected correctly, on most of incorrectly predicted frames leg toes are raised up. IMHO, it is likely that all net contraction trained with poor feet labeling. Is it correct assumption?

Now I am in the process of understanding what is the root cause of this issue, and what can be done
in order to alleviate feet prediction error.
I see that you predictor has convolutional backbone inherited from the SPIN solution.
https://github.com/nkolot/SPIN
But I haven't figured out on what data and with what labelling it is trained with?
I mean, does it have only one feet joint or several? Have you retrained SPIN on your datasets?
And in case of retraining CNN backbone, is it necessary to retrain the temporal part of VIBE too?
Or perhaps I can leave it untouched for a while?
Thanks a lot in advance, I would greatly appreciate your response.

[ikvision]

Some of the results you show above may be improved by cropping using a larger bounding box
https://github.com/nkolot/SPIN/blob/b95a00a7c0147f2c5bee0874ba0972c6389b6f99/demo.py#L86, but tuning bounding box is a limited solution and re-training is more beneficial.

SPIN is trained with multiple losses, the relevant ones I think are:

• keypoint_loss: Compute 2D reprojection L2 loss on the keypoints
• Keypoint_3d_loss: Compute 3D keypoint L2 loss for the examples that 3D keypoint annotations are available

according to nkolot/SPIN#39 (comment) ground truth keypoints include Ankle, but do not include keypoint on the feet.

OpenPose includes 2 key points on the feet, but openpose_train_weight is set to zero by default.
When trying to train with openpose_train_weight > 0 the model results deteriorated, probably due to nkolot/SPIN#43 (comment)

I think it is a shared gap of SPIN and VIBE, therefore I don't think it is related to the GRU (temporal) part of VIBE.

[ikvision]

Some papers suggest foot velocity loss
eg: https://arxiv.org/abs/1905.01680

[kallivad]

@ikvision Thank you a lot for clarification!

[mkocabas]

Thanks for the discussion @kallivad and @ikvision.

During training of the released pretrained models, we didn't incorporate openpose feet keypoint predictions to the projected 2d keypoint loss. We plan to incorporate those to improve feet accuracy which would be helpful to some extent.

[Tetsujinfr]

Do you have an idea on when you would release the feet enhancement? I think it would help a lot to improve further the overall quality of the poses estimations. Thanks for the great work.

[athn-nik]

That is true. We are trying to find a workaround for this. Either better feet keypoints or a constraint. We don't have a precise estimate, right now.