Questions on optimization/tailor module

[pablovela5620]

Hiya, I had a few questions about the tailor module

Just as a high-level overview (just to make sure I correctly understand the approach). It seems that you want to refine the predicted pose parameters from the inverse kinematics network with the insight that the 2d keypoints will generally have higher accuracy/ reliability allowing for not only a more accurate prediction but also helping with bad initializations and occlusion problems in typical optimization-based techniques face (along with fewer iterations due to a good initial condition)

1. Did you experiment with different optimizers in jax other than Adam? Such as the Levenberg-Marquat solver (not sure if this is implemented in jax) used in papers like MegaTrack/RGB2Hands/Online Optical Marker-based Hand Tracking with Deep Labels
2. For the in-the-wild examples, I've noticed certain poses lead to failure cases for estimating the scale camera parameter, is this something you've noticed as well? usually when the hand is horizontal rather than vertical

I enjoyed reading your paper greatly! Thank you so much for the contribution and I look forward to your response

[LyuJ1998]

Hi, I'm glad you discuss these questions with me.

1. I have experimented with different optimizers in JAX, but I found Adam is the best one. As for the Levenberg-Marquat solver, I tried before, but the performance is not good, I'm not sure why. I may check it in the future.
2. Recovery hand mesh without camera parameters can be unstable. Can you provide me some of your failure cases? I will try to find out the problems.

Thank you for this issue.

[pablovela5620]

Sorry for the late response, so I think the issue has to do with calculating the camera scale. I think it may have to do with the fact that I included a hand detector to automatically crop out a 256x256 hand and then I don't perform the refinement until AFTER doing an affine transform from crop back to the full image space.

In the paper you mention a proof for calculating s for the in the wild case in the supplementary materials, but I can't seem to find them. Could you potentially provide me that link to the supp materials? So I can see what I may be doing wrong on my side.

Thanks!

[LyuJ1998]

I will upload the sipplementary materials in this repo later. But actually the proof for calculating s is quite naive. I wonder can you show me some failure cases, or, if possible, show me your code. I would like to find out the problem.

[pablovela5620]

Sure here's a gif showing the failure case. I'm not using the original iknet/keypoint estimator, rather just the optimization module provided in the codebase. So I've added a left/right hand detector, 2.5 keypoint estimator + pose/shape regressor that are my own.

[LyuJ1998]

So the optimization module seems decrease the performance?

As for the proof for calculating s

[pablovela5620]

Yea thats what seems to be happening, BUT I think this may also be a problem of bad initialization. I'm noticing sometimes I have bad mano pose predictions from my network which could lead to a bad convergence on the optimization module. I'll do more digging. I appreciate the help and you posting the proof!