How to improve shape optimization

[ujjawalcse]

Hey @csbhr ,
Thanks a lot for releasing the code. I tried the given run_rgb_fitting.sh and it's working fine.
But output face shape is not matching with the input as it's giving round faces in most of the cases.

Any idea how to improve the shape optimization part running in step 3 of ours_fit_model.py module?

Reference input,

Reference output,

[csbhr]

From the pre-processing results in the figure below, the face parsing does not exclude the hand area. The fitting process would be affected by the hand texture. More accurate face parsing results might helpful.

[ujjawalcse]

@csbhr , This is a case of occlusion. But I have tested with other clear images too.
Normally, It's coming more round like 3D shapes.
Check this reference input and corresponding output,

[csbhr]

From the 68 landmarks in the image above (gt landmarks in blue and fitted landmarks in red), it can be seen that the landmarks at the bottom of the face are not well fitted. This may be caused by the weight used in landmark loss (see the image below, or the source codes).

On Line79-Line81, we weighted the landmarks of nose and mouth more heavily.
Perhaps, setting the weight of all landmarks to 1 is helpful for fitting the bottom contour of the face.

[ujjawalcse]

@csbhr Tried what you suggested and now the outer shapes are better. But on the cost of a slightly less optimised mouth and nose area. We can't even observe those if we see them separately. We can notice those gaps after overlapping the previous result with the new result.
For now, I tried with assigning all landmarks weights as 1.
I'll try with lesser weight for mouth and nose instead of 20. It may be a better trade-off.
Thanks @csbhr.

[csbhr]

Thank you for sharing your experiments. Hope our codes are useful to you.

[pyh007]

如何用更轻的口鼻