strange shape question

[flynnamy]

Hi，I use accurate 2D landmark to fit and add the weight to let it close eyes.It shows that it can close eyes but get strange shape. I try to add regularizer of shape and expression, but it does not work.Any suggestions? @TimoBolkart

[TimoBolkart]

Can you share the set of landmarks? Then I can have a look into it.

[flynnamy]

Hi,please try this landmarks file! @TimoBolkart
wink.zip

[flynnamy]

Hi, do you try this landmark file?Any suggestions? @TimoBolkart

[TimoBolkart]

Sorry for the late response. I did play with the landmarks a bit. The reason for the strage shape is that the model priors of shape / expression are too low, so the landmarks are tightly fit, but the reconstructed shape is unrealistic.

One thing to get better eye blinks reconstruction, you can use an additional loss to measure the distance between the upper / lower eyelid landmarks for the projected model landmarks and the target landmarks, and minimizing the difference between them. The loss (i.e. lmk_eye_dist) would be something like that:

'model_lmks_dist = tf.subtract(tf.gather(lmks_proj_2d, [20, 21, 26, 27]), tf.gather(lmks_proj_2d, [24, 23, 30, 29])) target_lmks_dist = target_2d_lmks[[20, 21, 26, 27]] - target_2d_lmks[[24, 23, 30, 29]] lmk_eye_dist = 1.0*tf.reduce_sum(tf.square(tf.subtract(model_lmks_dist, target_lmks_dist))) / (factor ** 2)`

Another problem is that while FLAME is able to blink with the eyelids, but the blink of both eyes is quite correlated in the training data. Therefore, blinking only with one eye is difficult. You could use these additive blendshapes for left eye and right eye with weight [0,1] to close either eye for the FLAME template.