Extract expression parameters from images

[uniBruce]

Hi, thanks a lot for the amazing work. I am wondering if I can extract expression parameters from images, so that I can get supervision of 3D

[TimoBolkart]

You can either fit the model to an image similar to the 3D landmark fitting example, but optimizing for the difference between predicted 2D landmarks and the projected 3D landmarks (i.e. as described in the FLAME paper), or use a pre-trained regression model like RingNet, which directly outputs FLAME model parameters for an image.

[nyp0x]

You can either fit the model to an image similar to the 3D landmark fitting example, but optimizing for the difference between predicted 2D landmarks and the projected 3D landmarks (i.e. as described in the FLAME paper), or use a pre-trained regression model like RingNet, which directly outputs FLAME model parameters for an image.

It seems RingNet targets another version of FLAME model since it outputs 100 shape parameters and 50 expression parameters (RingNet/config_test.py):

flags.DEFINE_integer('pose_params', 6, 'number of flame pose parameters')
flags.DEFINE_integer('shape_params', 100, 'number of flame shape parameters')
flags.DEFINE_integer('expression_params', 50, 'number of flame expression parameters')

It is possible to optimize TF_FLAME parameters (300/100) given RingNet vertices output, but it doesn't look like the best way to fit TF_FLAME to image.

[TimoBolkart]

It seems RingNet targets another version of FLAME model since it outputs 100 shape parameters and 50 expression parameters (RingNet/config_test.py)

RingNet does not use another version of FLAME, just only a subset of the parameter. The shape parameters RingNet returns are the first 100 shape parameters of FLAME, the expression parameters are the first 50 expression parameters.

Recently, a demo got added to this repo to fit FLAME to sparse 2D keypoints.

It is possible to optimize TF_FLAME parameters (300/100) given RingNet vertices output, but it doesn't look like the best way to fit TF_FLAME to image.

Fitting 2D image keypoints only provides a very sparse signal about the facial shape and expression. Using more parameters for optimization than landmark constraints requires solving an underconstrained optimization problem. The more parameters are optimized for, the more carefully the regularization must be set.

[nyp0x]

All good, this answer explains the mismatch.

[nyp0x]

Wow, the fastest answer ever =).
Thank you!