Can not meet the target(1.27 | 1.59 | 1.31) in NoW benchmark using the given pretrained checkpoint

[zhoupeizhu]

I checked the performance,but do it can not meet the target(1.27 | 1.59 | 1.31) in NoW benchmark using the given pretrained checkpoint. Thanks very much.

[zhoupeizhu]

This is the NoW benchmark(https://now.is.tue.mpg.de/nonmetricalevaluation.html).

[choyingw]

The released checkpoints are for the best performance on aflw2000. The checkpoint we submitted to NoW benchmark is not released.

…

On Thursday, November 3, 2022, ZPzhu ***@***.***> wrote: This is the NoW benchmark(https://now.is.tue.mpg.de/nonmetricalevaluation. html <https://urldefense.com/v3/__https://now.is.tue.mpg.de/nonmetricalevaluation.html__;!!LIr3w8kk_Xxm!oWozpMUVFiRGx5cfiWEhtS6hRpy3Q_SKDw8_lXH0sX5IWp8n4XNq6lcOw5s3uIWzSKgC1Iosk7TnRC1Sedw0s4Qt$> ). — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/choyingw/SynergyNet/issues/24*issuecomment-1302883014__;Iw!!LIr3w8kk_Xxm!oWozpMUVFiRGx5cfiWEhtS6hRpy3Q_SKDw8_lXH0sX5IWp8n4XNq6lcOw5s3uIWzSKgC1Iosk7TnRC1SeZ5JJpHc$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKMLAJM5BGSHWUCQTFWHLCLWGRZLTANCNFSM6AAAAAARWX6K6E__;!!LIr3w8kk_Xxm!oWozpMUVFiRGx5cfiWEhtS6hRpy3Q_SKDw8_lXH0sX5IWp8n4XNq6lcOw5s3uIWzSKgC1Iosk7TnRC1SeSPKUSFL$> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Ph.D. candidate of Computer Science Department University of Southern California

[zhoupeizhu]

The released checkpoints are for the best performance on aflw2000. The checkpoint we submitted to NoW benchmark is not released.
…
On Thursday, November 3, 2022, ZPzhu @.> wrote: This is the NoW benchmark(https://now.is.tue.mpg.de/nonmetricalevaluation. html <https://urldefense.com/v3/https://now.is.tue.mpg.de/nonmetricalevaluation.html;!!LIr3w8kk_Xxm!oWozpMUVFiRGx5cfiWEhtS6hRpy3Q_SKDw8_lXH0sX5IWp8n4XNq6lcOw5s3uIWzSKgC1Iosk7TnRC1Sedw0s4Qt$> ). — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/https://github.com/choyingw/SynergyNet/issues/24*issuecomment-1302883014;Iw!!LIr3w8kk_Xxm!oWozpMUVFiRGx5cfiWEhtS6hRpy3Q_SKDw8_lXH0sX5IWp8n4XNq6lcOw5s3uIWzSKgC1Iosk7TnRC1SeZ5JJpHc$>, or unsubscribe <https://urldefense.com/v3/https://github.com/notifications/unsubscribe-auth/AKMLAJM5BGSHWUCQTFWHLCLWGRZLTANCNFSM6AAAAAARWX6K6E;!!LIr3w8kk_Xxm!oWozpMUVFiRGx5cfiWEhtS6hRpy3Q_SKDw8_lXH0sX5IWp8n4XNq6lcOw5s3uIWzSKgC1Iosk7TnRC1SeSPKUSFL$> . You are receiving this because you are subscribed to this thread.Message ID: @.>
-- Ph.D. candidate of Computer Science Department University of Southern California

Thank you for your reply. Could you share me your code generating files of NoW benchmark. The accuracy on NoW benchmark trained with your published code is far from the target(1.27 | 1.59 | 1.31). This is my mail(zpz001@buaa.edu.cn).

[choyingw]

I share the prediction (mesh and sparse landmark) we submitted to now_challenge here if you would like to compare with each face sample.

We do the following steps to evaluate on NoW benchmark

For NoW challenge since it evaluates holistic faces, at the training time we further resample points on multiple face areas (such as eyes, forehead, cheek and sides) as groundtruth for supervised learning. This will not necessarily have better sparse landmark registration performance but would improve performance for overall face deformation. We didn't tune the weight and hyperparameter much so potentially the performance could be better.

At the inference time, crop the face in the now benchmark to a square (which is thew aspect ratio we use) using the bbox they provided. If the cropping margin matches the margins in 300W_LP, then potentially it can leave out the cropping misalignment factor. Then just use the test API we provided to estimate 3D face and sparse landmarks

The quantitative results shouldn’t be much worse since face difference is subtle. Even if using the mean face shape as prediction this can probably be a not very bad baseline. If you get a much worse number you can look into your mesh and see if the mesh is reasonable in terms of shape or scale (the monocular face output is in pixel so you need to scale to match the metrics)

[zhoupeizhu]

Hi, I am sorry. The peoblem has not be settled. The result(median: 1.323301, mean: 1.665297, std: 1.416094
) is tested by your submission, I only changed the path of NoW evaluation code.