Are you going to release the trained SAN model on 300w?

[bobetocalo]

Dear Xuanyi Dong,

First of all I would like to congratulate you for your excellent work. I'm a PhD student at Spain. My research is focused on face alignment. I have used your https://github.com/D-X-Y/SAN code and I would like to ask some questions:

• Are the 300W and AFLW best trained models publicly available? I have downloaded the SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8 that you provide but the reported results are far away from the reported in the paper https://arxiv.org/abs/1803.04108.

 > Full:
NME: 6.053968616522241
AUC: 32.29837440048558
FR: 15.965166908563134
 > Helen:
NME: 4.968793431617647
AUC: 39.18139403496543
FR: 5.454545454545457
 > LFPW:
NME: 5.1568256324448525
AUC: 36.52686151341836
FR: 6.696428571428569
 > Common:
NME: 5.044820891879911
AUC: 38.10815694365434
FR: 5.956678700361007
 > iBUG:
NME: 10.195211871721133
AUC: 8.350256928175838
FR: 57.03703703703704

We would like to repeat the 3.98 NME obtained in the Full set on 300W. I look forward to your response.

Best regards,
Roberto Valle

[D-X-Y]

Hi Roberto, thanks for trying our codes. Would you mind to let me know how did you get these numbers? What normalization distance did you use? The released model is trained by using https://github.com/D-X-Y/SAN#300-w and should obtain a similar performance compared to results in the paper.

[bobetocalo]

Hi,

I have processed 300W using the trained model that you provide called SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8

I use inter-pupil distance normalization as in the literature. As a result, I attach some examples of images that I have obtained with your model. Are these images correct? (NME is showed at the bottom left of each image).

Are these images similar to the ones that you have obtained in your paper? It is impossible to obtain a 3.98 NME in the Full data set with the model that I am using (https://github.com/D-X-Y/SAN#evaluation-on-the-single-image).

I look forward to your response.

Best regards,
Roberto Valle

[D-X-Y]

Hi, I just updated the readme, you can follow https://github.com/D-X-Y/SAN#evaluate-on-300-w to evaluate the released model on 300-W. I think there may be two reasons that cause the worse results from your evaluation:

1. We use inter-ocular distance to normalize, which is larger than inter-pupil distance and thus can obtain a small number (as mentioned in the second paragraph in Sec. 4.2 and in https://github.com/D-X-Y/SAN#normalization).
2. The pre-trained model uses the ground truth face bounding box from the official 300-W website. If you use a different type of "face bounding boxes" compared to the one used in the training procedure, you will get worse performance.

Note about the normalization distance: We follow "A deep regression architecture with two-stage re-initialization for high performance facial landmark detection, CVPR 2017" and "300 faces in-the-wild challenge: The first facial landmark localization challenge, ICCV-W 2013"

[bobetocalo]

I am using the ground truth face bounding box from the 300W annotations.

I have changed the normalization measure to corners distance. The reported results are not the same that you report in the paper. However, they are closer to the 3.98 mentioned before.

 > Total:
NME: 4.308152161611457
AUC: 48.61202523909483
FR: 5.224963715529752
 > Helen:
NME: 3.5710834782540104
AUC: 55.42145791970029
FR: 0.9090909090909038
 > LFPW:
NME: 3.7346050901785715
AUC: 53.46455804667038
FR: 0.8928571428571397
 > Common:
NME: 3.6372005198985984
AUC: 54.62610391820355
FR: 0.9025270758122761
 > iBUG:
NME: 7.061538898714597
AUC: 23.855890372885476
FR: 22.962962962962962

However, I encourage you to modify Table 1 in your paper because literature (SDM, ESR, LBF, CFSS) does not use corners normalization either.

Finally, could you provide some example of images of 300W with your prediction and the NME obtained?

On the other hand, are you going to release the trained SAN model on AFLW?

[D-X-Y]

If you use bash scripts/300W/300W-EVAL.sh 0, you will the NME results on the common set, challenge set (ibug) and full set, i.e., 3.37, 6.82, and 4.04. These results are very similar to the one reported in the paper. I guass the difference between your results and my results might be caused by some pre-processing procedure??

Thanks for your suggestion about modifing Table 1. Before we submit our paper, we didn't notice that SDM, ESR, LBF, CFSS are using inter-pupil, but simply copying the numbers from "A deep regression architecture with two-stage re-initialization for high performance facial landmark detection". That is our mistake. After several months of the CVRP camera ready, we noticed this mistake but can not change that version. I have updated this information in our README and will clarify it in our following papers.

You can refer Figure 8 in the paper for examples. I'm reaching some deadlines, and can not provide NME for specific examples right now.

For the trained SAN model on AFLW, if you want to reproduce the results, you can run commands following https://github.com/D-X-Y/SAN#aflw to obtain. I cannot find the trained model, but I just re-run the codes and should obtain models in several hours. After the training procedure finished, I will share a google driver link.