Cannot perform evaluation.

[CHELSEA234]

Hi,

I think your work is impressive, but I met some issues while reproducing it.

Now, I just want to use your provided pertained model for learning the code, so I did not download two complete datasets and skipped the training part, went to evaluation directly. Also, I changed some necessary codes so that I can run it on my Mac, which does not have GPU.

Evaluation on the single image: What I got is a series of output number (Figure below), can you tell me which files I can use for qualitative results, 2D image with landmarks (like Figure 8 in the paper).

Evaluate on 300-W or AFLW: I cannot find 300W-EVAL.sh and AFLW_CYCLE_128.FULL-EVAL.sh files in the script folder. Can you tell me where I may make a mistake?

Best,
XG

[D-X-Y]

Hi @CHELSEA234 , sorry for the inconvience. I just updated the scripts at https://github.com/D-X-Y/SAN/blob/master/scripts/300W/300W-EVAL.sh and updated san_eval.py.
Following https://github.com/D-X-Y/SAN#evaluation-on-the-single-image, you can use the save_path arg to save the visualization results.

[CHELSEA234]

Hi @D-X-Y ,

I am impressed by your timely reply and grateful for your help.

As I think I have to download the whole dataset before running Evaluate on 300-W or AFLW (right?), can you tell me what outputs I will get, I think they will be like Table 1 in the paper, right? Will the output format be like answer in issue#15, including results on all subsets in 300W?

BTW, I recommend, in order to save image, you should write image.save( args.save_path + '/results.png’) instead of image.save( args.save_path ), do you think so?

Best,
XG

[D-X-Y]

Yes, you need to download the whole dataset before running evaluation on 300W and AFLW.
You will obtain the following outputs on 300-W:

Evaluation => 3 image datasets and 3 video datasets, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049
  Evaluate => [ 0/ 3]-th image dataset : ./cache_data/lists/300W/Original/300w.test.common.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049/eval-set-00 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/070] Time  3.504 ( 3.504) Need [00:04:01] locations.shape=(8, 69, 2)
  Evaluation: [069/070] Time  0.112 ( 0.332) Need [00:00:00] locations.shape=(2, 69, 2)
Compute NME and AUC for 554 images with 68 points :: [(nms): mean=3.32, std=1.08], auc@0.07=52.74, auc@0.08-58.57, acc@0.07=98.92, acc@0.08=99.64
  Evaluate => [ 1/ 3]-th image dataset : ./cache_data/lists/300W/Original/300w.test.challenge.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049/eval-set-01 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/017] Time  1.653 ( 1.653) Need [00:00:26] locations.shape=(8, 69, 2)
  Evaluation: [016/017] Time  0.285 ( 0.368) Need [00:00:00] locations.shape=(7, 69, 2)
Compute NME and AUC for 135 images with 68 points :: [(nms): mean=6.58, std=3.26], auc@0.07=19.53, auc@0.08-26.59, acc@0.07=71.11, acc@0.08=81.48
  Evaluate => [ 2/ 3]-th image dataset : ./cache_data/lists/300W/Original/300w.test.full.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049/eval-set-02 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/087] Time  3.177 ( 3.177) Need [00:04:33] locations.shape=(8, 69, 2)
  Evaluation: [086/087] Time  0.092 ( 0.320) Need [00:00:00] locations.shape=(1, 69, 2)
Compute NME and AUC for 689 images with 68 points :: [(nms): mean=3.96, std=2.17], auc@0.07=46.24, auc@0.08-52.31, acc@0.07=93.47, acc@0.08=96.08
====>> Evaluate all image datasets done

In addition, on AFLW:

Train ITN-CPM using LR=0.000050, Decay=0.000500
=> loading checkpoint './snapshots/SAN_AFLW_FULL_GTB_itn_cpm_3_50_sigma4_128x128x8/checkpoint_49.pth.tar' start
Evaluation => 1 image datasets and 1 video datasets, save into ./snapshots/SAN_AFLW_FULL_GTB_itn_cpm_3_50_sigma4_128x128x8-EVAL/epoch-once
  Evaluate => [ 0/ 1]-th image dataset : ./cache_data/lists/AFLW/Original/test.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_AFLW_FULL_GTB_itn_cpm_3_50_sigma4_128x128x8-EVAL/epoch-once/eval-set-00 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/549] Time  3.798 ( 3.798) Need [00:34:41] locations.shape=(8, 20, 2)
  Evaluation: [100/549] Time  0.092 ( 0.126) Need [00:00:56] locations.shape=(8, 20, 2)
  Evaluation: [200/549] Time  0.087 ( 0.108) Need [00:00:37] locations.shape=(8, 20, 2)
  Evaluation: [300/549] Time  0.094 ( 0.102) Need [00:00:25] locations.shape=(8, 20, 2)
  Evaluation: [400/549] Time  0.086 ( 0.099) Need [00:00:14] locations.shape=(8, 20, 2)
  Evaluation: [500/549] Time  0.086 ( 0.097) Need [00:00:04] locations.shape=(8, 20, 2)
  Evaluation: [548/549] Time  0.087 ( 0.096) Need [00:00:00] locations.shape=(2, 20, 2)
Compute NME and AUC for 4386 images with 19 points :: [(nms): mean=1.72, std=2.25], auc@0.07=76.93, auc@0.08-79.73, acc@0.07=99.22, acc@0.08=99.36
====>> Evaluate all image datasets done

I prefer "image.save( args.save_path )" :), because I like to controll the name of the saved file.

[CHELSEA234]

@D-X-Y How can I get results like that of issue#15, including the performance of all subset performance. Do I need to write code on my own, or you have execution files for this?

Thanks for your patience.

[D-X-Y]

Using bash scripts/300W/300W-EVAL.sh 0,1, you will see

Evaluation => 3 image datasets and 3 video datasets, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049
  Evaluate => [ 0/ 3]-th image dataset : ./cache_data/lists/300W/Original/300w.test.common.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049/eval-set-00 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/070] Time  3.504 ( 3.504) Need [00:04:01] locations.shape=(8, 69, 2)
  Evaluation: [069/070] Time  0.112 ( 0.332) Need [00:00:00] locations.shape=(2, 69, 2)
Compute NME and AUC for 554 images with 68 points :: [(nms): mean=3.32, std=1.08], auc@0.07=52.74, auc@0.08-58.57, acc@0.07=98.92, acc@0.08=99.64
  Evaluate => [ 1/ 3]-th image dataset : ./cache_data/lists/300W/Original/300w.test.challenge.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049/eval-set-01 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/017] Time  1.653 ( 1.653) Need [00:00:26] locations.shape=(8, 69, 2)
  Evaluation: [016/017] Time  0.285 ( 0.368) Need [00:00:00] locations.shape=(7, 69, 2)
Compute NME and AUC for 135 images with 68 points :: [(nms): mean=6.58, std=3.26], auc@0.07=19.53, auc@0.08-26.59, acc@0.07=71.11, acc@0.08=81.48
  Evaluate => [ 2/ 3]-th image dataset : ./cache_data/lists/300W/Original/300w.test.full.GTB
 ==>start evaluation image dataset, save into ./snapshots/SAN_300W_GTB_itn_cpm_3_50_sigma4_128x128x8/epoch-049/eval-set-02 with the error bar : None, using the last stage, DEBUG SAVE DIR : None
  Evaluation: [000/087] Time  3.177 ( 3.177) Need [00:04:33] locations.shape=(8, 69, 2)
  Evaluation: [086/087] Time  0.092 ( 0.320) Need [00:00:00] locations.shape=(1, 69, 2)
Compute NME and AUC for 689 images with 68 points :: [(nms): mean=3.96, std=2.17], auc@0.07=46.24, auc@0.08-52.31, acc@0.07=93.47, acc@0.08=96.08
====>> Evaluate all image datasets done

The [ 0/ 3]-th image dataset is the common set of 300-W test set. [1/3]-th is the challenge set, [2/3]-th is the full set.

[CHELSEA234]

Hi @D-X-Y 👏👏:

I am new to this topic, maybe some questions sound silly, thanks for your patience and guidance.👍👍👍👍👍👍👍

I have seen your answer in #issue 14. How did you get the bounding box in link? Is this the “GT”, tight bounding box? It looks like the predefined bounding box imported from mat data.

Now I need to execute your code on my own image, how should I locate the tight bounding box, can you share the instruction link? or can you tell me how did you do on the ../cache_data/cache/test_1.png, that looks pretty good.

Best

[D-X-Y]

The bounding box for 300W is from the official website : https://ibug.doc.ic.ac.uk/resources/300-W/ (search bounding box initialisations).

You can use some face detectors to obtain the face bounding box and then use our algorithms to obtain the landmarks. For facial landmark detection, we usually assume the bounding box is already obtained.