Release 0.4.0

Release Note

• [A] Add tSNE demo in validation step
• [A] Add PartialFC training module
• [C] Remove playcard data
• [C] Update benchmark rules and model re-training
• [C] Update convert to onnx settings

Evaluation Results

• Comparison Based on the Number of Target Classes

  Name	Num_Classes	TPR@FPR=1e-4	ROC
  lcnet050-f256-r128-ln-arc	16,256	0.615	0.9867
  lcnet050-f256-r128-ln-arc	130,048	0.666	0.9919
  lcnet050-f256-r128-ln-arc	390,144	0.754	0.9951

  • The more classes there are, the better the model performs.

• MarginLoss Comparison

  Name	TPR@FPR=1e-4	ROC
  lcnet050-f256-r128-ln-softmax	0.663	0.9907
  lcnet050-f256-r128-ln-arc	0.754	0.9951
  lcnet050-f256-r128-ln-cos	0.784	0.9968

  • Using CosFace or ArcFace alone, ArcFace performs better.
  • With PartialFC, CosFace performs better.

• BatchNorm vs LayerNorm

  Name	TPR@FPR=1e-4	ROC
  lcnet050-f256-r128-bn-cos	0.721	0.9921
  lcnet050-f256-r128-ln-cos	0.784	0.9968

  • Using LayerNorm yields better results than BatchNorm.

• Pretrain vs From-Scratch

  Name	TPR@FPR=1e-4	ROC
  lcnet050-f256-r128-ln-cos-from-scratch	0.141	0.9273
  lcnet050-f256-r128-ln-cos	0.784	0.9968

  • Using Pretrain is necessary and can save us a lot of time.

• Ways to reduce model size

  Name	TPR@FPR=1e-4	ROC	Size (MB)	FLOPs (G)
  lcnet050-f256-r128-ln-cos	0.784	0.9968	5.54	0.053
  lcnet050-f256-r128-ln-cos-squeeze	0.772	0.9958	2.46	0.053
  lcnet050-f256-r128-ln-cos-gap	0.480	0.9762	2.67	0.053
  lcnet050-f128-r96-ln-cos	0.713	0.9944	2.33	0.029

  • Methods:
    • flatten: Flatten -> Linear (Default)
    • gap: GlobalAveragePooling2d -> Linear
    • squeeze: Conv2d -> Flatten -> Linear
    • Reduce resolution and feature dimensions
  • Use the squeeze method, which reduces the model size by half, although it sacrifices a little performance.
  • Using the gap method, the accuracy is greatly reduced.
  • Reduce the resolution and feature dimensions, and the accuracy will be slightly reduced.

• Increase Backbone

  Name	TPR@FPR=1e-4	ROC
  lcnet050-f256-r128-ln-cos	0.784	0.9968
  efficientnet_b0-f256-r128-ln-cos	0.682	0.9931

  • As the number of parameters increases, the effect decreases. We believe this is related to the data diversity of the training data set. Since our approach does not provide much diversity, increasing the number of parameters does not improve the performance.

Full Changelog: 0.3.0...0.4.0