Perception and Semantic Aware Regularization for Sequential Confidence Calibration

This is the official PyTorch implementation for the paper: "Perception and Semantic Aware Regularization for Sequential Confidence Calibration". For more details, please refer to our paper.

Runtime Environment

• This work was tested with PyTorch 1.9.0, CUDA 11.1, python 3.8
  
• Requirements : fastai==1.0.60 lmdb pillow opencv-python tensorboardX editdistance nltk natsort numpy

Datasets

We use lmdb dataset for training and evaluation. The training datasets can be downloaded in clovaai and the ensemble of evalutaion datasets can be downloaded here.

• Training datasets
  • MJSynth (MJ)
  • SynthText (ST)
• Evaluation datasets
  • Regular datasets
    • IIIT5K (IIIT)
    • Street View Text (SVT)
    • ICDAR2013 (IC13)
    • ICDAR2003 (IC03)
  • Irregular dataset
    • ICDAR2015 (IC15)
    • Street View Text Perspective (SVTP)
    • CUTE80 (CUTE)
• Tree structure of data_lmdb_release directory

  data_lmdb_release
  ├── charset_36.txt
  ├── charset_94.txt
  ├── test
  └── training
      ├── MJ
      │   ├── MJ_test
      │   ├── MJ_train
      │   └── MJ_valid
      └── ST
  

Calibration

1. Get Correlated Sequence

Perception Similarity Sequence

bash get_perc_sim_seq.sh

Semantic Correlation Sequence

CUDA_VISIBLE_DEVICES=0 python abinet/get_sem_cor_seq.py --checkpoint /path/to/checkpoint

• --checkpoint: path to pretrained model of BCN in ABINet.

2. Calibrate Model With PSSR

CUDA_VISIBLE_DEVICES=0 python train.py \
--train_data data_lmdb_release/training --valid_data data_lmdb_release/test \
--select_data MJ-ST --batch_ratio 0.5-0.5 --lr 0.1 \
--Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn \
--saved_model /path/to/checkpoint --calibrator PSSR --alpha 0.05 --with_vis

• --saved_model: path to model to be calibrated.
• --calibrator: select calibration method.
• --alpha: weight factor for balancing in the Eq. (2) in our paper.
  Note: Detailed argument settings can be found in clovaai.

Evaluation

CUDA_VISIBLE_DEVICES=0 python test.py \
--eval_data data_lmdb_release/test \
--Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn \
--saved_model pretrained_models/TPS-ResNet-BiLSTM-Attn.pth

• --saved_model: path to calibrated model.

Calibrated Models Download

Download calibrated model by PSSR from this link.

Acknowledgements

This implementation has been based on clovaai and ABINet.

Citation

If you find our work useful in your research, please cite the following:

@inproceedings{peng2023perception,
  title={Perception and Semantic Aware Regularization for Sequential Confidence Calibration},
  author={Peng, Zhenghua and Luo, Yu and Chen, Tianshui and Xu, Keke and Huang, Shuangping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10658--10668},
  year={2023}
}