Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels.

This is the official repository for the paper Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels. (Accepted by AAAI 2021).

@inproceedings{chen2021robustness,
  title={Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels.},
  author={Chen, Pengfei and Ye, Junjie and Chen, Guangyong and Zhao, Jingwei and Heng, Pheng-Ann},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}

Under diagonally-dominant class-conditional label noise, the main conclusions are as follows.

• A classifier maximizing its accuracy on the noisy distribution is guaranteed to maximize the accuracy on clean distribution.
• We can obtain an optimal classifier by maximizing training accuracy on sufficiently many noisy samples.
• A noisy validation set is reliable.

Regarding the the critical demand of model selection in scenarios like hyperparameter-tuning and early stopping, we theoretically prove that a noisy validation set is reliable. We empritically verify the utility of a noisy validation set by showing the impressive performance of a very simple method Noisy best Teacher and Student (NTS).

Requirements

• Python 3.6+
• PyTorch 1.2+
• torchvision 0.4+
• pillow 5.0+
• numpy 1.17+

Noisy best Teacher and Student (NTS)

File	Usage
hyperparameter.txt	containing detailed hyperparameters
command.txt	containing commands for running the training
train_cifar10_ce.py	training on CIFAR-10 with normal cross-entropy (NT and NS)
train_cifar10_ct.py	training on CIFAR-10 with Co-teaching (NT and NS)
train_cifar10_dmi.py	training on CIFAR-10 with DMI (NT and NS)
train_cifar10_gce.py	training on CIFAR-10 with GCE (NT and NS)
train_cifar100_ce.py	training on CIFAR-10 with normal cross-entropy (NT and NS)
train_cifar100_ct.py	training on CIFAR-100 with Co-teaching (NT and NS)
train_cifar100_dmi.py	training on CIFAR-100 with DMI (NT and NS)
train_cifar100_gce.py	training on CIFAR-100 with GCE (NT and NS)
train_clothing1m_ce.py	training on Clothing1M with normal cross-entropy (NS)
train_clothing1m_dividemix.py	training on Clothing1M with DivideMix (NT)

Example commands

Please refer to hyperparameter.txt and command.txt for detailed hyperparameters and commands.

CIFAR: training teacher and get NT

**CE CIFAR-10 Teacher**
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.2 --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.3 --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.4 --save_model

**CE CIFAR-100 Teacher**
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.2 --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.3 --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.4 --save_model

**GCE CIFAR-10 Teacher**
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.2 --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.3 --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.4 --save_model

**GCE CIFAR-100 Teacher**
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.2 --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.4 --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.6 --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.2 --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.3 --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.4 --save_model

**Co-T CIFAR-10 Teacher**
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.2 --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.4 --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.6 --e_warm 60 --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.2 --tau 0.1 --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.3 --tau 0.15 --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.4 --tau 0.2 --e_warm 60 --save_model

**Co-T CIFAR-100 Teacher**
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.2 --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.4 --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.6 --e_warm 60 --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.2 --tau 0.2 --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.3 --tau 0.3 --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.4 --tau 0.4 --e_warm 60 --save_model

**DMI CIFAR-10 teacher** (DMI requires a pretrained model, e.g., using CE, to initialize)
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.2 --init_path cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.3 --init_path cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.4 --init_path cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model

**DMI CIFAR-100 Teacher**
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.2 --init_path cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.3 --init_path cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.4 --init_path cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --save_model

CIFAR: training student and get NS

**CE CIFAR-10 Student**
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.2 --teacher_path cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.3 --teacher_path cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ce.py --noise_pattern asym --noise_rate 0.4 --teacher_path cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model

**CE CIFAR-100 Student**
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.2 --teacher_path cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.3 --teacher_path cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ce.py --noise_pattern pair --noise_rate 0.4 --teacher_path cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --save_model

**GCE CIFAR-10 Student**
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path gce_cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path gce_cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path gce_cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.2 --teacher_path gce_cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.3 --teacher_path gce_cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_gce.py --noise_pattern asym --noise_rate 0.4 --teacher_path gce_cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model

**GCE CIFAR-100 Student**
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.2 --teacher_path gce_cifar100_uniform0.2_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.4 --teacher_path gce_cifar100_uniform0.4_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern uniform --noise_rate 0.6 --teacher_path gce_cifar100_uniform0.6_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.2 --teacher_path gce_cifar100_pair0.2_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.3 --teacher_path gce_cifar100_pair0.3_dp0.0_augstandard_seed0_best.pth --save_model
python train_cifar100_gce.py --noise_pattern pair --noise_rate 0.4 --teacher_path gce_cifar100_pair0.4_dp0.0_augstandard_seed0_best.pth --save_model

**Co-T CIFAR-10 Student**
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.05 --teacher_path ct_cifar10_uniform0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.1 --teacher_path ct_cifar10_uniform0.4_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.15 --e_warm 60 --teacher_path ct_cifar10_uniform0.6_warm60_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.2 --tau 0.03 --teacher_path ct_cifar10_asym0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.3 --tau 0.04 --teacher_path ct_cifar10_asym0.3_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_ct.py --noise_pattern asym --noise_rate 0.4 --tau 0.05 --e_warm 60 --teacher_path ct_cifar10_asym0.4_warm60_dp0.2_augstrong_seed0_best.pth --save_model

**Co-T CIFAR-100 Student**
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.2 --tau 0.1 --teacher_path ct_cifar100_uniform0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.4 --tau 0.2 --teacher_path ct_cifar100_uniform0.4_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern uniform --noise_rate 0.6 --tau 0.3 --e_warm 60 --teacher_path ct_cifar100_uniform0.6_warm60_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.2 --tau 0.1 --teacher_path ct_cifar100_pair0.2_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.3 --tau 0.2 --teacher_path ct_cifar100_pair0.3_warm0_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_ct.py --noise_pattern pair --noise_rate 0.4 --tau 0.3 --e_warm 60 --teacher_path ct_cifar100_pair0.4_warm60_dp0.2_augstrong_seed0_best.pth --save_model      

**DMI CIFAR-10 Student**
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.2 --init_path cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_asym0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.3 --init_path cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_asym0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar10_dmi.py --noise_pattern asym --noise_rate 0.4 --init_path cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar10_asym0.4_dp0.2_augstrong_seed0_best.pth --save_model

**DMI CIFAR-100 Student**
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.2 --init_path cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_uniform0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.4 --init_path cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_uniform0.4_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern uniform --noise_rate 0.6 --init_path cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_uniform0.6_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.2 --init_path cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_pair0.2_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.3 --init_path cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_pair0.3_dp0.2_augstrong_seed0_best.pth --save_model
python train_cifar100_dmi.py --noise_pattern pair --noise_rate 0.4 --init_path cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --teacher_path dmi_cifar100_pair0.4_dp0.2_augstrong_seed0_best.pth --save_model

Clothing1M: training teacher and student

**Teacher and Student: Clothing1M**
python train_clothing1m_dividemix.py --root data/Clothing1M/
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path dividemix_net1.pth
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path dividemix_net2.pth

**Teacher and Student: Clothing1M (Noisy Validation)**
python train_clothing1m_dividemix.py --root data/Clothing1M/ --use_noisy_val
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path nv_dividemix_net1.pth --use_noisy_val
python train_clothing1m_ce.py --root data/Clothing1M/ --teacher_path nv_dividemix_net2.pth --use_noisy_val