Skip to content

Simon4Yan/Accuracy-estimation-with-self-supervision

Repository files navigation

What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

[Paper] [ICML'21 Project]

PyTorch Implementation

This repository contains:

  • the PyTorch implementation of AutoEavl.
  • the example on CIFAR-10 setup (use imgaug)
  • linear regression

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

  • Linux (tested on Ubuntu 16.04LTS)
  • NVIDIA GPU + CUDA CuDNN (tested on GTX 2080 Ti)
  • CIFAR-10 (download and unzip to PROJECT_DIR/data/)
  • CIFAR10.1 (download and unzip to PROJECT_DIR/data/CIFAR-10.1)
  • Please use PyTorch1.5 to avoid compilation errors (other versions should be good)
  • You might need to change the file paths, and please be sure you change the corresponding paths in the codes as well

Getting started

  1. Install dependencies
    # Imgaug (or see https://imgaug.readthedocs.io/en/latest/source/installation.html)
    conda config --add channels conda-forge
    conda install imgaug
  2. Creat synthetic sets
    # By default it creates 500 synthetic sets
    python generate_synthetic_sets.py
  3. Learn classifier on CIFAR-10 (DenseNet-10-12)
    # Save as "PROJECT_DIR/DenseNet-40-12-ss/checkpoint.pth.tar"
    # Modified based on the wonderful github of https://github.com/andreasveit/densenet-pytorch
    python train.py --layers 40 --growth 12 --no-bottleneck --reduce 1.0
  4. Test classifier on synthetic sets
    # 1) Get "PROJECT_DIR/accuracy_cls_dense_aug.npy" file
    # 2) Get "PROJECT_DIR/accuracy_ss_dense_aug.npy" file
    # 3) You will see Rank correlation and Pearsons correlation
    # 4) The absolute error of linear regression is also shown
    python test_many.py --layers 40 --growth 12 --no-bottleneck --reduce 1.0
  5. Correlation study
    # You will see correlation.pdf;
    python analyze_correlation.py
        

Citation

If you use the code in your research, please cite:

    @inproceedings{Deng:ICML2021,
      author    = {Weijian Deng and
                   Stephen Gould and
                   Liang Zheng},
      title     = {What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?},
      booktitle = {ICML},
      year      = {2021}
    }

License

MIT

About

[ICML'21] Estimate the accuracy of the classifier in various environments through self-supervision

Topics

Resources

License

MIT, BSD-3-Clause licenses found

Licenses found

MIT
LICENSE-A
BSD-3-Clause
LICENSE-B

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages