Yu Yang*, Xiaotian Cheng*, Chang Liu, Hakan Bilen, Xiangyang Ji. Distilling Representations from GAN Generator via Squeeze and Span. In NeurIPS 2022. [pdf, bibtex]
Put data or create a soft link to the dataset root directory in ./data/
. For example,
data/
├── CIFAR10
│ ├── cifar-10-batches-py
│ └── cifar-10-python.tar.gz
├── CIFAR100
│ ├── cifar-100-python
│ └── cifar-100-python.tar.gz
├── STL10
│ ├── stl10_binary
│ └── stl10_binary.tar.gz
Please download dataset from CIFAR, STL10.
Please download pre-trained GAN checkpoints from
File | Url |
---|---|
cifar10u-cifar-ada-best-fid.pkl | |
cifar100u-cifar-best-fid4.13.pkl | |
stl10u-my128-best-fid20.86.pkl |
checkpoints/
├── cifar100u-cifar-best-fid4.13.pkl
├── cifar10u-cifar-ada-best-fid.pkl
└── stl10u-my128-best-fid20.86.pkl
Knowledge Source | Transfer Method | Domain | CIFAR10 | CIFAR100 |
---|---|---|---|---|
Discriminator | Direct use (single feature) | Syn. & Real | 63.81 [script] | 30.11 [script] |
Discriminator | Direct use (multi-feature) | Syn. & Real | 77.58 [script] | 51.63 [script] |
Latent variable | Encoding | Syn. | 57.15 [script] | 32.19 [script] |
Latent variable | Encoding | Syn. & Real | 50.27 [script] | 28.43 [script] |
Latent variable | Vanilla distillation (w/ aug) | Syn. | 84.84 [script] | 53.26 [script] |
Latent variable | Squeeze | Syn. | 86.99 [script] | 58.56 [script] |
Latent variable | Squeeze and span | Syn. & Real | 90.95 | 66.17 |
Generator feature | Vanilla distillation (w/ aug) | Syn. | 84.48 [script] | 52.77 script] |
Generator feature | Squeeze | Syn. | 87.67 [script] | 57.35 [script] |
Generator feature | Squeeze and span | Syn. & Real | 92.54 [script] | 67.87 [script] |
Pretrain Data | Methods | CIFAR10 | CIFAR100 | STL10 |
---|---|---|---|---|
Real | SimSiam | 90.94 [script] | 62.44 [script] | 71.30 [script] |
Real | VICReg | 89.20 [script] | 63.31 [script] | 74.43 [script] |
Syn | SimSiam | 85.11 [script] | 47.89 [script] | 73.38 [script] |
Syn | VICReg | 84.68 [script] | 52.84 [script] | 70.80 [script] |
Syn | Squeeze (Ours) | 87.67 [script] | 57.35 [script] | 73.35 [script] |
Real & Syn | SimSiam | 90.88 [script] | 62.68 [script] | 71.70 [script] |
Real & Syn | VICReg | 90.46 [script] | 65.22 [script] | 75.05 [script] |
Real & Syn | Squeeze & Span (Ours) | 92.54 [script] | 67.87 [script] | 76.83 [script] |
Span | Top-1 Acc | ||||||
---|---|---|---|---|---|---|---|
a | 74.20 [script] | ||||||
b | 84.48 [script] | ||||||
c | 10.00 [script] | ||||||
d | 79.10 [script] | ||||||
e | 87.67 [script] | ||||||
f | 92.54 [script] |
CUDA_VISIBLE_DEVICES=0 python plot_umap --output-dir=output/plot_umap --gpath=checkpoints/cifar10u-cifar-ada-best-fid.pkl --cpath=checkpoints/cifar10_wrn.pth
python paper_plots/acc_vs_fid.py
Dataset | Script |
---|---|
CIFAR10 | scripts/cifar10_eval_linear.sh |
CIFAR100 | scripts/cifar100_eval_linear.sh |
STL10 | scripts/stl10_eval_linear.sh |
If you find this repository useful in your research, please consider citing:
@inproceedings{
yang2022distilling,
title={Distilling Representations from {GAN} Generator via Squeeze and Span},
author={Yu Yang* and Xiaotian Cheng* and Chang Liu and Hakan Bilen and Xiangyang Ji},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
year={2022},
url={https://openreview.net/forum?id=_P4JCoz83Mb}
}