CurBench: A Curriculum Learning Benchmark

A benchmark for Curriculum Learning.

The code of ICML 2024 paper CurBench: A Curriculum Learning Benchmark.

The paper can be downloaded from the official website or in the docs directory.

Environment

1. python >= 3.7

   https://www.python.org/downloads/

2. pytorch >= 1.12

   https://pytorch.org/

3. torch_geometric

   https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html

4. other requirements

pip install -r requirements.txt 

Dataset

Vision

CIFAR-10 and CIFAR-100 will be downloaded automatically.

Tiny-ImageNet is a subset of the ILSVRC2012 version of ImageNet and consists of 64 × 64 × 3 down-sampled images. It needs to be downloaded manually from the official website.

CurBench
└── data
    ├── cifar-10-batches-py
    │   ├── data_batch_1
    │   ├── data_batch_2
    │   ├── ...
    │   └── test_batch
    ├── cifar-100-python
    │   ├── train
    │   ├── test
    │   └── meta
    │   └── ...
    └── tiny-imagenet-200
        ├── train
        ├── val
        └── test

# For easier data processing, we use a Tiny-ImageNet dataset utility class for pytorch: https://gist.github.com/lromor/bcfc69dcf31b2f3244358aea10b7a11b
# After the processing, the directory becomes:

CurBench
└── data
    └── tiny-imagenet-200
        ├── train_batch
        ├── val_batch
        └── ...

Text

GLUE will be downloaded automatically and it consists of cola, sst2, mrpc, qqp, stsb, mnli, qnli, rte, ...

Graph

TUDataset will be downloaded automatically and it consists of many datasets, among which we choose MUTAG, PROTEINS, NCI1

OGB will be downloaded automatically and it consists of many datasets, among which we choose molhiv

Quick Start

# 1. clone from the repository
git clone 
cd CurBench

# 2. pip install local module: curbench
pip install -e .

# 3. prepare dataset

# 4. run the example code
python examples/base.py

Run

Single Run

# 1. vision standard
python examples/base.py --data <cifar10/cifar100/tinyimagenet> --net <lenet/resnet18/vit> --gpu <0/1/2/...>

# 2. text standard
python examples/base.py --data <rte/sst2/cola/...> --net <lstm/bert/gpt> --gpu <0/1/2/...>

# 3. graph standard
python examples/base.py --data <mutag/proteins/nci1/molhiv> --net <gcn/gat/gin> --gpu <0/1/2/...>

# Note: Do not use LRE, MW-Net and DDS when backbone model is LSTM, which is not suitable for direct gradient calculation.

Batch Run

python run.py

Cite

Please cite our paper as follows if you find our code useful:

@inproceedings{zhoucurbench,
  title={CurBench: Curriculum Learning Benchmark},
  author={Zhou, Yuwei and Pan, Zirui and Wang, Xin and Chen, Hong and Li, Haoyang and Huang, Yanwen and Xiong, Zhixiao and Xiong, Fangzhou and Xu, Peiyang and Zhu, Wenwu and others},
  booktitle={Forty-first International Conference on Machine Learning}
}

You may also find our survey paper helpful:

@article{wang2021survey,
  title={A survey on curriculum learning},
  author={Wang, Xin and Chen, Yudong and Zhu, Wenwu},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  publisher={IEEE}
}