[Feature] Support K-fold cross-validation

[mzr1996]

Motivation

K-Fold cross-validation is commonly used in small dataset training. Here we add the support of the K-fold dataset wrapper.
Closing #560

Modification

1. Add a new dataset wrapper, KFoldDataset.
2. Support to use indices to specify which samples to evaluate.

BC-breaking (Optional)

No

Use cases (Optional)

Here we modify the resnet18_8xb32_in1k.py as an example.

_base_ = [
    '../_base_/models/resnet18.py',
    '../_base_/schedules/imagenet_bs256.py', '../_base_/default_runtime.py'
]

# dataset settings
dataset_type = 'ImageNet'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='RandomResizedCrop', size=224),
    dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='ToTensor', keys=['gt_label']),
    dict(type='Collect', keys=['img', 'gt_label'])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='Resize', size=(256, -1)),
    dict(type='CenterCrop', crop_size=224),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='Collect', keys=['img'])
]


data = dict(
    samples_per_gpu=32,
    workers_per_gpu=2,
    train=dict(
        type='KFoldDataset',
        dataset=dict(
            type=dataset_type,
            data_prefix='data/imagenet/train',
            pipeline=train_pipeline),
        # Modify the `fold` to use different split. For 5-fold cross-validation,
        # five experiments need to be executed (fold=0, fold=1, fold=2, ...)
        fold=0,
        num_splits=5,
        # seed = 1,   # If set seed, shuffle samples before spliting dataset.
    ),
    val=dict(
        type='KFoldDataset',
        dataset=dict(
            type=dataset_type,
            data_prefix='data/imagenet/train',   # For K-Fold cross-validation, all images should be place in the same folder. 
            pipeline=test_pipeline),
        # All parameters need to be the same in train/val/test set, like fold, num_splits and seed
        fold=0,
        num_splits=5,
        # seed=1,
        # `test_mode` will be set to True automatically in `apis/train.py` and `tools/test.py`
        # and you can also specify it explicitly.
        # test_mode=True,
    ),
    test=dict(
        type='KFoldDataset',
        dataset=dict(
            type=dataset_type,
            data_prefix='data/imagenet/train',
            pipeline=test_pipeline),
        fold=0,
        num_splits=5,
        # seed=1,
        # test_mode=True,
    ))
evaluation = dict(interval=1, metric='accuracy')

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
• CLA has been signed and all committers have signed the CLA in this PR.

[codecov]

Codecov Report

Merging #563 (49c25cf) into dev (321ad09) will increase coverage by 0.31%.
The diff coverage is 88.57%.

@@            Coverage Diff             @@
##              dev     #563      +/-   ##
==========================================
+ Coverage   81.78%   82.10%   +0.31%     
==========================================
  Files         118      118              
  Lines        6820     6855      +35     
  Branches     1174     1181       +7     
==========================================
+ Hits         5578     5628      +50     
+ Misses       1082     1063      -19     
- Partials      160      164       +4     

Flag	Coverage Δ
unittests	82.10% <88.57%> (+0.31%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/datasets/__init__.py	100.00% <ø> (ø)
mmcls/datasets/base_dataset.py	91.11% <0.00%> (-2.08%)	⬇️
mmcls/datasets/multi_label.py	75.00% <0.00%> (-3.95%)	⬇️
mmcls/datasets/builder.py	90.62% <100.00%> (+17.41%)	⬆️
mmcls/datasets/dataset_wrappers.py	91.66% <100.00%> (+2.62%)	⬆️
mmcls/datasets/imagenet.py	55.31% <0.00%> (+21.27%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 321ad09...49c25cf. Read the comment docs.

[liluhu0]

目前还不支持K折交叉吗？

[mzr1996]

目前还不支持K折交叉吗？

这个 PR 目前正在开发中，我们会在近期完成支持。
功能本身应该已经可用了，你也可以参照这个 config 试用一下

_base_ = [
    '../_base_/models/resnet18_cifar.py',
    '../_base_/schedules/cifar10_bs128.py', '../_base_/default_runtime.py'
]

# dataset settings
dataset_type = 'CIFAR10'
img_norm_cfg = dict(
    mean=[125.307, 122.961, 113.8575],
    std=[51.5865, 50.847, 51.255],
    to_rgb=False)
train_pipeline = [
    dict(type='RandomCrop', size=32, padding=4),
    dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='ToTensor', keys=['gt_label']),
    dict(type='Collect', keys=['img', 'gt_label'])
]
test_pipeline = [
    dict(type='Normalize', **img_norm_cfg),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='Collect', keys=['img'])
]
data = dict(
    samples_per_gpu=16,
    workers_per_gpu=2,
    train=dict(
        type='KFoldDataset',
        dataset=dict(
            type=dataset_type,
            data_prefix='data/cifar10',
            pipeline=train_pipeline),
        fold=0,
        num_splits=5),
    val=dict(
        type='KFoldDataset',
        dataset=dict(
            type=dataset_type,
            data_prefix='data/cifar10',
            pipeline=test_pipeline),
        fold=0,
        num_splits=5),
    test=dict(
        type='KFoldDataset',
        dataset=dict(
            type=dataset_type,
            data_prefix='data/cifar10',
            pipeline=test_pipeline),
        fold=0,
        num_splits=5))