[Doc]SSOD tutorial

README.md

@@ -108,6 +108,7 @@ Please see [get_started.md](docs/en/get_started.md) for the basic usage of MMDet
 - [export ONNX to TRT](docs/en/tutorials/onnx2tensorrt.md)
 - [weight initialization](docs/en/tutorials/init_cfg.md)
 - [how to xxx](docs/en/tutorials/how_to.md)
+- [semi-supervised_detection](docs/en/tutorials/semi_det.md)
 
 ## Overview of Benchmark and Model Zoo
 

README_zh-CN.md

@@ -108,6 +108,7 @@ MMDetection 是一个基于 PyTorch 的目标检测开源工具箱。它是 [Ope
 - [ONNX 到 TensorRT 的模型转换](docs/zh_cn/tutorials/onnx2tensorrt.md)
 - [权重初始化](docs/zh_cn/tutorials/init_cfg.md)
 - [how to xxx](docs/zh_cn/tutorials/how_to.md)
+- [半监督目标检测](docs/zh_cn/tutorials/semi_det.md)
 
 同时，我们还提供了 [MMDetection 中文解读文案汇总](docs/zh_cn/article.md)
 

docs/en/tutorials/semi_det.md

@@ -0,0 +1,284 @@
+# Tutorial 14: Semi-supervised Object Detection
+
+The process of semi-supervised object detection is as follows:
+
+- Data Preparation
+- Data Pipeline
+- Semi-supervised dataloader
+- Model preparation
+- Model update
+- Model validation
+
+## Data Preparation
+
+We provide a dataset download script, which downloads the coco2017 dataset by default and decompresses it automatically.
+
+```shell
+python tools/misc/download_dataset.py
+```
+
+The decompressed dataset directory is as follows:
+
+```plain
+mmdetection
+├── data
+│   ├── coco
+│   │   ├── annotations
+│   │   │   ├── image_info_unlabeled2017.json
+│   │   │   ├── instances_train2017.json
+│   │   │   ├── instances_val2017.json
+│   │   ├── test2017
+│   │   ├── train2017
+│   │   ├── unlabeled2017
+│   │   ├── val2017
+```
+
+There are two common experimental settings for semi-supervised object detection on the coco2017 dataset:
+
+(1) Split `train2017` according to a fixed percentage (1%, 2%, 5% and 10%) as a labeled dataset, and the rest of `train2017` as an unlabeled dataset. Considering that splitting different data from `train2017` as labeled dataset has a great influence on the results of semi-supervised training, five-fold cross-validation is used to evaluate the performance of the algorithm. We provide the dataset split script:
+
+```shell
+python tools/misc/split_coco.py
+```
+
+By default, the script will split `train2017` according to the labeled data ratio  1%, 2%, 5% and 10%, and each split will be randomly repeated 5 times for cross-validation. The generated semi-supervised annotation file name format is as follows:
+
+- the name format of labeled dataset: `instances_train2017.{fold}@{percent}.json`
+- the name format of unlabeled dataset: `instances_train2017.{fold}@{percent}-unlabeled.json`
+
+Here, `fold` is used for cross-validation, and `percent` represents the ratio of labeled data. The directory structure of the divided dataset is as follows:
+
+```plain
+mmdetection
+├── data
+│   ├── coco
+│   │   ├── annotations
+│   │   │   ├── image_info_unlabeled2017.json
+│   │   │   ├── instances_train2017.json
+│   │   │   ├── instances_val2017.json
+│   │   ├── semi_anns
+│   │   │   ├── instances_train2017.1@1.json
+│   │   │   ├── instances_train2017.1@1-unlabeled.json
+│   │   │   ├── instances_train2017.1@2.json
+│   │   │   ├── instances_train2017.1@2-unlabeled.json
+│   │   │   ├── instances_train2017.1@5.json
+│   │   │   ├── instances_train2017.1@5-unlabeled.json
+│   │   │   ├── instances_train2017.1@10.json
+│   │   │   ├── instances_train2017.1@10-unlabeled.json
+│   │   │   ├── instances_train2017.2@1.json
+│   │   │   ├── instances_train2017.2@1-unlabeled.json
+│   │   ├── test2017
+│   │   ├── train2017
+│   │   ├── unlabeled2017
+│   │   ├── val2017
+```
+
+(2) Use `train2017` as the labeled dataset and `unlabeled2017` as the unlabeled dataset. Since `image_info_unlabeled2017.json` does not contain `categories` information, the `CocoDataset` cannot be initialized, so you need to write the `categories` of `instances_train2017.json` into `image_info_unlabeled2017.json` and save it as `instances_unlabeled2017.json`, the relevant script is as follows:
+
+```python
+from mmengine.fileio import load, dump
+
+anns_train = load('instances_train2017.json')
+anns_unlabeled = load('image_info_unlabeled2017.json')
+anns_unlabeled['categories'] = anns_train['categories']
+dump(anns_unlabeled, 'instances_unlabeled2017.json')
+```
+
+The processed dataset directory is as follows:
+
+```plain
+mmdetection
+├── data
+│   ├── coco
+│   │   ├── annotations
+│   │   │   ├── image_info_unlabeled2017.json
+│   │   │   ├── instances_train2017.json
+│   │   │   ├── instances_unlabeled2017.json
+│   │   │   ├── instances_val2017.json
+│   │   ├── test2017
+│   │   ├── train2017
+│   │   ├── unlabeled2017
+│   │   ├── val2017
+```
+
+## Data Pipeline
+
+There are two main approaches to semi-supervised learning, consistency constraints and pseudo-labels. Consistency constraints often require some careful design, while pseudo-labels have a simpler form and are easier to extend to downstream tasks.
+We adopt a teacher-student joint training semi-supervised object detection framework, so labeled data and unlabeled data need to configure different data pipeline:
+
+(1) Pipeline for labeled data：
+
+```python
+# pipeline used to augment labeled data,
+# which will be sent to student model for supervised training.
+sup_pipeline = [
+    dict(type='LoadImageFromFile', file_client_args=file_client_args),
+    dict(type='LoadAnnotations', with_bbox=True),
+    dict(type='RandomResize', scale=scale, keep_ratio=True),
+    dict(type='RandomFlip', prob=0.5),
+    dict(type='RandAugment', aug_space=color_space, aug_num=1),
+    dict(type='FilterAnnotations', min_gt_bbox_wh=(1e-2, 1e-2)),
+    dict(type='MultiBranch', sup=dict(type='PackDetInputs'))
+]
+```
+
+(2) Pipeline for unlabeled data：
+
+```python
+# pipeline used to augment unlabeled data weakly,
+# which will be sent to teacher model for predicting pseudo instances.
+weak_pipeline = [
+    dict(type='RandomResize', scale=scale, keep_ratio=True),
+    dict(type='RandomFlip', prob=0.5),
+    dict(
+        type='PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'flip', 'flip_direction',
+                   'homography_matrix')),
+]
+
+# pipeline used to augment unlabeled data strongly,
+# which will be sent to student model for unsupervised training.
+strong_pipeline = [
+    dict(type='RandomResize', scale=scale, keep_ratio=True),
+    dict(type='RandomFlip', prob=0.5),
+    dict(
+        type='RandomOrder',
+        transforms=[
+            dict(type='RandAugment', aug_space=color_space, aug_num=1),
+            dict(type='RandAugment', aug_space=geometric, aug_num=1),
+        ]),
+    dict(type='RandomErasing', n_patches=(1, 5), ratio=(0, 0.2)),
+    dict(type='FilterAnnotations', min_gt_bbox_wh=(1e-2, 1e-2)),
+    dict(
+        type='PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'flip', 'flip_direction',
+                   'homography_matrix')),
+]
+
+# pipeline used to augment unlabeled data into different views
+unsup_pipeline = [
+    dict(type='LoadImageFromFile', file_client_args=file_client_args),
+    dict(type='LoadEmptyAnnotations'),
+    dict(
+        type='MultiBranch',
+        unsup_teacher=weak_pipeline,
+        unsup_student=strong_pipeline,
+    )
+]
+```
+
+## Semi-supervised dataloader
+
+(1) Build a semi-supervised dataset. Use `ConcatDataset` to concatenate labeled and unlabeled datasets.
+
+```python
+labeled_dataset = dict(
+    type=dataset_type,
+    data_root=data_root,
+    ann_file='annotations/instances_train2017.json',
+    data_prefix=dict(img='train2017/'),
+    filter_cfg=dict(filter_empty_gt=True, min_size=32),
+    pipeline=sup_pipeline)
+
+unlabeled_dataset = dict(
+    type=dataset_type,
+    data_root=data_root,
+    ann_file='annotations/instances_unlabeled2017.json',
+    data_prefix=dict(img='unlabeled2017/'),
+    filter_cfg=dict(filter_empty_gt=False),
+    pipeline=unsup_pipeline)
+
+train_dataloader = dict(
+    batch_size=batch_size,
+    num_workers=num_workers,
+    persistent_workers=True,
+    sampler=dict(
+        type='GroupMultiSourceSampler',
+        batch_size=batch_size,
+        source_ratio=[1, 4]),
+    dataset=dict(
+        type='ConcatDataset', datasets=[labeled_dataset, unlabeled_dataset]))
+```
+
+(2) Multi-source dataset sampler. Use `GroupMultiSourceSampler` to sample data form batches from `labeled_dataset` and `labeled_dataset`, `source_ratio` controls the proportion of labeled data and unlabeled data in the batch. `GroupMultiSourceSampler` also ensures that the images in the same batch have similar aspect ratios. If you don't need to guarantee the aspect ratio of the images in the batch, you can use `MultiSourceSampler`. The sampling diagram of `GroupMultiSourceSampler` is as follows:
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/40661020/186149261-8cf28e92-de5c-4c8c-96e1-13558b2e27f7.jpg"/>
+</div>
+
+`sup=1000` indicates that the scale of the labeled dataset is 1000, `sup_h=200` indicates that the scale of the images with an aspect ratio greater than or equal to 1 in the labeled dataset is 200, and `sup_w=800` indicates that the scale of the images with an aspect ratio less than 1 in the labeled dataset is 800,
+`unsup=9000` indicates that the scale of the unlabeled dataset is 9000, `unsup_h=1800` indicates that the scale of the images with an aspect ratio greater than or equal to 1 in the unlabeled dataset is 1800, and `unsup_w=7200` indicates the scale of the images with an aspect ratio less than 1 in the unlabeled dataset is 7200.
+`GroupMultiSourceSampler` randomly selects a group according to the overall aspect ratio distribution of the images in the labeled dataset and the unlabeled dataset, and then sample data to form batches from the two datasets according to `source_ratio`, so labeled datasets and unlabeled datasets have different repetitions.
+
+## Model preparation
+
+We usually choose `Faster R-CNN` as `detector` for semi-supervised training. Take the semi-supervised object detection algorithm `SoftTeacher` as an example,
+the model configuration can be inherited from `_base_/models/faster-rcnn_r50_fpn.py`, replacing the backbone network of the detector with `caffe` style.
+Note that unlike the supervised training configs, `Faster R-CNN` as `detector` is an attribute of `model`, not `model` .
+In addition, `data_preprocessor` needs to be set to `MultiBranchDataPreprocessor`, which is used to pad and normalize images from different pipelines.
+Finally, parameters required for semi-supervised training and testing can be configured via `semi_train_cfg` and `semi_test_cfg`.
+
+```python
+_base_ = [
+    '../_base_/models/faster-rcnn_r50_fpn.py', '../_base_/default_runtime.py',
+    '../_base_/datasets/semi_coco_detection.py'
+]
+
+detector = _base_.model
+detector.data_preprocessor = dict(
+    type='DetDataPreprocessor',
+    mean=[103.530, 116.280, 123.675],
+    std=[1.0, 1.0, 1.0],
+    bgr_to_rgb=False,
+    pad_size_divisor=32)
+detector.backbone = dict(
+    type='ResNet',
+    depth=50,
+    num_stages=4,
+    out_indices=(0, 1, 2, 3),
+    frozen_stages=1,
+    norm_cfg=dict(type='BN', requires_grad=False),
+    norm_eval=True,
+    style='caffe',
+    init_cfg=dict(
+        type='Pretrained',
+        checkpoint='open-mmlab://detectron2/resnet50_caffe'))
+
+model = dict(
+    _delete_=True,
+    type='SoftTeacher',
+    detector=detector,
+    data_preprocessor=dict(
+        type='MultiBranchDataPreprocessor',
+        data_preprocessor=detector.data_preprocessor),
+    semi_train_cfg=dict(
+        freeze_teacher=True,
+        sup_weight=1.0,
+        unsup_weight=4.0,
+        pseudo_label_initial_score_thr=0.5,
+        rpn_pseudo_thr=0.9,
+        cls_pseudo_thr=0.9,
+        reg_pseudo_thr=0.02,
+        jitter_times=10,
+        jitter_scale=0.06,
+        min_pseudo_bbox_wh=(1e-2, 1e-2)),
+    semi_test_cfg=dict(predict_on='teacher'))
+```
+
+## Model update
+
+Usually, the teacher model is updated by EMA the student model, and then the teacher model is optimized with the optimization of the student model, which can be achieved by configuring `custom_hooks`:
+
+```python
+custom_hooks = [dict(type='MeanTeacherHook')]
+```
+
+## Model validation
+
+Since there are two models in the teacher-student joint training framework, we can replace `ValLoop` with `TeacherStudentValLoop` to test the accuracy of both models during the training process.
+
+```python
+val_cfg = dict(type='TeacherStudentValLoop')
+```