2023-AICOSS

Content

1. train val split
2. Training
3. preprocessing
4. Loss
5. Model
6. Ablation Study
7. Result
8. Leaderboard
9. Reference

train val split

result = []

for i in range(100):
    train_, val_ = train_test_split(train, test_size=0.1, random_state=i, shuffle=True)
    re_val = np.array(calcul_pro(val_))
    re_train = np.array(calcul_pro(train_))
    difference = re_train - re_val
    std_value = difference.std()
    if np.abs(std_value) < 0.25:
        mean_value = difference.mean()
        result.append((i, mean_value, std_value))

for state, mean, std in result:
    print(f'random_state: {state}  {mean:.2f} {std:.2f}')

We executed a methodical assessment of the class-wise ratio disparities between the training and validation sets across 100 different random states. Given that these measurements are expressed in percentages, the standard deviation (std) holds greater significance than the mean in our analysis. Hence, we proceeded with the selection of the random state that not only exhibits a std less than 0.25 but also possesses the minimal mean value within that subset.

Through the above method, we compared the training mAP of worst seed (77) and best seed (78) and the val mAP. While the training map is almost the same, the validation mAP of worst seed is performing much worse

split seed	mean	std	train mAP	val mAP
77	0.32	0.41	0.9557	0.9665
78	0.02	0.22	0.9533	0.9752

Training

Virtual Environment Settings

git clone https://github.com/seok-AI/2023-AICOSS
cd 2023-AICOSS/
conda env create -f aicoss.yaml

conda activate aicoss

training example

python main.py \
--model_name tresnet_xl_mldecoder \
--loss_name PartialSelectiveLoss \
--epochs 10 \
--lr 3e-4 \
--min_lr 1e-6 \
--weight_decay 1e-5 \
--augment weak \
--gpu 0

Training Codes

For multi-GPU training, you can use this repository.

preprocessing

Cutout

RandAugment

RandAugment paper

AutoAugment

AutoAugment paper

Loss

PartialSelectiveLoss

PartialSelectiveLoss paper

Model

TResNet + ML decoder

TresNet paper

ML-Decoder paper

Ablation Study

[ Ablation Study with TResNet_XL + ASL ] When augmentation was not performed, validation loss increased and overfitting occurred. When only one augmentation was added, the performance was always better than when not added, and when all augmentation was applied, the best performance was achieved without overfitting.

Augment	Train mAP	Validation mAP
None	1.0	0.9707
AutoAugment	0.9989	0.9720
Cutout	0.9971	0.9727
All	0.9872	0.9751

Result

Ensemble	Public mAP	Private mAP	Single GPU	Multi GPU
Single Model	0.96904	0.96805	1h	<20m
3-Model	0.97478	0.97547	3h	<1h
6-Model	0.97618	0.97707	6h	<2h
6-Model + CvT384	0.97705	0.97764	-	<9h
6-Model + CvT384 + SwinV2	0.97848	0.97914	-	<13h

Leaderboard

Reference

• https://github.com/Alibaba-MIIL/ML_Decoder
• https://github.com/Alibaba-MIIL/TResNet
• https://github.com/huggingface/pytorch-image-models/tree/main/timm
• https://arxiv.org/pdf/1909.13719.pdf
• https://arxiv.org/pdf/2110.10955.pdf
• https://arxiv.org/pdf/2003.13630.pdf
• https://arxiv.org/pdf/2111.12933.pdf