About the results on VOC2012 with 92 examples

[GritCS]

Thank you for your expressive and remarkable work and delicate code!! But when I reproducting the results on VOC2012 with 92 examples,I only get 59 with examples,while the reported result is 67.98.Can you offer some suggest? Thank you sincerely!

This is my log file!
seg_20220414_150853.txt

This is my tb about contrastive_loss and unsupervised loss! they're kind of weired and on an upward trend.

[GritCS]

This is my config file. And I run my code in 4 gpu(4*V100),batch_size = 4

dataset: # Required.
type: pascal_semi
train:
data_root: ../../../../../data/VOC2012
data_list: ../../../../../data/splits/pascal/92/labeled.txt
flip: True
GaussianBlur: False
rand_resize: [0.5, 2.0]
#rand_rotation: [-10.0, 10.0]
crop:
type: rand
size: [513, 513] # crop image with HxW size
val:
data_root: ../../../../../data/VOC2012
data_list: ../../../../../data/splits/pascal/val.txt
crop:
type: center
size: [513, 513] # crop image with HxW size
batch_size: 4
n_sup: 92
noise_std: 0.1
workers: 2
mean: [123.675, 116.28, 103.53]
std: [58.395, 57.12, 57.375]
ignore_label: 255

trainer: # Required.
epochs: 80
eval_on: True
optimizer:
type: SGD
kwargs:
lr: 0.001 # 4GPUs
momentum: 0.9
weight_decay: 0.0001
lr_scheduler:
mode: poly
kwargs:
power: 0.9
unsupervised:
TTA: False
drop_percent: 80
apply_aug: cutmix
contrastive:
loss_weight: 0.1
negative_high_entropy: True
low_rank: 3
high_rank: 20
current_class_threshold: 0.3
current_class_negative_threshold: 1
unsupervised_entropy_ignore: 80
low_entropy_threshold: 20
num_negatives: 50
num_queries: 256
temperature: 0.5

saver:
snapshot_dir: checkpoints/base_16
pretrain: ''

criterion:
type: CELoss
kwargs:
use_weight: False

net: # Required.
num_classes: 21
sync_bn: True
ema_decay: 0.99
encoder:
type: u2pl.models.resnet.resnet101
kwargs:
multi_grid: True
zero_init_residual: True
fpn: True
replace_stride_with_dilation: [False, True, True] #layer0...1 is fixed, layer2...4
decoder:
type: u2pl.models.decoder.dec_deeplabv3_plus
kwargs:
inner_planes: 256
dilations: [12, 24, 36]

[Haochen-Wang409]

Hi, thanks for your approval!

From the loss curve, unsup_loss seems to be a little bit wired.
The contra_loss can be regarded as a regularizer, which is expected to be around 0.2 during the whole training process.
The provided training log seems to be incomplete since it only contains 27 epochs? Could you provide the whole log?

By the way, with only 92 labeled samples, the training procedure is a little bit noisy.
Maybe you could try applying aux_loss, which could be found in: https://github.com/Haochen-Wang409/U2PL/blob/main/experiments/cityscapes/744/suponly/config.yaml#L57-L59

[Haochen-Wang409]

I guess it might beause unsup_loss is around 0.3, which is quite large than the sup_loss in your log file.
Therefore, aux_loss or ohem might help.

[GritCS]

Yeap,there're only 27 epochs.Due to the poor performance,I interrupted the training.
Thanks for your helpful suggestion!! I'll retraining the model with only 92 labeled samples with the config.yaml you provide. Btw, there's no need to change the value of relevant parameter about aux_loss (aux_plane: 1024 and loss_weight: 0.4 )?

Thanks sincerely again!

[Haochen-Wang409]

Yes, just set aux_plane: 1024 and loss_weight: 0.4.

Note that we did not adopt aux_loss or ohem in experiments on VOC.

[GritCS]

Hello! I have some question about the split of voc dataset .

When I train the model with classic voc 2012. Should I change 10582 to 1464 ( line 109 in the pascal_voc.py file. )?

What about you when you trains model on classic voc,Thanks!!

[Haochen-Wang409]

You do not need to change 10582.
In fact, n_sup in here actually means number of unlabeled images, since it is used to build dataloader here.

For semi-supervised settings, usually an epoch is defined as the model have been trained on all unlabeled images.

[GritCS]

But I understand that the classic VOC 2012 dataset is divided into training,validationand test sets including 1,464, 1,449 and 1,456 images, respectively. So the number of unlabeled images should be 1464 - num_labeled. So this's 1464 not 10582. Meybe I mistake the composition of classic VOC 2012 dataset ? Thanks!

[Haochen-Wang409]

Augmented VOC contains 10582 images in total.
For classical VOC, labeled images are selected from 1464 fine annotated samples and the remaining 9118 images are regarded as unlabeled images. Therefore, number of total images are 10582 rather than 1464.

[GritCS]

Augmented VOC contains 10582 images in total. For classical VOC, labeled images are selected from 1464 fine annotated samples and the remaining 9118 images are regarded as unlabeled images. Therefore, number of total images are 10582 rather than 1464.

Augmented VOC contains 10582 images in total. For classical VOC, labeled images are selected from 1464 fine annotated samples and the remaining 9118 images are regarded as unlabeled images. Therefore, number of total images are 10582 rather than 1464.

oo！！I see!! Thank you very much!!

[Haochen-Wang409]

Never mind~!