Fail to reimplement your paper's result for semi-supervised.

[TyroneLi]

I use the default config file to conduct experiments, but I only got 68.9mIoU for not adopting weak label and got 70.09mIoU for adopting weak label following your readme. These results are far lower than yours. My env is pytorch 1.7.0 and python 3.8.5. Could provide some advice?

[yassouali]

Hi @TyroneLi

for how many epochs did you train? and did you use more than 1 gpus?

[TyroneLi]

Hi @TyroneLi

for how many epochs did you train? and did you use more than 1 gpus?
@yassouali
Thanks for your reply. I use the default config you provided, here it is.

`{
"name": "CCT",
"experim_name": "CCT",
"n_gpu": 1,
"n_labeled_examples": 1464,
"diff_lrs": true,
"ramp_up": 0.1,
"unsupervised_w": 30,
"ignore_index": 255,
"lr_scheduler": "Poly",
"use_weak_lables":true,
"weakly_loss_w": 0.4,
"pretrained": true,

"model":{
    "supervised": false,
    "semi": true,
    "supervised_w": 1,

    "sup_loss": "CE",
    "un_loss": "MSE",

    "softmax_temp": 1,
    "aux_constraint": false,
    "aux_constraint_w": 1,
    "confidence_masking": false,
    "confidence_th": 0.5,

    "drop": 6,
    "drop_rate": 0.5,
    "spatial": true,

    "cutout": 6,
    "erase": 0.4,

    "vat": 2,
    "xi": 1e-6,
    "eps": 2.0,

    "context_masking": 2,
    "object_masking": 2,
    "feature_drop": 6,

    "feature_noise": 6,
    "uniform_range": 0.3
},


"optimizer": {
    "type": "SGD",
    "args":{
        "lr": 1e-2,
        "weight_decay": 1e-4,
        "momentum": 0.9
    }
},


"train_supervised": {
    "data_dir": "/data/voc/",
    "batch_size": 10,
    "crop_size": 320,
    "shuffle": true,
    "base_size": 400,
    "scale": true,
    "augment": true,
    "flip": true,
    "rotate": false,
    "blur": false,
    "split": "train_supervised",
    "num_workers": 8
},

"train_unsupervised": {
    "data_dir": "/data/voc/",
    "weak_labels_output": "pseudo_labels/result/pseudo_labels/",
    "batch_size": 10,
    "crop_size": 320,
    "shuffle": true,
    "base_size": 400,
    "scale": true,
    "augment": true,
    "flip": true,
    "rotate": false,
    "blur": false,
    "split": "train_unsupervised",
    "num_workers": 8
},

"val_loader": {
    "data_dir": "/data/voc/",
    "batch_size": 1,
    "val": true,
    "split": "val",
    "shuffle": false,
    "num_workers": 4
},

"trainer": {
    "epochs": 80,
    "save_dir": "saved_use_weak_labels_npy2img/",
    "save_period": 5,

    "monitor": "max Mean_IoU",
    "early_stop": 10,
    
    "tensorboardX": true,
    "log_dir": "saved_use_weak_labels_npy2img/",
    "log_per_iter": 20,

    "val": true,
    "val_per_epochs": 5
}

}`

I just ran on one single GPU with cuda 11.0. But when I change to cuda-10.0, python3.7 and pytorch1.1.0, I still cannot achieve more than 71% mIoU whatever. I donnot know which part I missed or some config I mistook. Hope you could give me some suggestions to reimplement your results. Applying multi-scale inference is able to obtain 72% mIoU but it's inconsistent with your paper's result.
Note that, I mainly want to reimplement for 1.5k supervised plus 9.0k weakly supervised generated for pretrained IRNet.

[yassouali]

well this is a bit weird, the config seems to be correct, pytorch can be quite mysterious sometimes :)

try pytorch 1.4 - 1.6 since these are the version I tested with (same thing for torchvision with the corresponding version)

[TyroneLi]

well this is a bit weird, the config seems to be correct, pytorch can be quite mysterious sometimes :)

try pytorch 1.4 - 1.6 since these are the version I tested with (same thing for torchvision with the corresponding version)

All right. I would try these versions. But it would take a little long time to finish. Hope I can obtain new results today. Thanks.

[TyroneLi]

well this is a bit weird, the config seems to be correct, pytorch can be quite mysterious sometimes :)

try pytorch 1.4 - 1.6 since these are the version I tested with (same thing for torchvision with the corresponding version)

I changed to pytorch1.4.0 and conducted experiment for semi(1.5k supervised and 9k weak labels), I got 72.06mIoU. However, it is still a little bit (nearly 1%) lower than your paper's result.

[yassouali]

hi @TyroneLi

did you also rerun the pseudo labeling process, maybe try regenerating the pseudo labels with the current setting

[TyroneLi]

hi @TyroneLi

did you also rerun the pseudo labeling process, maybe try regenerating the pseudo labels with the current setting

What's the meaning of rerun the pseudo labeling process? Do you mean rerun the pseudo labeling process with IRNet's second step to perform pseudo labeling expansion? But did you perform this in your paper's experiment?

[yassouali]

hi @TyroneLi

no, just rerun the same process to generate the pseudo labels as described in the readme but with the current version of pytorch & torchvision