custom dataset with 4 classes

[SuzannaLin]

Thank you so far for all your great help.
I have an issue that I also found in the closed issues, but for me it isn't solved.
I have my own custom data set with 4 classes (background and 3 objects, labeled 0-3), so I changed num_classes = 4 in voc.py
The results with training fully supervised are as shown in the image below. There is one class with an IoU of 0.0.

I ran multiple tests, using semi and weakly supervised settings, the results are unpredictable and often show 0.0 for the object classes. Only the background has good results.
Is there something I need to adjust in the code?

[yassouali]

Hi @SuzannaLin

For your dataset, how much labeled instances you have? and is the bg class the dominant class?
can you also post your config, thanks.

[SuzannaLin]

I haven't done a pixel count yet, but the BG is the biggest one, object label 1 is least present. There are lots of images with only BG present. Would that be the reason?
I have a total of 312 labeled images, and 1064 in the unsupervised data set (those all have at least 1 object present). I'll post my config below.

[SuzannaLin]

{
"name": "CCT",
"experim_name": "CCT_B_4",
"n_gpu": 1,
"n_labeled_examples": 312,
"diff_lrs": true,
"ramp_up": 0.1,
"unsupervised_w": 30,
"ignore_index": 255,
"lr_scheduler": "Poly",
"use_weak_lables":false,
"weakly_loss_w": 0.4,
"pretrained": true,

"model":{
    "supervised": true,
    "semi": false,
    "supervised_w": 1,

    "sup_loss": "ABCE",
    "un_loss": "MSE",

    "softmax_temp": 1,
    "aux_constraint": false,
    "aux_constraint_w": 1,
    "confidence_masking": false,
    "confidence_th": 0.5,

    "drop": 6,
    "drop_rate": 0.5,
    "spatial": true,

    "cutout": 6,
    "erase": 0.4,

    "vat": 2,
    "xi": 1e-6,
    "eps": 2.0,

    "context_masking": 2,
    "object_masking": 2,
    "feature_drop": 6,

    "feature_noise": 6,
    "uniform_range": 0.3
},


"optimizer": {
    "type": "SGD",
    "args":{
        "lr": 1e-2,
        "weight_decay": 1e-4,
        "momentum": 0.9
    }
},


"train_supervised": {
    "data_dir": "/mnt/GeomaticsStaff/Suzanna Cuypers/CCT/data",
    "batch_size": 2,
    "crop_size": 320,
    "shuffle": true,
    "base_size": 400,
    "scale": true,
    "augment": true,
    "flip": true,
    "rotate": false,
    "blur": false,
    "split": "train_supervised",
    "num_workers": 8
},

"train_unsupervised": {
    "data_dir": "/mnt/GeomaticsStaff/Suzanna Cuypers/CCT/data",
    "weak_labels_output": "pseudo_labels/result/pseudo_labels",
    "batch_size": 2,
    "crop_size": 320,
    "shuffle": true,
    "base_size": 400,
    "scale": true,
    "augment": true,
    "flip": true,
    "rotate": false,
    "blur": false,
    "split": "train_unsupervised",
    "num_workers": 8
},

"val_loader": {
    "data_dir": "/mnt/GeomaticsStaff/Suzanna Cuypers/CCT/data",
    "batch_size": 2,
    "val": true,
    "split": "val",
    "shuffle": false,
    "num_workers": 4
},

"trainer": {
    "epochs": 30,
    "save_dir": "saved/",
    "save_period": 5,

    "monitor": "max Mean_IoU",
    "early_stop": 10,
    
    "tensorboardX": true,
    "log_dir": "saved/",
    "log_per_iter": 5,

    "val": true,
    "val_per_epochs": 5
}

}

[SuzannaLin]

The tensorboard also shows strange results.
CCT_B_2: semi, using weak labels
CCT_B_1: semi, no weak labels
CCT_B_4: fully supervised

[SuzannaLin]

I've done a pixel count of the 312 training images.

[SuzannaLin]

I switched the labeled and unlabeled dataset, to see if training with images with at least another object besides BG would make a difference. The least present object still gives nothing 1: 0.0

Tensorboard graph of mIoU also flattens out quickly. (light blue)

[yassouali]

Hi @SuzannaLin

I must say, the low mIoU of the second class is expected, you can see the pixel count is a bit low, can you train with a standard CE instead of abCE ? can you also use a larger batch size to help with batch norm fine-tuning.

[SuzannaLin]

I'll run with CE and see what happens. I can't increase the batch size due to low memory.
I ran inference using the net trained with the switched datasets. The only labeled object is class 3. There is no single image with class 1 or 2 labeled.

[SuzannaLin]

fully-supervised with 312 labeled images and CE loss, the results are a bit lower.

[yassouali]

Can you train for a bit longer (say 100 epochs) with a lower LR.

[SuzannaLin]

I ran two tests for 100 and 200 epochs with first CE loss (top), then ABCE loss (bottom)

It seems that CE loss performs slightly better and has less over fitting. Changing the LR definitely helped, but only in the first 30 epochs. Then, the model focused too much on the BG and LR labels.
The best VAL mIoU was obtained for both tests at 90 epochs. (0.304 for CE, 0.298 for ABCE)
I will try different LR

[yassouali]

Yes, I think the model overfits to the frequent classes quickly, maybe try using a higher abCE loss threshold (thresh), if it does not work, you can try focal loss (can be found here: https://github.com/JunMa11/SegLoss)

[SuzannaLin]

Finally I can report to you some of my results. It took me a bit to get the Focal Loss function going, but the results are a big improvement!

Now, the model, trained supervised achieves a mIoU of 0.424 at 84 epochs, LR 1e-3.
Although the smallest class doesn't do that well, it is expected since its low presence in the training images.
Now, I move on to adding the unlabeled data branch!

Thank you so much for your help and advice!

[yassouali]

@SuzannaLin glad things improved.
Just a suggestion, but given that I suppose you've adapted the focal loss for this repo, maybe you can propose a pull request adding the focal loss to the available losses for future use cases.

[akmaralAW]

@SuzannaLin, hello !

You've mentioned labels from 0-3 for 4 classes. Where should I add this change ? I am trying to train on my own dataset. But got mIoU 0 for another class.

Thank you !

[wgcban]

You can change it in the dalaloarder function.

[SuzannaLin]

Yes, indeed. In dataloaders/voc.py, you can set the number of classes.

class VOCDataset(BaseDataSet):
    def __init__(self, **kwargs):
        self.num_classes = 4

[00000022234]

@SuzannaLin Hello, my dataset num_ Classes = 2 (0-1, where 0 is the background) In the inference process, the target object result is 1:0.0. Only when num_ When classes = 21, there will be normal inference results. How to solve this problem? Thank you.

[SuzannaLin]

@00000022234 which loss function are you using? I used Focal Loss because my objects were very small and I had mostly background pixels. If you want to try Focal Loss, set "sup_loss": "FL" in the config file. You can use the function get_alpha or choose your own alpha values for your BG and object class.

[00000022234]

@SuzannaLin Thank you very much for your reply. I used your method to solve the above problems. How to adjust the parameters of the number of decoders in the code? And how to adjust the parameters of the number of noise? Thank you！

[SuzannaLin]

The number of decoders is set in the config file:
drop: 6 = DropOut
cutout: 6 = G-CutOut
vat: 2 = I-VAT
context_masking: 2 = Con-Msk
object_masking: 2 = Obj-Mask
feature_drop: 6 = F-Drop
feature_noise: 6 = F-Noise

[00000022234]

@SuzannaLin Thank you very much for your reply. I have solved the problem that has bothered me for a long time.