Assigning class weights to imbalanced dataset

[keerti-ds]

Hi, I am new to Pytroch and I have a difficulty in understanding the concept of setting class weights for imbalanced dataset.

I know I can set class weights in Tensorflow and Keras using from sklearn.utils import class_weight as

model.fit(X_train, Y_train, nb_epoch=5, batch_size=32, class_weight=class_weight)

I would like to know if this labels_to_class_weights function is for the same purpose? Because I see that they are normalizing the weights. I am quite confused, is it not possible to assign the weights as we do in Tensorlfow like?

class_weight = class_weight.compute_class_weight('balanced' ,np.unique(Y_train) ,Y_train)

or this function is doing exactly the same thing?

yolov5/utils/general.py

Line 495 in 7ebb5e5

def labels_to_class_weights(labels, nc=80):

Thanks in advance.

[glenn-jocher]

@keerti-ds class weights are computed automatically for all datasets in train.py here:

yolov5/train.py

Line 255 in 7ebb5e5

model.class_weights = labels_to_class_weights(dataset.labels, nc).to(device) * nc # attach class weights

They're not used during training by default, but you can do whatever you want with them.

[keerti-ds]

@glenn-jocher Thanks for the reply. What do I need to change to use them for training? Because I don't see any parameter where I can pass them.

[glenn-jocher]

CELoss and BCELoss functions have weight arguments. You might experiment with using these weights in classification loss in loss.py
https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html

yolov5/utils/loss.py

Lines 91 to 101 in 7ebb5e5

	class ComputeLoss:
	# Compute losses
	def __init__(self, model, autobalance=False):
	self.sort_obj_iou = False
	device = next(model.parameters()).device # get model device
	h = model.hyp # hyperparameters
	# Define criteria
	BCEcls = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['cls_pw']], device=device))
	BCEobj = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['obj_pw']], device=device))

[keerti-ds]

I see BCEWithLogitsLoss also has a weight argument. Can I not use this? Is there any particular reason you suggested me BCELoss or CELoss?

And if I would want to pass the class weights, can I just add them in hyp.yaml and access in loss.py from this file as you are accessing h['cls_pw']?

[glenn-jocher]

@keerti-ds yes sure you can do whatever you want.

pw is a positive weight, which is different than a class weight. See the PyTorch docs.

[kalfasyan]

The original question is how to use class_weights and I don't see how it got answered in this discussion. How do we use the model's class weights during training?

[glenn-jocher]

@kalfasyan CELoss and BCELoss functions have weight arguments. You might experiment with using these weights in classification loss in loss.py
https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html

yolov5/utils/loss.py

Lines 91 to 101 in 7ebb5e5

	class ComputeLoss:
	# Compute losses
	def __init__(self, model, autobalance=False):
	self.sort_obj_iou = False
	device = next(model.parameters()).device # get model device
	h = model.hyp # hyperparameters
	# Define criteria
	BCEcls = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['cls_pw']], device=device))
	BCEobj = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['obj_pw']], device=device))

[keerti-ds]

Thanks! But I see weight argument in docs.

torch.nn.BCEWithLogitsLoss(weight=None, size_average=None, reduce=None, reduction='mean', pos_weight=None)
https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss

And pw is here pos_weight if I understand correctly. And weight should be the class weight. Please correct me if I am wrong.

And thanks, I will try with other losses.

[glenn-jocher]

@keerti-ds as far as I understand it pos_weight is a scalar and weight may be a vector. The two arguments serve two completely different purposes. But I'm not the module author, if you have questions regarding it you may want to raise them on the pytorch forums.

[RicoSuaveGuapo]

Correct me if I was wrong

Authors provide sampling method for imbalance dataset through --image-weights flag in train.py. One can check

yolov5/train.py

Lines 288 to 291 in 63ddb6f

	if opt.image_weights:
	cw = model.class_weights.cpu().numpy() * (1 - maps) ** 2 / nc # class weights
	iw = labels_to_image_weights(dataset.labels, nc=nc, class_weights=cw) # image weights
	dataset.indices = random.choices(range(dataset.n), weights=iw, k=dataset.n) # rand weighted idx

where they use class weights and instances' classes in a image to decide how dataloader sample data (since this satistics needs dataloader to be static, so they use Dataloader not InfiniteDataloader). In the other word, sampling more images with rare classes than the others.

The other method is directly change the class weight which is this thread mainly discussed. We can get the class weights directly from authors' code

yolov5/train.py

Line 266 in 63ddb6f

model.class_weights = labels_to_class_weights(dataset.labels, nc).to(device) * nc # attach class weights

with the shape of (nc). One can save/copy it, then put it to hyp.scratch.yaml's option cls_pw.
To be clearify, https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html
says that

• pos_weight: has to shape of classes, which should a vector
• weight: is to control how you weight elements in a batch. I.e, for a batch [tensor_a, tensor_b], we set weight=[10, 1] means you handle the loss computation with additional 10 times weight for tensor_a

[saulcanortiz]

Hi!
Then, imagine you are training an imbalanced custom dataset with 4 classes (3 minor classes and 1 major class) and you want to set this class_weights = [0.3, 0.3, 0.3, 0.1]. How can you do it using any yolov5 sub-model? @glenn-jocher

[glenn-jocher]

model.class_weights are automatically created from your dataset based on inverse frequency. This can be used to customize loss functions etc, but are not used by default.

[saulcanortiz]

Therefore, if I declare model.class_weights = torch.tensor([0.3, 0.3, 0.3, 0.1]), the alpha weights of the Focal loss in losses.py are going to be the model.class_weights. Is it correct? Should I add extra lines of code to make the alpha-weights = model.cass_weights?

Regards

[glenn-jocher]

@saulcanortiz yes you can access your model.class_weights inside the loss functions in utils/loss.py and apply them however you want.