Issue about updating training loss

[hpkhanh1610]

Hi, in the "Transfer_Learning_Solution.ipynb", in the training section part, when we update the training loss, there's a line code as follows:

update training loss
train_loss += loss.item()

While in the other 2 notebooks, which are "cifar10_cnn_solution.ipynb" and "mnist_mlp_solution.ipynb", the updating training loss is as follows:

update training loss
train_loss += loss.item()*data.size(0)

Why is there a difference here? Which one is correct: with data.size(0) or without it?
Thank you.

[cezannec]

Hello! These notebooks are calculating slightly different losses. loss.item() is the average loss over a batch of data. So, if a training loop processes 64 inputs/labels in one batch, then loss.item() will be the average loss over those 64 inputs.

The transfer learning notebook is summing up the running loss and then calculating the average batch loss every 20 batches, which you can see in the print statement train_loss / 20.

The mnist and cifar notebooks are calculating the average loss over a single set of inputs, so they first multiply the average batch loss, loss.item(), by the batch_size, data.size(0), and after one epoch has completed, this accumulated loss is divided by the total number of images in the training dataset len(trainloader.dataset). This gives the average loss over a single set of inputs.

[pranaysingh25]

thanks for the explanation! I was stuck.

[pytorchgirl]

Hello! These notebooks are calculating slightly different losses. loss.item() is the average loss over a batch of data. So, if a training loop processes 64 inputs/labels in one batch, then loss.item() will be the average loss over those 64 inputs.

The transfer learning notebook is summing up the running loss and then calculating the average batch loss every 20 batches, which you can see in the print statement train_loss / 20.

The mnist and cifar notebooks are calculating the average loss over a single set of inputs, so they first multiply the average batch loss, loss.item(), by the batch_size, data.size(0), and after one epoch has completed, this accumulated loss is divided by the total number of images in the training dataset len(trainloader.dataset). This gives the average loss over a single set of inputs.

Can I know why seperate losses are calculated?