num_items_in_batch larger than the actual useful token when computing loss

[SHIFTTTTTTTT]

def fixed_cross_entropy(source, target, num_items_in_batch: int = None, ignore_index: int = -100, **kwargs):
I check the shape of the inputs and find follows:
In [1]: logits.shape
Out[1]: torch.Size([4, 896, 152064])

In [2]: labels.shape
Out[2]: torch.Size([4, 896])

In [3]: num_items_in_batch
Out[3]: 4390

Why is 4390>4*896?

[juyterman1000]

looks like the problem is with the gradient accumulation and num_items_in_batch is calculation needs to be rectified

[Rocketknight1]

cc @SunMarc

[SunMarc]

This is indeed a bit strange. Can you share a minimal reproducer @SHIFTTTTTTTT. If you want to have a look at how it is computed, you can find it in get_batch_samples. The calculation is actually pretty simple:

num_items_in_batch = sum([(batch["labels"].ne(-100)).sum() for batch in batch_samples])

[SHIFTTTTTTTT]

I have understand the reason. It is because I set gradient accumulation step =2.So this get_batch_samples contains two batches !