`num_input_tokens_seen` should exclude padding tokens

[RylanSchaeffer]

Feature request

The Trainer enables tracking the number of input tokens. I was puzzling over why the logged number of input tokens was so much higher than the number of tokens in my dataset and then I discovered that the logged number of input tokens doesn't exclude padding tokens:

input_tokens = inputs[main_input_name].numel()
input_tokens = torch.tensor(input_tokens, device=self.args.device, dtype=torch.int64)
self.state.num_input_tokens_seen += (
    self.accelerator.gather(input_tokens).sum().cpu().item()
)

I see two solutions:

1. Setting include_num_input_tokens_seen=True should exclude padding tokens by default

2. Introduce a new TrainingArguments flag likeinclude_num_input_tokens_seen_excluding_padding_tokens

Motivation

I was puzzling over why the logged number of input tokens was so much higher than the number of tokens in my dataset and then I discovered that the logged number of input tokens doesn't exclude padding tokens.

Your contribution

If you tell me which solution you prefer, I might be able to draft a PR :)

[RylanSchaeffer]

Related Forum Post: https://discuss.huggingface.co/t/track-include-num-input-tokens-seen-in-trainer/101138

[PrathmeshAdsod]

Introducing a new parameter (Boolean) in TrainingArgument will be a better choice according to me. It's simple and clear to the developer about what they want. Also it is good for backward compatibility. If you are not working on this issue yet, I will be happy to work on this. It's completely fine if you want to handle it yourself; just let me know.

[RylanSchaeffer]

Go for it!

[PrathmeshAdsod]

Thanks for the green light!

Before I create a PR, I have a query. According to my understanding, I see two backward-compatible options:

1. Add a new parameter

• Add TrainingArguments.count_input_tokens_excluding_padding (default False).
• When True, count tokens via attention_mask.sum (i.e., “without_pad”); when False, keep the current behavior (raw element count including padding = “with_pad”).
• Pros: Straightforward and Clear, minimal risk.

2. Reuse the existing parameter with modes

• Keep include_num_input_tokens_seen as-is for booleans:
  • True → counting “with_pad”
  • False → counting off
• Also accept string modes to enable “without_pad”:
  • "with_pad" → same as True (count raw elements, includes padding)
  • "without_pad" → count via attention_mask.sum (exclude padding)
• Pros: Single already present parameter for both;
• Cons: the accepted type becomes bool OR str, which may be a problem, not sure and need to check properly.

Do you have a preference (new parameter vs. extending the existing one with modes)? What do you guys think on this @ArthurZucker @SunMarc

[RylanSchaeffer]

I think (2) makes more sense.