[QUESTION] Why is reset_attention_mask=False by default?

[dtamayo-nlp]

Your question

When we want to make a training in LLMs with a lot of corpora, I understand that the usual approach is to introduce the documents with the following format:
[doc 1] <sep> [doc 2] <sep> ...
Until the context length is full. However, the intuitive way of optimizing that I see is using something that you call reset_attention_mask and you have implemented here.

What I did not expect is to find this attribute as False in most yaml configurations of open models. Examples:

• https://github.com/NVIDIA/Megatron-LM/blob/main/examples/gpt3/gpt_config.yaml
• https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml
• https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_llama_config.yaml

While I understand that there might be some potential benefits to not using masking, I don't trivially see why it should be the default approach. I haven't found much on the internet on this topic, any information would be welcome!

[Advait251206]

reset_attention_mask=False is the default in many Megatron-LM and NeMo configurations primarily because it matches the training recipe used for most large-scale GPT-style pretraining and avoids the additional overhead of constructing per-document attention masks.

Here's the intuition behind the trade-offs:

Why is it False by default?

When reset_attention_mask=False, all tokens in a packed sequence can attend to previous tokens in the same training sample, even if those tokens belong to different documents:

[doc1] <eod> [doc2] <eod> [doc3]

The <eod> (or <sep>) token serves as a boundary marker, but it does not prevent cross-document attention.

This approach has several practical advantages:

• It matches the training methodology used by many GPT-family models.
• It avoids generating different attention masks for each packed sample, simplifying the data pipeline.
• It enables highly efficient fixed-length packing, maximizing GPU utilization.
• In practice, the amount of information leakage across document boundaries is relatively small because document boundaries occur infrequently compared to the total number of tokens seen during pretraining.

What does reset_attention_mask=True do?

With reset_attention_mask=True, attention is blocked across document boundaries.

Conceptually, the attention mask becomes:

doc1 → can attend only within doc1
doc2 → can attend only within doc2
doc3 → can attend only within doc3

This more closely matches the assumption that documents are independent samples.

Potential benefits include:

• Preventing information leakage across unrelated documents.
• Cleaner training semantics when documents are truly independent.
• Potentially better behavior for datasets consisting of many short, unrelated documents.

Why isn't it always enabled?

There are several reasons:

• It requires constructing more complex attention masks for each packed sequence.
• It introduces a small amount of additional computation and memory overhead.
• Historically, large GPT-style models achieved excellent performance without resetting the attention mask, making it the established default.
• Keeping the default unchanged also ensures reproducibility with previously published checkpoints and training recipes.

Should you enable it?

It depends on your dataset:

• Large-scale web pretraining (e.g., Common Crawl, C4, The Pile): reset_attention_mask=False is the common choice and aligns with most published Megatron-LM training recipes.
• Fine-tuning or training on many short, independent documents: reset_attention_mask=True may be worth considering to prevent cross-document attention.
• Research comparing packing strategies: It's a useful ablation to evaluate, as the impact can depend on document lengths and packing efficiency.

So, the default of False is less about it being theoretically superior and more about historical precedent, computational efficiency, and consistency with the training procedures used for many existing GPT-style models.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206