Regarding attribute "internal_attn_scores"

[cece00]

Hi!

Thanks for the code.
I'm currently trying to run this project, and I noticed something in the utils.py file (lines 2900 and 3031).

If the "internal_attn_scores" parameter is set to True by default, it seems that the Lookback-Lens ratio calculation is skipped—is that correct? I couldn't find any mention of this parameter or changes to its default behavior in the README.md. Could you clarify how this is intended to be configured?

Should I set "internal_attn_scores" as False to use the lookback-lens ratio during generation?

Thanks, best.

[voidism]

Hi Siya,

The calculation is not skipped. If the internal_attn_scores is set to True, the attn scores will be calculated internally in the forward pass of the transformer model. Please see these lines:

Lookback-Lens/transformers-4.32.0/src/transformers/models/llama/modeling_llama.py

Lines 534 to 541 in f2846ef

	if output_attentions:
	if context_length is not None:
	attn_on_context = self_attn_weights[0, :, -1, :context_length].mean(-1).detach().cpu()
	attn_on_new_tokens = self_attn_weights[0, :, -1, context_length:].mean(-1).detach().cpu()
	attn_scores = attn_on_context / (attn_on_context + attn_on_new_tokens)
	outputs += (attn_scores,)
	else:
	outputs += (self_attn_weights,)

Setting internal_attn_scores to either True or False should get the same results. But if we set it to False, it will always return the full attention matrices from all the layers/heads to calculate the attn_scores after the forward pass, and sometimes it gets cuda out of memory error. That’s why we need internal_attn_scores to compute the same thing within each layer and return only the scores, not the full attention matrices.