Why have [PAD] tokens in the masked spans?

[bminixhofer]

Hi, I was wondering what's the rationale for having [PAD] tokens in masked spans of length more than one instead of just removing the remaining tokens? Here:

splinter/pretraining/masking.py

Lines 316 to 323 in 1df4c13

	new_tokens[j] = "[QUESTION]"
	masked_spans.append(MaskedSpanInstance(index=j,
	begin_label=unmasked_span_beginning,
	end_label=unmasked_span_ending))
	num_predictions += 1
	else:
	new_tokens[j] = "[PAD]"
	input_mask[j] = 0

Is the reason just computational efficiency?

[oriram]

Hi @bminixhofer, thanks for expressing your interest in Splinter :)
The main reason to do that is to keep the same span start and end indices. Otherwise, masking the span changes the indices of all other spans.
Note that these pad tokens aren't attended to, and that position embeddings are only calculated w.r.t "valid" (non-pad) tokens, so overall this implementation is equivalent to removing them.
Hope my answer is clear :)

[bminixhofer]

Thanks for the quick answer! I didn't know tokens which are not attended to are ignored by positional embeddings, it makes sense then.

[oriram]

It's not by default, I implemented it this way for this reason :) @bminixhofer