Can fixed-window LM packing, degrade long term context following

[Rakshith12-pixel]

Maybe my question doesn't make much sense but
Since we have a fixed block size, it can cut the untokenized English sequence mid word (since the tokenizer is subword).
i.e - x,y where decoded y is the continuation of an English word.

Since the English sentence is cut midway can it affect overall long term context maintaining?

The other approach would be to feed tokens, which are full English sentences separated by fullstop so that context is maintained.

I know that either way the model is trained to complete next tokens which are (usually) sub word. but can the uncut English sentence help in anyway?

Just curious, thanks

[Charlie284]

The important distinction is that nanochat never cuts inside a token. BPE runs first, so the training data is already a sequence of atomic token IDs. A boundary can fall between two subword tokens, but there is no partially represented token.

Also, the current nanochat loader does not carry the remainder of a cropped document into the next row. It uses BOS-aligned best-fit packing: each document is tokenized with a prepended BOS token, and when no complete document fits, it takes only the prefix that fills the row. The unused tail is discarded rather than becoming a contextless continuation at the start of the next row.

So if the crop happens in the middle of an English word, the model still gets valid next-token examples up to that boundary. It simply receives no training target for what came after the boundary. That single boundary does not teach a false continuation.

What does limit long-range context is the window length itself. With T = 2048, the model cannot learn a dependency whose endpoints never appear together in one training row. Ending every row at a full stop would not fix that; it would mostly add padding or discard more tokens, and it would bias training windows toward sentence boundaries.

If preserving long-document dependencies is the goal, better options are a larger T or document-aware overlapping/random windows. For nanochat specifically, the current tradeoff is intentional: every row starts at a document BOS and sees a complete document prefix, at the cost of discarding about 35% of tokens at T=2048 (loader notes).