[Fix] Supervise GLM-5.2 assistant stop tokens

[jayhenry]

Summary

• supervise the GLM-5.2 stop token at every loss-enabled assistant boundary
• preserve the official GLM-5.2 multi-turn and tool-call serialization
• append <|endoftext|> only when no following role token can terminate the assistant turn
• keep loss=False, thinking cleanup, the slow prefix-diff oracle, and the fast tokenizer consistent
• add regression coverage for all three generation stop tokens

GLM-5.2 chat-template semantics

The official GLM-5.2 chat template does not append a dedicated end-of-turn token after an intermediate assistant message. Instead, the role token beginning the next message is also a configured generation stop token:

Token	ID	Boundary
`<	endoftext	>`
`<	user	>`
`<	observation	>`

All three IDs are listed in the official GLM-5.2 eos_token_id generation configuration.

Previously, XTuner masked <|user|> and <|observation|> and did not add <|endoftext|> after the final assistant message. Consequently, an SFT sample could supervise the assistant body without supervising any token that tells the model to stop.

This PR assigns the loss of the preceding assistant to the applicable user/observation role token. If there is no such next-role boundary, it appends and supervises <|endoftext|>. The following user or tool payload remains masked. An assistant with loss=False also keeps its stop token masked.

Why not append <|endoftext|> after every assistant?

That simpler approach would change training histories to:

assistant answer<|endoftext|><|user|>next question
assistant tool call<|endoftext|><|observation|>tool result

Inference engines such as vLLM normally rebuild chat history with the model's official chat template, which produces:

assistant answer<|user|>next question
assistant tool call<|observation|>tool result

vLLM can still stop the current generation when the model emits <|endoftext|>, but subsequent multi-turn and tool-call prompts would omit the intermediate EOS seen during SFT. This creates an unnecessary train/serve format mismatch and may degrade multi-turn or tool-call behavior. Using the official role boundary as the supervised stop target avoids that mismatch.

Tests

GLM5_2_TINY_MOE_PATH=... python -m pytest -q tests/datasets/test_glm52_openai_tokenize_fn.py
6 passed

The tests cover normal multi-turn conversations, reasoning preservation/cleanup, tool calls, loss=False, generation prompts, default system messages, fast/slow parity, and the label state of all three GLM-5.2 stop tokens.