[Fix][Generator] Use custom_chat_template in each step retokenization

[CharlieFRuan]

Prior to this PR, in the re-tokenize chat history codepath, we did not use self.custom_chat_template in some places (only used it in the final tokenization). Though currently only Qwen3 enters this codepath, which would not be affected, it is still good to apply the custom_chat_template for consistency.

[gemini-code-assist]

Code Review

This pull request correctly applies the custom_chat_template in the re-tokenization step for consistency. While the change is good in isolation, it highlights a critical bug in the agent_loop logic. The initial prompt length is calculated using the default chat template, but the token IDs from which the prompt is later extracted are generated using the custom chat template. This mismatch will lead to incorrect prompt data. I've added a critical review comment with a detailed explanation of the issue and a suggested fix.

[CharlieFRuan]

/gemini review

[gemini-code-assist]

Code Review

This pull request addresses an inconsistency by ensuring custom_chat_template is used during the re-tokenization of chat history. The changes in agent_loop and _get_next_input_ids_by_retokenizing_chat_history correctly apply the custom template when needed. The fix is sound and improves code consistency. I have provided one suggestion to enhance the code style in agent_loop.