to_hf.py + PromptFormatter: produce vLLM-ready SpeechLM checkpoints with backbone-native chat_template

[DongjiGao]

Background

Reopens #15617 after its base branch (speechlm2-with-nemo-automodel-merge) was deleted; now rebased on main. Incorporates @pzelasko's review comments from the closed thread.

Summary

Produces a single checkpoint artifact that serves both HF / NeMo SALM inference and vLLM (SpeechLM plugin) without an opt-in flag, and eliminates the drift between our hand-rolled jinja and the backbone LLM's canonical chat_template.

Commit 1 - PromptFormatter uses backbone chat_template as source of truth

Replaces the hardcoded TEMPLATE dict + to_jinja() reconstruction in QwenPromptFormatter and NemotronNanoV3PromptFormatter with a shared base helper that delegates to tokenizer.apply_chat_template on the backbone's native jinja. Training and exported-ckpt inference now share one jinja string shipped by the backbone's HF repo - no more train/infer drift from reconstruction mismatches.

• formatter.py: add AUDIO_TOKEN, _turn_to_openai_message, _is_reasoning_divergence, _encode_dialog_from_chat_template. Drop the abstract to_jinja stub.
• qwen.py: QwenPromptFormatter.encode_dialog delegates; hardcoded TEMPLATE, INFERENCE_PREFIX and to_jinja removed. Qwen3PromptFormatter (reasoning, out of refactor scope) unchanged.
• nemotron_nano_v3.py: encode_dialog delegates and forwards enable_thinking to the backbone jinja. Default is False; reasoning training raises a clear 'not supported' error because the prefix-based loss mask can't align with the template's fresh <think> generation prompt.

Commit 2 - to_hf.py unconditionally produces vLLM-ready checkpoints

• Removes the opt-in vllm: bool = False flag; every export is vLLM-ready (addresses @pzelasko's review comment on to_hf.py: add --vllm flag for vLLM-ready checkpoint export #15617).
• _detect_vllm_architecture now raises ValueError on missing architectures instead of silently defaulting (addresses the 'broad except' review comment).
• _try_prepare_for_vllm wrapper downgrades a ValueError to a warning so legacy SALM callers loading the same dir still get a valid HF-only checkpoint.
• Tokenizer saved from the backbone (brings its native chat_template) + augmented with <|audio|>. extra_special_tokens normalized to dict (HF loaders crash on the list form transformers writes for additional_special_tokens).
• For reasoning backbones (nemotron-nano-v3), the exported chat_template's enable_thinking default is flipped to False so vLLM's request-time render matches training-time render; otherwise vLLM silently prepends <think> and WER regresses from 1.57 to 5.92.
• hf_hub.py: setdefault('model_type', ...) and setdefault('architectures', [...]) in HFHubMixin.save_pretrained so NeMo-saved checkpoints carry the metadata vLLM / transformers need.

Architecture mapping

LLM backbone	Architecture name
Hybrid (NemotronH)	NeMoSpeechLMHybridForConditionalGeneration
Standard (Qwen3, etc.)	NeMoSpeechLMForConditionalGeneration

Usage

torchrun --nproc-per-node=8 --nnodes=4 \
    examples/speechlm2/to_hf.py \
    class_path=nemo.collections.speechlm2.models.salm_automodel.SALMAutomodel \
    ckpt_path=/path/to/checkpoint.ckpt \
    ckpt_config=/path/to/exp_config.yaml \
    output_dir=/path/to/hf_output

No vllm=True flag any more - output is always vLLM-ready.

Test plan

• Parity test against backbone jinja for Qwen2.5-7B-Instruct and Nemotron-3-Nano-30B-A3B-BF16; rendered text byte-identical to the previous Python + to_jinja pipeline when enable_thinking=False.
• Mask-construction invariants on training turns: len(context) + len(answer) == len(input) and sum(mask) == len(answer) across Qwen / Nemotron, plain / multi-turn, with / without system, multimodal list content.
• End-to-end vLLM librispeech-pc eval on re-exported nemotron-nano-v3 ckpt: WER 1.57 (== baseline). Without the enable_thinking default flip, WER regressed to 5.92 - regression confirmed, fix verified.
• End-to-end NeMo SALM librispeech-pc eval on same ckpt: WER 1.62, 0 empty hypotheses (previous pipeline had 75 empties and 5.50 WER-all).
• Backward compat: legacy SALM caller exports fine -- _try_prepare_for_vllm downgrades missing-prereq ValueError to a warning.
• CI: add 'Run CICD' label.

Made with Cursor

[pzelasko]

I like your thinking about using chat template directly but for the reasons below I don't think it is feasible in the current shape.

Let's approach this from two angles.

1. In this PR don't add chat template. Revert the changes from nemotron and qwen prompt formatter. Then fix Nemotron...PromptFormatter so that it's logic is equivalent to that of jinja, but using PromptFormatter.
2. We can add ChatTemplatePromptFormatter in a separate PR which would be auto-selected when prompt_format: "tokenizer_chat_template" and make it generic for HF models.

[pzelasko]

This PR also needs unit test coverage of the changes.

[DongjiGao]

I like your thinking about using chat template directly but for the reasons below I don't think it is feasible in the current shape.

Let's approach this from two angles.

1. In this PR don't add chat template. Revert the changes from nemotron and qwen prompt formatter. Then fix Nemotron...PromptFormatter so that it's logic is equivalent to that of jinja, but using PromptFormatter.
2. We can add ChatTemplatePromptFormatter in a separate PR which would be auto-selected when prompt_format: "tokenizer_chat_template" and make it generic for HF models.

Could you please explain a bit why it is not feasible in current shape? Thanks!

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pzelasko]

LGTM