refactor(rollout): align vllm_rollout.py with slime sglang_rollout.py

[aoshen02]

Summary

• Mirror slime sglang_rollout.py structure in vllm_rollout.py (984→756 lines, 29→16 top-level defs)
• Remove 13 vime-only helper functions not present in slime — inlined where needed, removed where over-defensive
• Add get_rollout_num_engines() in http_utils.py to mirror slime (replaces inline rollout_num_gpus // rollout_num_gpus_per_engine)
• Add no_stop_trim → include_stop_str_in_output mapping in _build_inference_sampling_params (subsumes PR fix(vllm-rollout): map no_stop_trim to vLLM include_stop_str_in_output #71)
• Remove no-op return_routed_experts from request payload (vLLM is engine-level --enable-return-routed-experts, not per-request like SGLang)
• Remove dead output_token_logprobs construction (update_from_meta_info never reads it)

Removed functions (no slime equivalent)

Function	Lines	Reason
_base_dataset_prompt_ids	13	vime-only partial-continuation budget; slime does not have it
_apply_vllm_routed_experts	41→7	Collapsed to inline block mirroring slime
_inference_generate_tokens_and_logprobs	30→6	Simplified inline
_align_engine_tokens_and_logprobs	13	Over-defensive padding
_warn_if_sampling_filters_may_affect_logprobs	9	Noise warning
_vllm_meta_from_generate_choice	18	Inlined into generate()
_decode_vllm_routed_experts	3	Inlined
_router_worker_urls	9	Inlined into abort()
_resume_vllm_workers	10	Inlined into abort()

Kept (vLLM-specific, no sglang equivalent)

• _coerce_flat_int_token_ids — vLLM /inference/v1/generate requires flat list[int]
• _build_inference_sampling_params — sglang to vllm param translation
• _mm_render_response_to_generate_body + _find_token_subsequence + _align_mm_feature_placeholders_to_tokens — vLLM MM uses render+generate two-step (vs sglang direct image_data)

sglang to vllm API divergences (intentionally NOT mirrored)

• return_routed_experts: SGLang supports per-request, vLLM is engine-level — removed from payload

• routed_experts location: SGLang returns in output meta_info, vLLM returns in output choices[0]

• abort: SGLang uses POST /abort_request, vLLM uses POST /pause?mode=abort + POST /resume

• multimodal: SGLang sends image_data directly, vLLM needs render+generate two-step

  ┌────────────────────────┬────────┬────────┬──────┐
  │ metrics │ without R3 │ R3 │ diff │
  ├────────────────────────┼────────┼────────┼──────┤
  │ eval/aime24 │ 63.3% │ 63.3% │ same │
  ├────────────────────────┼────────┼────────┼──────┤
  │ logprob_abs_diff step0 │ 0.0214 │ 0.0135 │ -37% │
  ├────────────────────────┼────────┼────────┼──────┤
  │ logprob_abs_diff step1 │ 0.0184 │ 0.0114 │ -38% │
  ├────────────────────────┼────────┼────────┼──────┤
  │ logprob_abs_diff step2 │ 0.0160 │ 0.0097 │ -39% │
  └────────────────────────┴────────┴────────┴──────┘

[gemini-code-assist]

Code Review

This pull request refactors the vLLM rollout logic by removing several helper functions, inlining token and logprob parsing, and introducing a helper to determine the number of HTTP engines. However, several issues were identified: the .reshape(...) call on rollout_routed_experts is redundant and risks throwing an error if MoE arguments are missing; removing the list-handling check in generate_and_rm breaks its public API and will cause crashes when a list of samples is passed; and using strict=True in zip without the previous alignment helper risks runtime failures on mismatched token and logprob lengths.

[gemini-code-assist]

The .reshape(...) call is redundant because np.load already restores the original shape of the array from the serialized .npy format. Furthermore, accessing args.num_layers and args.moe_router_topk directly is risky as they may not be defined on args (or may be None), which would cause a runtime AttributeError or TypeError if a user runs a dense model or doesn't pass these arguments.

We should remove the .reshape(...) call to keep it simple and safe.

        sample.rollout_routed_experts = np.ascontiguousarray(arr.astype(np.int32, copy=True))

[gemini-code-assist]

The removal of the initial isinstance(sample, list) check breaks the public API of generate_and_rm, which is typed to accept Sample | list[Sample]. If a list of samples is passed directly to this function, it will now immediately crash with an AttributeError when trying to access sample.response_length on line 421.

We should restore the recursive list handling at the beginning of the function to maintain backward compatibility and adhere to the function's type signature.

    if isinstance(sample, list):
        return await asyncio.gather(
            *[generate_and_rm(args, s, sampling_params, evaluation=evaluation) for s in sample]
        )

    # mask previous off-policy generation for partial rollout
    if args.partial_rollout and args.mask_offpolicy_in_partial_rollout and sample.response_length > 0:
        sample.loss_mask = [0] * sample.response_length

[gemini-code-assist]

Using strict=True in zip is risky here because the alignment helper _align_engine_tokens_and_logprobs was removed. If there is any discrepancy in the lengths of new_response_log_probs and new_response_tokens returned by the vLLM API, this will raise a ValueError and crash the rollout process.

To make the code more robust against API discrepancies, we should avoid strict=True or handle length mismatches gracefully.

Suggested change

	if new_response_tokens:
	meta["output_token_logprobs"] = [
	[float(lp_val), int(tid)] for lp_val, tid in zip(new_response_log_probs, new_response_tokens, strict=True)
	]
	if new_response_tokens:
	meta["output_token_logprobs"] = [
	[float(lp_val), int(tid)] for lp_val, tid in zip(new_response_log_probs, new_response_tokens)
	]

[CalvinXKY]

LGTM