Fix structured_outputs handling and tool normalization in vLLM backend

[ehofm]

Summary

This PR adds support for JSON Schema-based structured generation in the vLLM backend and improves the handling of empty tool lists in the GRPOTrainer.

Key Features:

• JSON Schema Support: Enables passing full JSON schemas (e.g., from Pydantic models) through generation_kwargs when using vLLM-serve.
• Robust Parameter Handling: trl/scripts/vllm_serve.py now automatically converts dictionary-based structured_outputs into the required StructuredOutputsParams objects.
• Empty Tool Normalization: Updated VLLMClient and GRPOTrainer to treat empty tool lists as None, preventing NotImplementedError when tools are initialized but not used.

Usage Instructions

Using JSON Schemas (New)

You can now use Pydantic models to define structured outputs for vLLM-backed generation:

from pydantic import BaseModel
from trl import GRPOConfig, GRPOTrainer

# 1. Define your schema as a Pydantic model
class ReasoningResponse(BaseModel):
    reasoning: str
    answer: float

# 2. Convert to JSON schema dictionary
my_schema = ReasoningResponse.model_json_schema()

# 3. Pass the schema to GRPOConfig via generation_kwargs
training_args = GRPOConfig(
    output_dir="./output",
    use_vllm=True,
    generation_kwargs={
        "structured_outputs": {"json": my_schema}
    }
)

trainer = GRPOTrainer(
    model="...",
    args=training_args,
    train_dataset=dataset,
    reward_funcs=[...]
)

Fixes # 5154

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[qgallouedec]

thanks!

[qgallouedec]

@codex review

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 62e03f23a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[qgallouedec]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9efdfce628

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[qgallouedec]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".