[GPT-OSS 1/N] Initial GPT-OSS support for single turn training

[SumanthRH]

What does this PR do?

Initia GPT-OSS support for single turn training.

Support training in mixed precision (BF16) and inference in half precision only at the moment.

Given some quirks in chat templating, the current SkyRLGymGenerator is not compatible with GPT-OSS for multi-turn tasks.

This PR further adds overrides for chat templating with chat_template_kwargs to be used in the agent loop in SkyRLGymGenerator.

For GPT-OSS, we can provide the reasoning effort in the system prompt, so this feature is important.

[gemini-code-assist]

Code Review

This pull request introduces initial support for single-turn training with GPT-OSS models. This is achieved by adding a chat_template_kwargs configuration to pass specific arguments to the tokenizer's chat template, which is necessary for GPT-OSS. The changes include a new example script for running GSM8K with GPT-OSS, updates to the base configuration, and modifications to SkyRLGymGenerator to utilize the new kwargs.

My review focuses on improving script robustness and code maintainability. I've suggested making the new shell script more robust by adding error handling flags. I've also pointed out an opportunity to refactor the repeated use of the new configuration in SkyRLGymGenerator to improve code clarity and make future changes easier.

[SumanthRH]

Receipts:

Reward curve for gms8k training :

Comparing rewards and generation lengths for reasoning effort high and low:

Example sequence from sequences array after generation (got from dump_data_batch=true) with reasoning_effort=low (default is medium):

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-10-04

Reasoning: low

# Valid channels: analysis, commentary, final. Channel must be included for every message.
Calls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>user<|message|>The selling price of a bicycle that had sold for $220 last year was increased by 15%. What is the new price? Let's think step by step and output the final answer after "####".<|end|><|start|>assistant<|channel|>analysis<|message|>Compute 15% of 220 = 33. So new price = 220+33=253.<|end|><|start|>assistant<|channel|>final<|message|>The price increased by 15% means it is multiplied by \(1 + 0.15 = 1.15\).

\[
220 \times 1.15 = 220 \times \left(\frac{115}{100}\right)
               = 220 \times 1.15
               = 220 + 33
               = 253
\]

The new selling price is **$253**.

####<|return|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[tyler-griggs]

Do you need + since this config param exists?

[SumanthRH]

Yeah....without the plus I get:

Could not override 'generator.chat_template_kwargs'.
To append to your config use +generator.chat_template_kwargs={reasoning_effort:low}
Key 'reasoning_effort' is not in struct
    full_key: generator.chat_template_kwargs.reasoning_effort
    object_type=dict

[tyler-griggs]

oh funny, got it

[tyler-griggs]

thanks :)