Skip to content

v0.10.1.dev274

@mikasenghaas mikasenghaas tagged this 20 Nov 13:57
## Summary

- Add `examples/advanced/qwen3.6-35b-a3b/`, an agentic RL example for
`Qwen3.6-35B-A3B` at 256k context. Compose the base with a GPU overlay:
  ```
uv run rl @ examples/advanced/qwen3.6-35b-a3b/rl.toml @
examples/advanced/qwen3.6-35b-a3b/gpu/h200.toml
  ```
- `rl.toml` (base): train on `scaleswe`, eval on SWE-bench Verified
every 25 steps, bash harness. Batch 256 × group 8, `max_staleness = 16`,
NCCL weight broadcast, text-only vLLM with prefix caching.
- `gpu/h200.toml`: 1 trainer node (ep 8, cp 4, optimizer offload, no
activation offload) plus 3 one-node DEP8 inference replicas (dp 8 with
expert parallelism), as in vLLM's [H200 DEP
recipe](https://recipes.vllm.ai/Qwen/Qwen3.6-35B-A3B?hardware=h200&strategy=single_node_dep).
Without EP, vLLM shards each 512-wide expert 8 ways with TP.
- `examples/README.md` points at the new example.

## Verification

Trainer sweep on dummy weights, one node, one 262144-token sample per
micro batch:

| trainer variant (ep 8) | tok/s at 256k | peak GiB |
|---|---|---|
| cp 1 | OOM | – |
| cp 2, no AC offload | 25,895 | 128.4 (too tight) |
| **cp 4, no AC offload** | **25,573** | **87.4** |
| cp 4 | 24,182 | 79.4 |
| cp 8 | 23,764 | 93.2 |

A 5-step smoke run on real weights passed: one trainer node + one
inference node, scaleswe, batch 64, AdamW at lr 1e-6 (the trainer
default).
- Reward went from 0.45 to 0.59. Mismatch KL stayed at 0.0011–0.0013. No
truncation, timeouts or preemptions.
- Trainer, steps 2–5: 21.7k tok/s (2.7k per GPU). Samples averaged 27.4k
tokens.
- Peak memory was 114.0 GiB at step 1 and about 95.5 GiB after. Real
routing is uneven (EP imbalance ~1.3), which raised memory above the
balanced-routing sweep.
- The trainer computed 27% of the time and waited for batches the rest.
So the overlay sets 3 inference replicas. The smoke run used 1 replica;
the 3-replica setting was not run.
- Inference: 914 output tok/s and 21.4k uncached prefill tok/s per GPU,
with a 74% prefix-cache hit rate. KV cache was at most 73% full with up
to 1,019 running requests.
- I cancelled the job during the final SWE-bench Verified eval, after
the trainer had finished step 5.
- The smoke run served inference as tp 2 × dp 4 without EP. The DEP8
setting in the overlay was not run.

The smoke run used the same settings, split as a recipe plus a task
config. Earlier commits on this branch keep that `recipes/` layout and
its fitting skill for reference.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Adds example TOML configs and README documentation only; no runtime or
library code paths are modified.
> 
> **Overview**
> Adds **`examples/advanced/qwen3.6-35b-a3b/`**, a composable RL launch
pattern (base **`rl.toml`** + **`gpu/h200.toml`**) for agentic SWE
training at **256k** context.
> 
> The base config trains on **`scaleswe`** with a bash harness,
evaluates **SWE-bench Verified** every 25 steps, and enables text-only
vLLM (no vision) with prefix caching and NCCL weight broadcast. The H200
overlay targets **1 trainer node** (`ep = 8`, `cp = 4`) and **3 one-node
inference replicas** using vLLM **DEP8** (`data_parallel_size = 8`,
expert parallel) to reduce trainer batch wait time.
> 
> **`examples/README.md`** documents the new advanced example alongside
the other frontier-model folders.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
d0bc049b836e1297953627fb9fd1438df46d98fb. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Assets 2
Loading