Skip to content

feat(microduck): 训练基建对齐上游(env RNG 播种 / init_at_random_ep_len / 规模与 seed) - #1462

Merged
TATP-233 merged 2 commits into
dev/issue-1452-microduck-rl-alignmentfrom
feat/issue-1456-microduck-infra-alignment
Sep 3, 2026
Merged

feat(microduck): 训练基建对齐上游(env RNG 播种 / init_at_random_ep_len / 规模与 seed)#1462
TATP-233 merged 2 commits into
dev/issue-1452-microduck-rl-alignmentfrom
feat/issue-1456-microduck-infra-alignment

Conversation

@TATP-233

@TATP-233 TATP-233 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #1456 | Parent roadmap: #1452 | Base: dev/issue-1452-microduck-rl-alignment | 依赖:#1461(已合入;本分支已同步集成分支)

交付内容

  1. env 级 RNG 播种:链路本就存在(YAML env:BackendAdapter.build_task_env_cfg_overrideManagerBasedRlEnvCfg.seedenv.rng,command/noise/delay/DR 全部消费它),缺的只是 YAML 设值。三个 microduck mjwarp owner YAML 设 env.seed: 42(上游默认);默认 None→随机行为不变(mujoco owner 测试仍断言 None)。新增测试:同 seed 两次建 env 的 command 采样序列一致、不同 seed 不同。
  2. init_at_random_ep_len 真实生效:上游核实 mjlab 的 wrapper episode_length_buf setter 直接写 env 内计数(mjlab/rl/vecenv_wrapper.py:54-59)且 train.py:174 确实传 True——上游错峰生效,故 UniLab 修适配层对齐而非关闭:RslRlVecEnvWrapper.episode_length_buf 改 property+setter,传播进 env 新增的冷路径 set_episode_length_buf(同步写 episode_length_bufstate.info["steps"],形状/非负校验);env 无此方法时只更新本地记账(向后兼容)。附带收益:him_ppo/runner.py:119 的同类赋值同步生效。
  3. 规模对齐:三任务 mjwarp owner algo.num_envs 2048→4096、algo.seed 1→42、max_iterations 500→2000(2000×24=48000 env step 恰好覆盖全部 curriculum 终档;上游为 50000,最终预算由 microduck 对齐 child 5/5: 受控对比验收 benchmark 与统计一致性报告 #1457 对比实验决策)。
  4. contract 翻绿infra.num_envsinfra.seed 翻 match,新增 infra.env_seed 守卫;audit GAP 0(velocity;NOTE 剩 max_iterations 预算与 mujoco-warp 版本两个非判定项)。

用户可见行为变化

microduck 三任务(ppo tree, mjwarp)默认并行规模与随机种子变化,训练 run 变为可复现(env RNG 播种后)。其他任务/后端行为不变(opt-in)。macOS/Linux 无差异。

Validation

  • uv run pytest tests/envs/locomotion/microduck/ -m "" -q:40 passed(含新增 test_infra_alignment.py 7 条:seed 复现、错峰传播后剩余长度非全同且 timeout 错峰触发)。
  • tests/envs/ tests/algos/ -k "seed or episode_length or random_ep":12 passed;test_rsl_rl_ppo.py 12 passed。
  • uv run scripts/audit_microduck_alignment.py:MATCH 184 / GAP 0 / MISMATCH 0。
  • mjwarp smoke(64 envs × 200 steps):NaN=0。
  • 改动文件 ruff / mypy 干净。
  • 最终 head(含集成分支同步 merge)本地 make test-all:2551 passed, 28 skipped, 1 xfailed;pre-commit 与 benchmark 检查全部通过。

Base 非 main,按 gate 规则使用本地验证 + review。

- thread env-level RNG seed through the three microduck mjwarp owners
  (env.seed=42): the Hydra -> BackendAdapter -> registry chain already
  reaches ManagerBasedRlEnvCfg.seed, so command/noise/delay/DR sampling is
  now reproducible across runs; unset owners keep the None default
- fix init_at_random_ep_len: RslRlVecEnvWrapper.episode_length_buf becomes
  a property whose setter propagates into the env's real episode counters
  via the new ManagerBasedRlEnv.set_episode_length_buf cold-path entry
  (keeps episode_length_buf and state.info["steps"] in sync); upstream
  mjlab's wrapper setter writes the env buffer directly, so staggering is
  effective upstream and now matches here
- scale alignment: algo.num_envs 2048 -> 4096 and algo.seed 1 -> 42 in all
  three mjwarp owners; max_iterations 500 -> 2000 (2000 x 24 = 48000 env
  steps, exactly covering every curriculum terminal stage; final budget
  remains a child 5 decision)
- flip infra.num_envs / infra.seed contract entries to match and add an
  infra.env_seed match entry guarding the new YAML field
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant