Release 2026-07-29 (i4 44acef03) - #143
Merged
Merged
Conversation
_source_commit: 44acef0355e345c8ea0800804955d93da16a578e-dirty _dest_commit: dcb3761
lfengad
approved these changes
Jul 29, 2026
pengcuo
approved these changes
Jul 29, 2026
erichudev
pushed a commit
to KunlunxinAD/cosmos-framework
that referenced
this pull request
Aug 6, 2026
* fix(action): validate nano DROID policy SFT TOML against the schema (NVIDIA#138) (NVIDIA#139) ## Summary The `action_policy_droid_nano` post-training config could not be loaded via the documented `--sft-toml` entry point: `examples/toml/sft_config/action_policy_droid_nano.toml` carried `[model.rectified_flow_training_config]` and `[dataloader_train.dataloader]` blocks that `SFTExperimentConfig` (`extra="forbid"`) does not model, so `load_experiment_from_toml` raised `ValidationError`. Addresses NVIDIA#138. ## Fix (mirrors the edge recipe) Keep these values in the registered recipe instead of the TOML: - Set the inner dataloader scalars (`batch_size=16` / `num_workers=16` / `prefetch_factor=2`, the reference values) in `action_policy_droid_nano`. - Drop both blocks from the TOML (`loss_scale=10` was already set in the recipe). The TOML now contains only schema-registered blocks, so it validates with no schema change — consistent with how `action_policy_droid_edge` already works, and with i4 (which keeps `loss_scale`/dataloader in the experiment config). ## Test `TestExampleTomlConfigs` in `sft_config_test.py` is parametrized over **every** `examples/toml/sft_config/*.toml`, validating each against `SFTExperimentConfig` and building its Hydra override list. The existing `unittest` job in `gpu-tests.yml` runs `pytest cosmos_framework/`, so the new test is picked up automatically (no new workflow needed). Verified locally: all 12 example TOMLs pass validation + override construction with the fix applied. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com> * Add post-training recipe for forward dynamics experiments on Droid (NVIDIA#85) ## Summary Add a complete SFT recipe for training Cosmos3 **forward dynamics** on the [Cosmos3-DROID](https://huggingface.co/datasets/nvidia/Cosmos3-DROID) dataset (LeRobot v3.0 format). - Register `action_fd_droid_posttrain` experiment with `mode="forward_dynamics"`, 10-D ee-pose action space (`midtrain` alias), 16-frame chunks at 480p resolution. - Add `DROIDMergedLeRobotDataset` to load both `success/` and `failure/` splits from a single parent root, without changing the existing single-root `DROIDLeRobotDataset` used by the policy recipe. - Add `get_action_droid_merged_lerobot_sft_dataset()` factory and wire it through `ActionSFTDataset` / `ActionTransformPipeline`. - Ship paired TOML + launch shell (`action_fd_droid_posttrain.toml`, `launch_sft_action_fd_droid_posttrain.sh`) and reproduction docs (`docs/action_fd_droid_posttrain.md`). ## Motivation Cosmos3 supports forward-dynamics inference (`forward_dynamics` mode: predict future video from first frame + action sequence), but there was no public post-training recipe for fine-tuning on Cosmos3-DROID. This PR closes that gap and mirrors the structure of existing SFT recipes (e.g. action policy on DROID/LIBERO). ## Test plan - [ ] Config dry-run passes: ```shell PYTHONPATH=. python -m cosmos_framework.scripts.train \ --sft-toml examples/toml/sft_config/action_fd_droid_posttrain.toml \ --dryrun --------- Co-authored-by: lfengad <liangf@nvidia.com> * Release 2026-07-28 (i4 e2464681) (NVIDIA#141) Forward release from imaginaire4 to cosmos-framework: mapped file moves + import-path rewrites only, no code-level changes. **499 mapped entries — 14 changed, 2 new.** Drift gate clean. New files: `cosmos_framework/model/tokenizer/checkpoint_identity.py` and `cosmos_framework/model/tokenizer/models/latent_denoising.py` — i4 added `checkpoint_identity` (checkpoint provenance resolution/hashing, stdlib + boto3 only), which `tokenizers/uniae/noncausal_4x16x16.py` and `latent_denoising.py` import; it is now part of the release mapping. `.file_mapping.json` is updated as usual — it is the baseline the next release checks against. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Cosmos-Framework Release Bot <cosmos-framework-release-bot@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Release 2026-07-29 (i4 44acef03) (NVIDIA#143) Automated release from i4. _source_commit: `44acef0355e345c8ea0800804955d93da16a578e-dirty` _dest_commit (base): `dcb37610049e0de423586d02e83729df75f816b6` * fix(checkpoints): repin Super 4Step checkpoints to main (NVIDIA#144) ## Problem `distilled-inference-smoke` fails during checkpoint download, before inference ever starts: ``` 403 Forbidden Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/01/58/.../be75606093db... <Error><Code>AccessDenied</Code><Message>Access Denied</Message></Error> subprocess.CalledProcessError: ['uvx', ..., 'hf@1.16.4', 'download', 'nvidia/Cosmos3-Super-Text2Image-4Step', '--revision', '1ba94110bc...'] ``` rank 0 fails → ranks 1-3 raise `RuntimeError: Rank 0 download failed` → `ChildFailedError`. Seen on NVIDIA#134 ([job](https://github.com/NVIDIA/cosmos-framework/actions/runs/30432072419/job/90521649702)), where both the T2I and I2V cases died this way — nothing to do with that PR's changes. ## Root cause The LFS blobs behind the two pinned commits are still served from the legacy `cdn-lfs-us-1.hf.co` bucket, and those objects were reclaimed after the repos migrated to Xet. The same files on each repo's `main` branch serve fine from `us.aws.cdn.hf.co`. Reproduced deterministically with plain `curl` (3/3 attempts): | repo | file | pinned rev | `main` | |---|---|---|---| | `Cosmos3-Super-Text2Image-4Step` | `text_tokenizer/tokenizer.json` | **403** | 200 | | `Cosmos3-Super-Image2Video-4Step` | `transformer/diffusion_pytorch_model-00001-of-00027.safetensors` | **403** | 206 | `tokenizer.json` has a different oid and size across the two revisions (`be7560…`/11422650 vs `aeb133…`/11422654), i.e. the file was rewritten after we pinned and the old blob is gone. `main` CI stays green only because its runner (`computeinstance-e01yhncr…`) has these files cached — the download command returns in 1.2 s without touching the CDN. NVIDIA#134 landed on `computeinstance-e01kqprb…` with a cold cache and hit the real fetch. **Any PR scheduled onto a cold-cache runner fails the same way.** ## Fix Repin both to `revision="main"`, matching the other five Cosmos3 checkpoints in the same registry. ## Scope check Swept every 40-hex `revision=` pin in the repo against the Hub with an `HF_TOKEN` (first 8 LFS files per repo, ranged GET). Only these two are affected: | repo @ pinned rev | result | |---|---| | `Cosmos3-Super-Text2Image-4Step` @ `1ba94110` | **403** | | `Cosmos3-Super-Image2Video-4Step` @ `f85d3335` | **403** | | `Cosmos3-Nano-Reasoner` @ `6406357c` | OK | | `Cosmos3-Super-Reasoner` @ `b9b716f3` | OK | | `Cosmos3-Experimental` @ `6ca42c5d` / `a3743aa1` | OK | | `Cosmos-Guardrail1` @ `d6d4bfa8` | OK | | `Wan2.1-T2V-14B`, `Wan2.2-TI2V-5B` | OK | | `BridgeData2-Subset-Synthetic-Captions`, `LIBERO_LeRobot_v3`, `bridge_lerobot_v3` | OK | `Cosmos3-Edge` already uses `revision="main"` in both `inference/common/checkpoints.py` and `configs/base/experiment/sft/models/edge_model_config.py`. ## Verification ``` $ pytest cosmos_framework/inference/args_test.py -k distilled_checkpoint_uses_published 2 passed $ pytest cosmos_framework/inference/args_test.py -k test_checkpoints 1 passed ``` `test_checkpoints` really downloads `config.json` / `transformer/config.json` for every registered checkpoint, so it confirms both `main` revisions resolve and fetch. ## Follow-up (not in this PR) The comment at `args_test.py:129-131` says the 4-step repos are gated, but the Hub now reports `gated: false, private: false` for both and they fetch anonymously. The `HF_TOKEN` skip branch in `test_checkpoints` looks droppable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add cu130-torch213 dependency group (torch 2.13 on CUDA 13.0) (NVIDIA#134) ## What Adds `cu130-torch213` and `cu130-torch213-train` dependency groups pinning **torch 2.13.0+cu130** (with matching torchvision 0.28.0, torchcodec 0.14.0, torchao 0.17.0, triton 3.7.1, `natten 0.21.6+cu130.torch213` on aarch64, and the cu13 NVIDIA runtime libs), plus a `conflicts` entry making it mutually exclusive with the other backend groups. `uv.lock` is regenerated for the new closure. ## Why the code change torchvision 0.28 removed `torchvision.io.read_video`, so `cosmos_framework/inference/vision.py` switches to a torchcodec-based THWC uint8 decoder. It's forced onto CPU so an active default-CUDA context during generation doesn't route torchcodec's internal frame-index tensor to CUDA (which would raise `NotImplementedError`). ## Docs Bumps the recommended NGC base image to `nvcr.io/nvidia/pytorch:26.06-py3` in `README.md` and `docs/setup.md` to match the torch 2.13 line. ## Verification On 4× GB200 (aarch64): - `uv sync --all-extras --group=cu130-torch213-train` resolves clean; `torch.__version__ == 2.13.0+cu130`, `torch.version.cuda == 13.0`. - Cosmos3-Nano `throughput` inference over **all 13** `inputs/omni/*.json` samples (t2i, t2v, i2v, v2v, t2vs, i2vs, and the 6 action modes) produced valid outputs: videos decode with real pixel variation (std 44–76, well above the degenerate floor), audio modes carry an audio track, and all action arrays are non-empty and finite. ### Known limitations (in code comments) - x86_64 has no torch2.13 wheels yet for flash-attn / TE / natten — those are commented out / left on the torch2.10 build; this line is validated on **aarch64**. - torch 2.13.0+cu130 hard-pins cuDNN to 9.20.0.48; the cuDNN attention backend wants ≥9.22 (else it falls back to NATTEN). Override at runtime if needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com> * fix(inference): skip generation VAE for reasoner-only runs (NVIDIA#132) Reasoner-only CLI inference to no longer instantiate the generation vision tokeniser or resolves the video VAE. Mixed and generative input sets retain the existing eager-loading behaviour. ## Root cause `OmniMoTModel.set_up_tokenizers` instantiated `model.config.tokenizer` unconditionally. For Cosmos3-Edge, that tokeniser resolves `Wan2.2_VAE.pth`, even when inference only produces reasoner text. ## Changes - detect all-reasoner input sets before model construction and apply a narrow `model.config.load_vision_tokenizer=false` override - make the generation vision tokeniser optional only under that override; the default remains enabled - use configured compression factors when no live tokeniser exists, covering network construction and token-budget batching - treat separate vision-decode GPU placement as a no-op when the generation tokenizer is intentionally absent - cover all-reasoner, mixed, empty, default-load, skip-load, batching, and separate-decode cases ## Behaviour - **Reasoner-only inputs:** the VLM processor and checkpoint-local vision tower still load; the generation VAE does not. - **Mixed or generative inputs:** unchanged; the generation tokenizer follows the existing eager path. - **Other entry points:** unchanged by default because `load_vision_tokenizer` defaults to `true`. Fixes NVIDIA#128 --------- Co-authored-by: lfengad <liangf@nvidia.com> * Release 2026-07-31 (i4 e7f82ffe) (NVIDIA#148) Automated release from i4. _source_commit: `e7f82ffe91d21c5ff162efba2b8a78d0f868d8f1-dirty` _dest_commit (base): `5e67049cd94acb667786f1e6dd0dab821cb90c97` --------- Co-authored-by: Cosmos-Framework Release Bot <cosmos-framework-release-bot@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * attention: lower the cuDNN backend minimum to 9.20 (NVIDIA#153) ## Problem `torch==2.13.0+cu130` hard-pins `nvidia-cudnn-cu13==9.20.0.48` — it is an `==` in torch's own wheel METADATA: ``` Requires-Dist: nvidia-cudnn-cu13==9.20.0.48; platform_system == "Linux" ``` So the `cu130-torch213` / `cu130-torch213-train` groups cannot ship a newer cuDNN from `pyproject.toml` (a different version needs a global `override-dependencies`, not a group-scoped pin). Against the previous `CUDNN_MIN_BACKEND_VERSION = 92200` gate, that made `CUDNN_SUPPORTED` **False** on every `cu130-torch213` environment, and all attention silently fell back to NATTEN. The workaround in the `pyproject.toml` note — `uv pip install nvidia-cudnn-cu13==9.25.0.15` at runtime — was the only way to get the backend back. ## Change - `CUDNN_MIN_BACKEND_VERSION`: `92200` → `92000` - Drop the now-stale `pyproject.toml` note and point it at the constant instead No dependency pins changed; `uv lock --check` passes, so `uv.lock` is untouched. ## Verification 4× GB200 (sm_100, aarch64), `cu130-torch213-train` group, `Cosmos3-Nano` over `inputs/omni/*.json`: - **13/13 samples, exit 0, zero errors** — covers the action modes, `t2i` / `t2v` / `i2v` / `v2v`, and the audio-enabled `t2vs` / `i2vs` Pre-flight: ``` cudnn.version() = 92000 <-- exactly at the new gate CUDNN_SUPPORTED = True <-- False under the old 92200 arch_tag = 100 get_backend_list(100) = ['cudnn', 'natten', 'flash2'] ``` cuDNN was confirmed to actually run, at three independent levels: | Level | Config | Result | |---|---|---| | Selector | 1 GPU, eager | `choose_backend` → `cudnn` **8856/8856**; natten 0, flash2 0 | | ATen dispatch | 1 GPU, eager | `aten._scaled_dot_product_cudnn_attention` **3896×**; natten 0, flash 0 | | **GPU kernels** | **4 GPU + torch.compile** | `cudnn_generated_fort_native_sdpa_sm100_flash_fprop_*` **5040×/rank**; **0 NATTEN kernels on any rank** | The third row matters most: it is the same configuration the 13/13 batch ran under, and the name is a device-side cuDNN-generated SDPA kernel, so it is not an artifact of op naming or of inductor's fusion labels. The residual `fmha_cutlassF_bf16_aligned_*` (25–73 calls/rank) is PyTorch's mem-efficient attention reached by a direct `F.scaled_dot_product_attention` call outside the framework's attention frontend — it does not go through `choose_backend` and is two orders of magnitude below the cuDNN count, so it is not a fallback. Conclusion: the previous `>= 9.22` requirement was **conservative rather than a hard block**, at least for inference. ## Not verified Flagging these explicitly rather than letting the numbers above imply more than they show: 1. **Training.** The `cu130-torch213-train` group carries `transformer-engine==2.12.0+cu130.torch210` — built against the torch 2.10 ABI. It imports cleanly under torch 2.13 (no undefined symbols), but its kernels were not exercised. 2. **Numerical equivalence between cuDNN 9.20 and 9.22+.** 13/13 shows every mode runs and produces output; it is not an output-vs-baseline comparison against torch 2.10 or against a newer cuDNN. 3. **Why 92200 was chosen originally.** There is no comment or issue link in the repo recording it, so this change cannot rule out a specific bug it was guarding against. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: LiangHao <hliangac@connect.ust.hk> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com> Co-authored-by: Wei-Cheng Tseng <weicheng.tseng@mail.utoronto.ca> Co-authored-by: yy-code-nv <yangyangt@nvidia.com> Co-authored-by: Cosmos-Framework Release Bot <cosmos-framework-release-bot@nvidia.com> Co-authored-by: pengcuo <pzeren@nvidia.com> Co-authored-by: Chris von Csefalvay <chris@chrisvoncsefalvay.com> Co-authored-by: daichaonan <daichaonan@baidu.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated release from i4.
_source_commit:
44acef0355e345c8ea0800804955d93da16a578e-dirty_dest_commit (base):
dcb37610049e0de423586d02e83729df75f816b6