中文 ｜ English

🖥️ Interactive Demo  |   🌐 Official Website  |   📄 Technical Report (Coming Soon)

• Model Introduction
• Key Features
• Usage
  • Environment Setup
  • Model Assets
  • Inference
    • Inference with Low-Mem GPU
    • Inference with Diffusers
  • Interactive Demo
• LoRA Fine-Tuning
  • SFT Data Preparation
  • Offline Sequence Packing
  • Training and Merging
• Contributing
• Security
• License
• Contact Us

Hy-Nomas is a free-form interactive video world generation model. It supports free-form multi-turn text prompt control or camera control. This repository provides the training, inference, interactive demo, and data annotation code.

• Segment-based generation: Autoregressive segment-wise world generation controlled by either segment level prompt or camera control signal, maintaining a lifelong KV cache for consistency.
• Free-form Prompt Control: Control the world generation with any kind of free form user prompt to generate imaginary object, style, scene, and actions.
• Camera control: Condition generation on camera trajectories alongside text prompts.
• Distilled inference: Inference uses 8 denoising steps and a finetuned PixelUMM decoder for 32-channel DCAE-V latents.
• Interactive demo: A web interface supports generation across multiple turns and decoding at segment boundaries.
• Fine-tuning and data preparation: Lora fine-tuning with customized video dataset, and prepare training data from local videos with our annotation workflow.

Choose one of the following two setup options. We strongly recommend the pre-packaged Docker image, which provides the Python environment, PyTorch, and the required CUDA attention kernels together.

Prepare a Linux host with a compatible NVIDIA driver, Docker, and NVIDIA Container Toolkit. Run these commands from the repository root:

export HY_NOMAS_IMAGE="hynomasoss/hynomasoss:cu130"

docker pull "$HY_NOMAS_IMAGE"
docker run --rm -it --gpus all --ipc=host --network=host \
  -v "$PWD:/workspace/Hy-Nomas" \
  -w /workspace/Hy-Nomas \
  "$HY_NOMAS_IMAGE" bash

Run the remaining commands inside the container, using its pre-installed Python environment. Place model assets under the mounted repository or add a volume mount for your model directory; use the paths visible inside the container in the launch commands below.

Prepare Linux, Python 3.12+, uv, a compatible NVIDIA driver, the CUDA 13.0 Toolkit (nvcc), and a C++ compiler. Run the following from the repository root:

uv sync --extra all
source .venv/bin/activate

The all extra includes PyTorch 2.10.0 (CUDA 13.0), FlashAttention 2, FlashAttention 3, and MagiAttention. pyproject.toml configures the PyTorch CUDA index, the official FA3 source, and disables build isolation for the attention kernels so their builds can use the environment's PyTorch. Kernel compilation may take some time.

The model repository contains DiT generation models, LoRA adapters, video autoencoders and decoders, and Nomas-PE for online prompt enhancement.

Download all model weights from tencent/Hy-Nomas on Hugging Face, preserving the repository directory structure:

export Hy_Nomas_Model_Path=/path/to/Hy-Nomas
hf download tencent/Hy-Nomas --local-dir "${Hy_Nomas_Model_Path}"

Set Hy_Nomas_Model_Path to your download directory once. The commands below reuse its existing value, defaulting to /path/to/Hy-Nomas in a new shell. Click a model name below to view its files on Hugging Face.

Run from the repository root with configs/experiment/inference.yaml, overriding the model paths, prompt file, and output directory:

export Hy_Nomas_Model_Path="${Hy_Nomas_Model_Path:-/path/to/Hy-Nomas}"

bash scripts/inference/inference_from_config.sh configs/experiment/inference.yaml \
  dit_weight="${Hy_Nomas_Model_Path}/DiT/distilled/diffusion_pytorch_model_ema_0_995.safetensors" \
  model.vae.checkpoint="${Hy_Nomas_Model_Path}/VAE/PixelUMM_FT/diffusion_pytorch_model.safetensors" \
  inference.prompt=/path/to/prompts.json \
  inference.save_path=./outputs/distill

Hy-Nomas supports inference on a single NVIDIA GeForce RTX 4090 with configs/experiment/single_gpu_low_mem_inference.yaml. This configuration reduces GPU memory use through:

• INT8 DiT weights and KV cache: Store weights and cached keys/values in INT8 while retaining BF16 computation for DiT linear layers and attention.
• Sliding-window history: Bound the retained context instead of letting the KV cache grow with every segment.
• CPU offload: Move the text encoder, DiT, and VAE onto the GPU for their respective processing stages and return them to CPU afterward.
• Single-GPU kernels: Use FlexAttention for the DiT and CUTLASS FP8 for decoding, with sequence parallelism disabled (sp_size=1).

Use a compatible INT8 DiT checkpoint and FP8 decoder checkpoint. Run on a host or container exposing one GPU to nvidia-smi:

export Hy_Nomas_Model_Path="${Hy_Nomas_Model_Path:-/path/to/Hy-Nomas}"

bash scripts/inference/inference_from_config.sh configs/experiment/single_gpu_low_mem_inference.yaml \
  dit_weight="${Hy_Nomas_Model_Path}/DiT/distilled_int8/diffusion_pytorch_model_ema_0_995_int8.safetensors" \
  model.vae.checkpoint="${Hy_Nomas_Model_Path}/VAE/PixelUMM_FT_FP8/diffusion_pytorch_model_fp8.safetensors" \
  inference.prompt=/path/to/prompts.json \
  inference.save_path=./outputs/single_gpu

CPU offload requires enough system RAM for the models and adds CPU–GPU transfer overhead. Peak GPU memory depends on resolution and segment length.

Upgrade your diffusers to use HyNomasModularPipeline.

pip install -e "diffusers[hy-nomas] @ git+https://github.com/jiaxiangc/diffusers.git@hy-nomas"

Hy-Nomas supports multi-turn interactive generation. This example generates two five-second segments.

import json

import torch
from diffusers import HyNomasModularPipeline
from diffusers.utils import export_to_video

pipe = HyNomasModularPipeline.from_pretrained("tencent/Hy-Nomas-Diffusers")
pipe.load_components(dtype=torch.bfloat16)
pipe.to(device="cuda", dtype=torch.bfloat16)
# Use the FlashAttention and MagiAttention kernels installed with Hy-Nomas.
pipe.transformer.enable_reference_attention(text_backend="flash_varlen", deterministic=True)

prompts = [
    {
        "Style": "Cinematic fantasy", "perspective_view": "Third-person view",
        "subject_list": [{
            "subject_id": "fox", "display_name": "a small orange fox",
            "subject_attribute": "Orange fur, white chest, and a fluffy tail.",
        }],
        "segments": [{
            "visual_description_long": (
                "A small orange fox stands on a stone path in a luminous forest. "
                "Blue fireflies drift between the trees."
            ),
            "contained_actions": [{
                "start_time": "00:00.000", "end_time": "00:05.000",
                "action_description": "The fox walks slowly along the path.",
            }],
        }],
    },
    {
        "segments": [{
            "description_change": "Glowing blue flowers open beside the path as the fox continues forward.",
            "contained_actions": [{
                "start_time": "00:00.000", "end_time": "00:05.000",
                "action_description": "The fox passes the flowers and looks around.",
            }],
        }],
    },
]

videos = pipe(
    prompt=[json.dumps(prompt) for prompt in prompts],
    height=704,
    width=1280,
    num_frames=240,
    segment_latent_lengths=[30, 30],
    fps=24,
    num_inference_steps=4,
    generator=torch.Generator("cpu").manual_seed(42),
    output="videos",
)
export_to_video(videos[0], "hy_nomas_video.mp4", fps=24, quality=None)

The Reactor integration guide provides a streaming service and browser demo with live prompt updates, combined WASD/arrow camera control, and uploaded camera trajectories while retaining world history. The tested profile uses eight B200 GPUs, Docker with NVIDIA Container Toolkit, the Reactor CLI, and Node.js 22.

From this repository root:

cd reactor
reactor build
reactor run --gpus '"device=0,1,2,3,4,5,6,7"' --shm-size 8g --port 8080

Wait for curl -fsS http://localhost:8080/health to report available. In another terminal, from the repository root:

cd reactor/demo
npm ci
npm run dev -- --port 5173

Open http://localhost:5173, use Runtime endpoint http://localhost:8080, press Connect, enter a prompt, then press Start world. Click the video to explore with WASD and arrow keys. Update prompt changes future segments without restarting. See the SSH and WebRTC guidance when the GPU runs remotely.

Try the online interactive demo.

See the demo deployment guide for launching the web gateway and GPU workers, configuring model assets, and running on one or multiple hosts. The demo uses configs/experiment/inference.yaml by default.

Use the captioning-harness to prepare training data from local videos. It annotates each segment, encodes video frames with DCAE-V, and packages the result as an Arrow or Parquet file for SFT. See the guide for serving K3, running annotation and packing, and attaching normalized camera trajectories.

Build a packing plan from the encoded training file. Run from the repository root:

python scripts/data_processing/build_sft_packing_plan.py \
  --input /path/to/train.arrow \
  --output /path/to/train_packing.json \
  --max-seq-len 360000 \
  --num-groups 16 \
  --seed 42

This example creates 16 groups for DP16 × SP8 (128 GPUs). Choose a group count that is a multiple of your data-parallel degree, does not exceed the sample count, and has enough capacity under --max-seq-len. Use the same training file and row order in the training command, and pass the output JSON as dataset.token_packing_sorted_groups_path. See Sequence Packing for details.

LoRA fine-tuning starts from a non-distilled checkpoint and uses segment forcing training pipeline. After training, merge the learned LoRA weights into the corresponding distilled checkpoint to produce a new merged checkpoint for inference.

configs/experiment/sft.yaml enables LoRA with rank/alpha 256, zero dropout, and Muon at a constant learning rate of 1e-4 for 2,000 steps. Only LoRA parameters are trained.

export Hy_Nomas_Model_Path="${Hy_Nomas_Model_Path:-/path/to/Hy-Nomas}"

bash scripts/train/train_from_config.sh configs/experiment/sft.yaml \
  dit_weight="${Hy_Nomas_Model_Path}/DiT/base_ar/diffusion_pytorch_model_fp32.safetensors" \
  model.vae.checkpoint="${Hy_Nomas_Model_Path}/VAE/DCAE-V/dc-ae-v-1.0-f32t4c32-bf16.pt" \
  dataset.data_file=/path/to/train.arrow \
  dataset.token_packing_sorted_groups_path=/path/to/train_packing.json \
  output_dir=./outputs/segment_lora

The training configuration uses SP8; 128 GPUs give DP16 × SP8. Set the launch size for your hardware and build the packing plan for the corresponding data-parallel degree.

Export the saved LoRA checkpoint, then merge it into the distilled checkpoint with matching architecture and parameter shapes:

export Hy_Nomas_Model_Path="${Hy_Nomas_Model_Path:-/path/to/Hy-Nomas}"

python -m scripts.train.export_lora_checkpoint \
  ./outputs/segment_lora/checkpoints/global_step-2000 \
  ./outputs/segment_lora/adapter

python -m scripts.train.merge_lora \
  --base "${Hy_Nomas_Model_Path}/DiT/distilled/diffusion_pytorch_model_ema_0_995.safetensors" \
  --adapter ./outputs/segment_lora/adapter \
  --output ./outputs/merged_model.safetensors

Use dit_weight=./outputs/merged_model.safetensors in the inference command.

This project is licensed under the Apache License 2.0.

We thank the teams behind Gemma 4, Qwen3.8, Kimi-K3, DCAE-V, and MagiAttention for their valuable contributions to the community and the models, research, and tools that support this project.

For questions, non-security bug reports, or suggestions, please open an issue in this repository or contact us at hy_nomas@tencent.com. See CONTRIBUTING.md for contribution guidance. Report suspected vulnerabilities privately using SECURITY.md.

Hy-Nomas · Free-Form Interactive World Generation.