Any recipe for multiGPU run?

[vird]

Seems full offload works. But it uses only 1 GPU

Tue Jan  7 14:59:47 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.28.03              Driver Version: 560.28.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:01:00.0 Off |                  N/A |
| 81%   69C    P2            388W /  390W |   22265MiB /  24576MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:45:00.0 Off |                  N/A |
|  0%   38C    P8             25W /  370W |      18MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        Off |   00000000:C1:00.0 Off |                  N/A |
|  0%   57C    P8             21W /  350W |      18MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        Off |   00000000:C2:00.0 Off |                  N/A |
|  0%   49C    P8             48W /  370W |     133MiB /  24576MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Is there any way to launch it on multiple GPUs? (And probably disable some offloads for better performance)

[Aravind-Adhith]

$(pwd) python cosmos1/models/diffusion/inference/text2world.py
--checkpoint_dir checkpoints
--diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Text2World
--prompt "$PROMPT"
--video_save_name Cosmos-1.0-Diffusion-7B-Text2World_memory_efficient
--offload_tokenizer
--offload_diffusion_transformer
--offload_text_encoder_model
--offload_prompt_upsampler
--offload_guardrail_models

Runs out of memory on a 4090. Could you let me know if you changed anything?

[vird]

Did you miss \ ?

My setup

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

PROMPT="A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves. \
The robot's metallic body gleams under the bright, even lighting, highlighting its futuristic design and intricate joints. \
A glowing blue light emanates from its chest, adding a touch of advanced technology. The background is dominated by rows of boxes, \
suggesting a highly organized storage system. The floor is lined with wooden pallets, enhancing the industrial setting. \
The camera remains static, capturing the robot's poised stance amidst the orderly environment, with a shallow depth of \
field that keeps the focus on the robot while subtly blurring the background for a cinematic effect."

export PYTHONPATH=$(pwd)

/usr/bin/time -v python cosmos1/models/diffusion/inference/text2world.py \
    --checkpoint_dir checkpoints \
    --diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Text2World \
    --prompt "$PROMPT" \
    --video_save_name Cosmos-1.0-Diffusion-7B-Text2World_memory_efficient3debug \
    --offload_tokenizer \
    --offload_diffusion_transformer \
    --offload_text_encoder_model \
    --offload_prompt_upsampler \
    --offload_guardrail_models

[Aravind-Adhith]

No, I did not miss it. This is the error I get.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.25 GiB. GPU 0 has a total capacity of 23.54 GiB of which 1.07 GiB is free. Process 130185 has 22.18 GiB memory in use. Of the allocated memory 21.73 GiB is allocated by PyTorch, and 3.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[vird]

Some differences

1. export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
2. I have multiple GPUs and Xorg is on other GPU

N.b. check nvidia-smi that GPU memory is not occupied by any other processes (18M occupied - seems default)

[ielh1]

Make sure your GPU is not occupied as vird said, the inference commands in the readme works on 24GB (validated multiple times).

[Aravind-Adhith]

Yes, the Xorg was the other process running on this GPU and was using about 116 MiB.

[ethanhe42]

multi-GPU inference is available here: https://github.com/NVIDIA/Cosmos/tree/main/cosmos1/models/diffusion/nemo/inference#run-the-inference-script-with-base-model

[ielh1]

Closing the issue. Feel free to reopen if needed.

[MikeAiJF]

Uninstalling everything, I am unable to perform diffusion 14b inference for 48g, but I can perform autoregressive model 13b inference. May I ask if this setting can perform 14b diffusion inference? Thank you

[foxkw]

multi-GPU inference is available here: https://github.com/NVIDIA/Cosmos/tree/main/cosmos1/models/diffusion/nemo/inference#run-the-inference-script-with-base-model

How about Video2world ? Is there a plan to release multi-gpu inference code for Video2world

[ethanhe42]

@MikeAiJF 14B inference for 48G is challenging.
@foxkw video2world is coming soon.