Mochi OutOfMemory #11454
Description
It was run success(2-3 days ago) but It doesn't run.
I've tryed same command.
Command:
.....,"fps":24,"width":848,"height":480,"num_frames":80,"inference_steps":30
My env:
accelerate 1.2.1
addict 2.4.0
aiofiles 23.2.1
aiosignal 1.3.2
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.9.0
attrs 25.3.0
av 13.1.0
bitsandbytes 0.45.0
certifi 2025.4.26
charset-normalizer 3.4.1
click 8.1.8
decorator 4.4.2
diffusers 0.32.0
einops 0.8.1
exceptiongroup 1.2.2
fastapi 0.115.12
ffmpy 0.5.0
filelock 3.18.0
frozenlist 1.6.0
fsspec 2025.3.2
gradio 5.19.0
gradio_client 1.7.2
h11 0.16.0
httpcore 1.0.9
httpx 0.28.1
huggingface-hub 0.27.0
idna 3.10
imageio 2.36.1
imageio-ffmpeg 0.6.0
importlib_metadata 8.7.0
Jinja2 3.1.6
jsonschema 4.23.0
jsonschema-specifications 2025.4.1
markdown-it-py 3.0.0
MarkupSafe 2.1.5
mdurl 0.1.2
moviepy 1.0.3
mpmath 1.3.0
msgpack 1.1.0
networkx 3.4.2
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-cusparselt-cu12 0.6.2
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.1.105
omegaconf 2.3.0
opencv-python 4.10.0.84
orjson 3.10.16
packaging 25.0
pandas 2.2.3
patch-conv 0.0.1b0
pillow 11.1.0
pip 22.0.2
proglog 0.1.11
protobuf 6.30.2
psutil 7.0.0
py3nvml 0.2.7
pydantic 2.11.3
pydantic_core 2.33.1
pydub 0.25.1
Pygments 2.19.1
python-dateutil 2.9.0.post0
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.2
ray 2.43.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rich 14.0.0
rpds-py 0.24.0
ruff 0.11.7
safehttpx 0.1.6
safetensors 0.4.5
semantic-version 2.10.0
sentencepiece 0.2.0
setuptools 75.8.2
shellingham 1.5.4
six 1.17.0
sniffio 1.3.1
starlette 0.46.2
sympy 1.13.1
tokenizers 0.21.1
tomlkit 0.13.2
torch 2.5.1+cu121
torchvision 0.20.1+cu121
tqdm 4.67.1
transformers 4.47.1
triton 3.1.0
typer 0.15.3
typing_extensions 4.13.2
typing-inspection 0.4.0
tzdata 2025.2
urllib3 2.4.0
uvicorn 0.34.2
websockets 15.0.1
xformers 0.0.28.post3
xmltodict 0.14.2
zipp 3.21.0
GPU: A6000 Ada Generation
How to solve it?
Traceback (most recent call last):
File "/home/usr/genai_workspace/ptenv/workspace/env/socket_host.py", line 127, in
sent_message = load_and_run_task(param_dict, initial=False)
File "/home/usr/genai_workspace/ptenv/workspace/env/socket_host.py", line 70, in load_and_run_task
active_task.run()
File "/home/usr/genai_workspace/ptenv/workspace/env/src/projects/mochi/run.py", line 91, in run
frames = self.pipeline(
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/pipelines/mochi/pipeline_mochi.py", line 739, in call
video = self.vae.decode(latents, return_dict=False)[0]
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_mochi.py", line 992, in decode
decoded = self._decode(z).sample
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_mochi.py", line 963, in _decode
dec, _ = self.decoder(z)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_mochi.py", line 678, in forward
hidden_states, new_conv_cache["block_out"] = self.block_out(
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_mochi.py", line 326, in forward
hidden_states, new_conv_cache[conv_cache_key] = resnet(
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_mochi.py", line 116, in forward
hidden_states = self.norm1(hidden_states)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/usr/genai_workspace/ptenv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_mochi.py", line 64, in forward
output = torch.cat([self.norm_layer(chunk) for chunk in x.split(self.chunk_size, dim=0)], dim=0)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.24 GiB. GPU 0 has a total capacity of 47.51 GiB of which 2.37 GiB is free. Including non-PyTorch memory, this process has 45.13 GiB memory in use. Of the allocated memory 38.92 GiB is allocated by PyTorch, and 5.71 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)