How to use DeepFloyd IF

[FurkanGozukara]

The hugging face repo has to many files
Currently I am downloading via git clone but they are over 100 gb
So how to use DeepFloyd IF large model? thank you

also how to set a different port to launch

[FurkanGozukara]

I have cloned the entire repo with git large clone it is over 100 gb :/

what files we need cant it automatically download only necessary files?

@ddPn08 @p1atdev

[ddPn08]

I'm sorry, but I'm temporarily rebuilding the implementation of DeepFloydIF. I will let you know as soon as it is completed.

[FurkanGozukara]

I'm sorry, but I'm temporarily rebuilding the implementation of DeepFloydIF. I will let you know as soon as it is completed.

sad people were expecting a tutorial for this from me :/

but thanks a lot looking forward to that.

so how do we add other models? does it have feature to auto download necessary files?

lets say this repo : https://huggingface.co/dreamlike-art/dreamlike-anime-1.0

[ddPn08]

Click here for instructions on how to add a model.
https://ddpn08.github.io/Radiata/en/usage/model.html

Or you can put ckpt and safetensors files in models/checkpoints.

[FurkanGozukara]

Click here for instructions on how to add a model. https://ddpn08.github.io/Radiata/en/usage/model.html

Or you can put ckpt and safetensors files in models/checkpoints.

nice thanks

any ETA for DeepFlyod ?

[ddPn08]

any ETA for DeepFlyod ?

I'm planning on implementing it this week. Tomorrow at the earliest. It's not that hard of a task.

[ddPn08]

Implemented.
https://ddpn08.github.io/Radiata/en/usage/deepfloyd_if.html

[FurkanGozukara]

Implemented. https://ddpn08.github.io/Radiata/en/usage/deepfloyd_if.html

thank you so much testing now

[FurkanGozukara]

Implemented. https://ddpn08.github.io/Radiata/en/usage/deepfloyd_if.html

looks like automatic install fails on windows

venv "F:\deepfloyd ai\Radiata\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Commit hash: fd66c1b
Installing requirements
Installing tensorrt requirements
Traceback (most recent call last):
File "F:\deepfloyd ai\Radiata\launch.py", line 224, in
prepare_environment()
File "F:\deepfloyd ai\Radiata\launch.py", line 206, in prepare_environment
install_tensorrt()
File "F:\deepfloyd ai\Radiata\launch.py", line 134, in install_tensorrt
libfile_path is not None
AssertionError: Could not find TensorRT. Please check if it is installed correctly.
Press any key to continue . . .

what library we need i will try with activating venv

[FurkanGozukara]

I tried

pip install TensorRT

this is the error below

F:\deepfloyd ai\Radiata\venv\Scripts>activate TensorRT

(venv) F:\deepfloyd ai\Radiata\venv\Scripts>pip install TensorRT
Collecting TensorRT
  Downloading tensorrt-8.6.1.tar.gz (16 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: TensorRT
  DEPRECATION: TensorRT is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for TensorRT ... error
  error: subprocess-exited-with-error

  × Running setup.py install for TensorRT did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      running install
      F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      Looking in indexes: https://pypi.nvidia.com
      ERROR: Could not find a version that satisfies the requirement tensorrt_libs==8.6.1 (from versions: none)
      ERROR: No matching distribution found for tensorrt_libs==8.6.1

      [notice] A new release of pip available: 22.3.1 -> 23.1.2
      [notice] To update, run: python.exe -m pip install --upgrade pip
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\King\AppData\Local\Temp\pip-install-_godouj9\tensorrt_5ca92a1f0036444ebaa7be3529bfd9e9\setup.py", line 49, in <module>
          setup(
        File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
        File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\_distutils\dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\dist.py", line 1217, in run_command
          super().run_command(command)
        File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\setuptools\_distutils\dist.py", line 987, in run_command
          cmd_obj.run()
        File "C:\Users\King\AppData\Local\Temp\pip-install-_godouj9\tensorrt_5ca92a1f0036444ebaa7be3529bfd9e9\setup.py", line 43, in run
          install_dep("{:}_libs".format(tensorrt_module))
        File "C:\Users\King\AppData\Local\Temp\pip-install-_godouj9\tensorrt_5ca92a1f0036444ebaa7be3529bfd9e9\setup.py", line 41, in install_dep
          status.check_returncode()
        File "C:\Python3108\lib\subprocess.py", line 457, in check_returncode
          raise CalledProcessError(self.returncode, self.args, self.stdout,
      subprocess.CalledProcessError: Command '['F:\\deepfloyd ai\\Radiata\\venv\\Scripts\\python.exe', '-m', 'pip', 'install', 'tensorrt_libs==8.6.1', '--index-url', 'https://pypi.nvidia.com']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> TensorRT

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[ddPn08]

If you only use deepfloyd if you don't need to turn on tensorrt.
Try removing --tensorrt from COMMANDLINE_ARGS.

[FurkanGozukara]

If you only use deepfloyd if you don't need to turn on tensorrt. Try removing --tensorrt from COMMANDLINE_ARGS.

i need it for deepfloyd :)

[ddPn08]

Oh sorry. Currently TensorRT is not yet compatible with DeepfloydIF.
There is a possibility that it will be supported by future TensorRT version upgrades, but I do not know yet.

[FurkanGozukara]

wait i am confused
you said web ui now supports deep floyd
does it run on your computer?

[ddPn08]

Yes, radiata supports deepfloydif.
You are probably getting the above error because you have enabled tensorrt mode, another feature of radiata.
So turning off tensorrt mode should solve the error.

[FurkanGozukara]

Yes, radiata supports deepfloydif. You are probably getting the above error because you have enabled tensorrt mode, another feature of radiata. So turning off tensorrt mode should solve the error.

on your documentation it shows this

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--tensorrt

call launch.bat

https://ddpn08.github.io/Radiata/en/usage/deepfloyd_if.html

[ddPn08]

oh sorry understood. This is a mistake in the documentation.

[ddPn08]

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--deepfloyd_if

call launch.bat

Try setting like this.

[FurkanGozukara]

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--deepfloyd_if

call launch.bat

Try setting like this.

started downloading files without it. will test once files downloaded. thank you so much

[ddPn08]

I'm glad. IF has just been implemented and is in an unstable state. It would be helpful if you could contact me if there is a problem.

[FurkanGozukara]

I'm glad. IF has just been implemented and is in an unstable state. It would be helpful if you could contact me if there is a problem.

sure i will hopefully. by the way does it support optimizations? what command line arguments should i use for lower vram?

hopefully I will make a video so i can explain my audience

i am owner of https://www.youtube.com/secourses

we are having 200k+ monthly views mostly generative AI

[ddPn08]

Currently there are 5 modes.
lowvram: Mode for lowest specs. Save memory by removing the pipeline after each stage.
sequential_off_load: This mode is for machines with large main memory but little VRAM. It will work with at least 6GB of VRAM.
medvram: Settings for the middle end.
off_load: Effective when the size of main memory is large. Save VRAM by moving the pipeline to main memory instead of deleting it.
normal: Do not remove the pipeline, keep it in VRAM. It's fast, but it uses a lot of VRAM.

[ddPn08]

Using auto automatically selects the mode from the amount of main memory and VRAM.

[LIU-ZHIHAO]

Using auto automatically selects the mode from the amount of main memory and VRAM.

Use 'auto' model ,I found a problem, after the first two steps to generate images, RAM and VRAM are not released(Always like this until closing the program), resulting in insufficient resources to start the third step:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 22.00 GiB total capacity; 9.46 GiB already allocated; 10.38 GiB free; 9.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

system info:
win11、2080ti（22GB）、32GB RAM、Radiata （ latest version）

[FurkanGozukara]

without this it doesnt work atm

i am testing on windows 10 and python 3.10.9

set CUDA_VISIBLE_DEVICES=0

and with that even with 24 GB vram RTX 3090 i am getting out of memory error :d

looks like there is memory leak or something else missing

**i will make a tutorial for my subscribers and also i will show your web ui

can you fix them?

here errors**

when set as auto and both gpu 0 and gpu 1 visible and testing only the first step - 64 pixels

Downloading shards: 100%|██████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1995.39it/s]
Loading checkpoint shards:   0%|                                                        Loading checkpoint shards:  50%|██████████████████████████████▌                         Loading checkpoint shards: 100%|████████████████████████████████████████████████████████Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████| 2/2 [00:14<00:00,  7.24s/it]
watermarker\diffusion_pytorch_model.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00002-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00001-of-00002.bin, safety_checker/pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Traceback (most recent call last):
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1036, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\utils.py", line 488, in async_iteration
    return next(iterator)
  File "F:\deepfloyd ai\Radiata\modules\tabs\deepfloyd_if.py", line 61, in generate_image
    for data in fn(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 179, in stage_I
    prompt_embeds, negative_prompt_embeds = self._encode_prompt(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 157, in _encode_prompt
    prompt_embeds, negative_embeds = self.IF_I.encode_prompt(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\deepfloyd_if\pipeline_if.py", line 324, in encode_prompt
    prompt_embeds = self.text_encoder(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\transformers\models\t5\modeling_t5.py", line 1926, in forward
    encoder_outputs = self.encoder(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\transformers\models\t5\modeling_t5.py", line 1086, in forward
    layer_outputs = layer_module(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\transformers\models\t5\modeling_t5.py", line 693, in forward
    self_attention_outputs = self.layer[0](
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\transformers\models\t5\modeling_t5.py", line 599, in forward
    normed_hidden_states = self.layer_norm(hidden_states)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\transformers\models\t5\modeling_t5.py", line 260, in forward
    return self.weight * hidden_states
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

when only cuda 0 is visible

stage 3 testing

option is medvram

To create a public link, set `share=True` in `launch()`.
text_encoder\model.safetensors not found
Traceback (most recent call last):
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1036, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\utils.py", line 488, in async_iteration
    return next(iterator)
  File "F:\deepfloyd ai\Radiata\modules\tabs\deepfloyd_if.py", line 61, in generate_image
    for data in fn(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 229, in stage_III
    images = self.IF_III(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_upscale.py", line 585, in __call__
    self.check_inputs(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_upscale.py", line 431, in check_inputs
    raise ValueError(
ValueError: `image` has to be of type `torch.Tensor`, `PIL.Image.Image` or `list` but is <class 'NoneType'>

when normal mode selected - i did restart

stage 1 done
stage 2 done
stage 3 error below. i have rtx 3090

ValueError: `image` has to be of type `torch.Tensor`, `PIL.Image.Image` or `list` but is <class 'NoneType'>
Downloading shards: 100%|█████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2959.99it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.79s/it]
safety_checker\model.fp16.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:07<00:00,  6.76it/s]
safety_checker\model.fp16.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.30it/s]
vae\diffusion_pytorch_model.safetensors not found
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:11<00:00,  4.37it/s]
Traceback (most recent call last):
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1036, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\utils.py", line 488, in async_iteration
    return next(iterator)
  File "F:\deepfloyd ai\Radiata\modules\tabs\deepfloyd_if.py", line 61, in generate_image
    for data in fn(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 229, in stage_III
    images = self.IF_III(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_upscale.py", line 716, in __call__
    image = self.decode_latents(latents)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_upscale.py", line 376, in decode_latents
    image = self.vae.decode(latents).sample
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\utils\accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\autoencoder_kl.py", line 191, in decode
    decoded = self._decode(z).sample
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\autoencoder_kl.py", line 178, in _decode
    dec = self.decoder(z)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\vae.py", line 233, in forward
    sample = self.mid_block(sample)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\unet_2d_blocks.py", line 463, in forward
    hidden_states = attn(hidden_states)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\attention.py", line 168, in forward
    torch.empty(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 23.99 GiB total capacity; 18.06 GiB already allocated; 2.48 GiB free; 18.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[FurkanGozukara]

sequential_off_load gives this below error i have 64 gb ram

To create a public link, set `share=True` in `launch()`.
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1936.43it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.62s/it]
safety_checker\model.fp16.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin, safety_checker/pytorch_model.fp16.bin, unet/diffusion_pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Traceback (most recent call last):
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1036, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\utils.py", line 488, in async_iteration
    return next(iterator)
  File "F:\deepfloyd ai\Radiata\modules\tabs\deepfloyd_if.py", line 61, in generate_image
    for data in fn(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 179, in stage_I
    prompt_embeds, negative_prompt_embeds = self._encode_prompt(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 156, in _encode_prompt
    self.load_pipeline("I", "IF_I", text_encoder=self.t5)
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 121, in load_pipeline
    self.IF_I.enable_sequential_cpu_offload()
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\deepfloyd_if\pipeline_if.py", line 165, in enable_sequential_cpu_offload
    cpu_offload(cpu_offloaded_model, device)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\big_modeling.py", line 182, in cpu_offload
    attach_align_device_hook(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 394, in attach_align_device_hook
    attach_align_device_hook(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 394, in attach_align_device_hook
    attach_align_device_hook(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 385, in attach_align_device_hook
    add_hook_to_module(module, hook, append=True)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 145, in add_hook_to_module
    remove_hook_from_module(module)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 186, in remove_hook_from_module
    module._hf_hook.detach_hook(module)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 114, in detach_hook
    module = hook.detach_hook(module)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 300, in detach_hook
    set_module_tensor_to_device(module, name, device, value=self.weights_map.get(name, None))
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\utils\modeling.py", line 130, in set_module_tensor_to_device
    if tensor_name not in module._parameters and tensor_name not in module._buffers:
AttributeError: 'NoneType' object has no attribute '_parameters'

[FurkanGozukara]

off_load -
stage 1 works
stage 2 works
stage 3 error

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00001-of-00002.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:10<00:00,  4.79it/s]
text_encoder\model.safetensors not found
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:11<00:00,  4.25it/s]
Traceback (most recent call last):
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1036, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\utils.py", line 488, in async_iteration
    return next(iterator)
  File "F:\deepfloyd ai\Radiata\modules\tabs\deepfloyd_if.py", line 61, in generate_image
    for data in fn(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 229, in stage_III
    images = self.IF_III(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_upscale.py", line 716, in __call__
    image = self.decode_latents(latents)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_upscale.py", line 376, in decode_latents
    image = self.vae.decode(latents).sample
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\utils\accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\autoencoder_kl.py", line 191, in decode
    decoded = self._decode(z).sample
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\autoencoder_kl.py", line 178, in _decode
    dec = self.decoder(z)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\vae.py", line 233, in forward
    sample = self.mid_block(sample)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\unet_2d_blocks.py", line 463, in forward
    hidden_states = attn(hidden_states)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\models\attention.py", line 180, in forward
    attention_probs = torch.softmax(attention_scores.float(), dim=-1).type(attention_scores.dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB (GPU 0; 23.99 GiB total capacity; 9.46 GiB already allocated; 12.11 GiB free; 9.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[FurkanGozukara]

lowvram first stage fails

To create a public link, set `share=True` in `launch()`.
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 3151.24it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:13<00:00,  6.80s/it]
safety_checker\model.fp16.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Traceback (most recent call last):
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\blocks.py", line 1036, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\gradio\utils.py", line 488, in async_iteration
    return next(iterator)
  File "F:\deepfloyd ai\Radiata\modules\tabs\deepfloyd_if.py", line 61, in generate_image
    for data in fn(
  File "F:\deepfloyd ai\Radiata\modules\diffusion\pipelines\deepfloyd_if.py", line 184, in stage_I
    images = self.IF_I(
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "F:\deepfloyd ai\Radiata\venv\lib\site-packages\diffusers\pipelines\deepfloyd_if\pipeline_if.py", line 722, in __call__
    height = height or self.unet.config.sample_size
AttributeError: 'NoneType' object has no attribute 'config'

[FurkanGozukara]

tested all options and stage 3 fails :)
i got rtx 3090 and rtx 3060 cant make it work

[Stax124]

@ddPn08 in that offload mode, are you doing

gc.collect()
with torch.cuda.device(self.gpu_id):
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()

I haven't looked at the codebase but it seems that some model wasn't unloaded (or offloaded) or there is some reference to it that prevents it from doing so.

[FurkanGozukara]

@ddPn08 i sent a message to you from twitter to
i am willing to test and let you fix this error for a tutorial

my discord : MonsterMMORPG#2198

[FurkanGozukara]

@ddPn08 there is a kaggle notebook that shows how to load into 2 gpu that may help

https://www.kaggle.com/furkangozukara/deepfloyd-if-4-3b-generator-of-pictures-video-vers

[ddPn08]

Thank you for all the experiments. The OOM is probably caused by your version of torch. Try 2.0.0. Other errors seem to require modification of device-related code.

[FurkanGozukara]

Is this fixed?

Hopefully I am planning a tutorial very soon

[ddPn08]

Sorry for leaving it for a while. Sorry, I gave up on the Deepfloyd IF implementation. We are moving towards SDXL support instead.