Skip to content

TouchDesigner implementation for real-time Stable Diffusion interactive generation with StreamDiffusion.

Notifications You must be signed in to change notification settings

olegchomp/TouchDiffusion

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

58 Commits
 
 
 
 
 
 
 
 

Repository files navigation

TouchDiffusion

Discord Shield

TouchDesigner implementation for real-time Stable Diffusion interactive generation with StreamDiffusion.

Benchmarks with stabilityai/sd-turbo, 512x512 and 1 batch size.

GPU FPS
4090 55-60 FPS
4080 47 FPS
3090ti 37 FPS
3090 30-32 FPS
4070 Laptop 24 FPS
3060 12GB 16 FPS

Disclaimer

Notice: This repository is in an early testing phase and may undergo significant changes. Use it at your own risk.

Usage

Tip

TouchDiffusion can be installed in multiple ways. Portable version have prebuild dendencies, so it prefered way to install or Manuall install is step by step instruction.

Portable version:

Includes preinstalled configurations, ensuring everything is readily available for immediate use.

  1. Download and extract archive
  2. Run webui.bat. It will provide url to web interface (ex. http://127.0.0.1:7860)
  3. Open install & update tab and run Update dependencies.

Manuall install:

You can follow YouTube tutorial

Required TouchDesigner 2023 & Python 3.11

  1. Install Python 3.11
  2. Install Git
  3. Install CUDA Toolkit 11.8 (required PC restart)
  4. Download TouchDiffusion.
  5. Open webui.bat with text editor and set path to Python 3.11 in set PYTHON_PATH=. (ex. set PYTHON_PATH="C:\Program Files\Python311\python.exe")
  6. Run webui.bat. After installation it will provide url to web interface (ex. http://127.0.0.1:7860)
  7. Open install & update tab and run Update dependencies. (could take ~10 minutes, depending on your internet connection)
  8. If you get pop up window with error related to .dll, run Fix pop up
  9. Restart webui.bat

Accelerate model:

Models in .safetensors format must be in models\checkpoints folder. (as for sd_turbo, it will be auto-downloaded).

Internet connection required, while making engines.

  1. Run webui.bat
  2. Select model type.
  3. Select model.
  4. Set width, height and amount of sampling steps (Batch size)
  5. Select acceleration lora if available.
  6. Run Make engine and wait for acceleration to finish. (could take ~10 minutes, depending on your hardware)

TouchDesigner inference:

  1. Add TouchDiffusion.tox to project
  2. On Settings page change path to TouchDiffusion folder (same as where webui.bat).
  3. Save and restart TouchDesigner project.
  4. On Settings page select Engine and click Load Engine.
  5. Connect animated TOP to input. Component cook only if input updates.

Known issues / Roadmap:

  • Fix Re-init. Sometimes required to restart TouchDesigner for initializing site-packages.
  • Code clean-up and rework.
  • Custom resolution (for now fixed 512x512)
  • CFG not affecting image
  • Add Lora
  • Add Hyper Lora support
  • Add ControlNet support
  • Add SDXL support

Acknowledgement

Based on the following projects:

  • StreamDiffusion - Pipeline-Level Solution for Real-Time Interactive Generation
  • TopArray - Interaction between Python/PyTorch tensor operations and TouchDesigner TOPs.

About

TouchDesigner implementation for real-time Stable Diffusion interactive generation with StreamDiffusion.

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published