Combine SD with ... ?

[Ehplodor]

Stable Diffusion, as well as a number of other generative models, is great at generative image creation and modification.

I do believe as a number of other people that this sort of generative models will revolutionize many things (#3219 for a beginning discussion about this subject)

Now I wonder if and how this new tool can be combined with existing software ?

Examples of softwares already incorporating SD in one way or another :

• 3D modeling

  • Blender
    • AI render free plugin
    • Tileable texture generation
  • Unity
  • UE5 : https://github.com/albertotrunk/ue-stable-diffusion
  • NerF
    • DreamFusion
    • Paving the way to "txt2asset" plugin for 3d editors

• 2D editing

  • Photoshop
  • Gimp
  • Krita
    • [Feature Request]: New extension auto-sd-paint-ext, how to PR to wiki's extensions list? #4172

Ideas

• Video editing

  • few shot patch based learning
    • https://twitter.com/NicolaiNightVi1/status/1586434671663013889?s=20&t=61O_TNlIib5JcSc3rnLnxA
    • https://github.com/nicolai256/Few-Shot-Patch-Based-Training

• Illustration

  • libreoffice ? (that would be a very awesome addition)

• Design

  • example : Microsoft * Dall-e

• Pseudo image search in web browsers or browser-based search engines

  • example (again) : Microsoft * Dall-e

• Social medias

• Conferencing

  • Realtime face swap / modifiers / anonimization
  • Realtime Background swap / background anonymization

• Other models

  • Large language models (GPT3 etc...)
    • automatic prompt generation based on text input, for automatic illustration ("txt2txt2img")
    • NovelAI, of course

• Deepfly

• Games !

I'll add to this list as I encounter, or as I (you ?) think about, new SD potential implementations.

[Dement242]

Something Node based, like ChaiNNer/TouchDesigner, Where every step is broken out into operators, so you can manipulate prompt(s), latent space, img2img and mask etc.

[TdoubleAV]

I think a good background removal option could greatly improve workflow and capabilities:
https://huggingface.co/spaces/ECCV2022/dis-background-removal

[Ehplodor]

Thank you for the link this seems really good ! Will you file a feature request for this ?