Diffusers

A Unity package to run pretrained diffusion models with Unity Sentis

OpenUPM · Documentation (coming soon) · Feedback/Questions

About

This is essentially a port of Hugging Face’s diffusers library.

It is still very early though, so as of today only a limited number of pipelines and schedulers are supported (see below).

Installation

via OpenUPM

1. In Edit -> Project Settings -> Package Manager, add a new scoped registry:

   Name: Doji
   URL: https://package.openupm.com
   Scope(s): com.doji
   

2. In the Package Manager install com.doji.diffusers either by name or select it in the list under Package Manager -> My Registries

3. For the time being, you also have to use a custom fork of Sentis: In the package Manager -> Package Manager -> Install package from git URL-> https://github.com/julienkay/com.unity.sentis.git

Roadmap

Some things that might be worked on next are:

• add/fix classifier-free guidance
• support more models
  • SD 1.5
  • SD 2.1
  • SDXL
  • SD-Turbo
  • SDXL-Turbo
  • LoRA models
• more scheduler implementations
  • PNDM
  • DDIM
  • EulerDiscrete
  • EulerAncestralDiscrete
  • LCM
  • DDPM
  • KDPM2Discrete
  • KDPM2AncestralDiscrete
• more pipeline/model types
  • txt2img
  • img2img,
  • inpaint,
  • upscale,
  • depth2img
  • marigold
• async pipeline methods (sliced inference)
• support multiple images per prompt
• write some documentation

Dependencies

• com.doji.transformers
• com.doji.pngcs