CKPT/Safetensors, already converted models

[NikitaPZ]

Is there any way to load already converted ONNX models, or somehow convert and load models like those found on civitai in the .ckpt/.safetensors format?

[ssube]

For already-converted ONNX models, if they work with the Stable Diffusion pipelines, I think they should work. Do you have anything in mind? Happy to test a few.

You can add models by putting them in the models directory and starting the file/folder name with diffusion-, upscaling-, or correction- (https://github.com/ssube/onnx-web/blob/v0.6.1/api/onnx_web/serve.py#L302). I'll put a note in the user guide about that naming convention. In this case, probably diffusion-whatever.

For downloading and converting models from Civitai, I need to look into that. The code currently uses diffusers to download and cache the CKPT files automagically, but I would like to add more sources for models.

[NikitaPZ]

I meant ONNX models that I've already converted myself with different scripts that I used before, I'm not aware of any that are available on the internet. Thank you for clarifying about the naming convention, didn't realize that.

[ssube]

I had some time to mess around with this and after hacking together some conversion code, I was able to convert and run a bunch of models from Civitai:

[2023-02-08 21:58:05,547] INFO: onnx_web.convert.diffusion_stable: exporting ONNX model
[2023-02-08 21:58:05,600] INFO: onnx_web.convert.diffusion_stable: ONNX pipeline saved to ../models/../models/diffusion-stably-diffused-onnx-v2-6
[2023-02-08 21:58:15,984] INFO: onnx_web.convert.diffusion_stable: ONNX pipeline is loadable
[2023-02-08 21:58:16,223] INFO: onnx_web.convert.diffusion_original: ONNX pipeline saved to diffusion-stably-diffused-onnx-v2-6
[2023-02-08 21:58:16,224] INFO: onnx_web.convert.diffusion_original: Converting original Diffusers checkpoint diffusion-unstable-ink-dream-onnx-v6: ../models/tensors/unstableinkdream_v6.safetensors -> ../models/diffusion-unstable-ink-dream-onnx-v6
[2023-02-08 21:58:16,224] INFO: onnx_web.convert.diffusion_original: Converting original Diffusers check to Torch model: ../models/tensors/unstableinkdream_v6.safetensors -> ../models/diffusion-unstable-ink-dream-torch-v6
...
[2023-02-08 22:02:38,496] INFO: onnx_web.convert.diffusion_stable: exporting ONNX model
[2023-02-08 22:02:38,554] INFO: onnx_web.convert.diffusion_stable: ONNX pipeline saved to ../models/../models/diffusion-unstable-ink-dream-onnx-v6
[2023-02-08 22:02:49,206] INFO: onnx_web.convert.diffusion_stable: ONNX pipeline is loadable
[2023-02-08 22:02:49,516] INFO: onnx_web.convert.diffusion_original: ONNX pipeline saved to diffusion-unstable-ink-dream-onnx-v6

The conversion is two steps: from CKPT/Safetensors -> Diffusers directory -> ONNX models (in a directory). You don't need to convert Safetensors to CKPT, they were both loading just fine. The intermediate files are not deleted (yet), so it will take a little bit more disk space, but I started this project to learn/use ONNX for better or worse. 😁

I need to clean up the code, write something for the user guide, and finish some other TODOs before merging it, so the code is on https://github.com/ssube/onnx-web/tree/feat/117-convert-safetensors for now. If you want to try it with your own models, add them to the extras.json file with onnx in the name somewhere (a hack I will fix) and make sure second item in the tuple ends with .ckpt or .safetensors:

{
  "diffusion": [
    ["diffusion-knollingcase", "Aybeeceedee/knollingcase"],
    ["diffusion-openjourney", "prompthero/openjourney"],
    ["diffusion-stably-diffused-onnx-v2-6", "../models/tensors/stablydiffuseds_26.safetensors"],
    ["diffusion-unstable-ink-dream-onnx-v6", "../models/tensors/unstableinkdream_v6.safetensors"]
  ],
  "correction": [],
  "upscaling": []
}

[ssube]

I've cleaned up a few of the hacks and added a generic way to download models in the conversion script, with a section in the user guide: https://github.com/ssube/onnx-web/blob/feat/117-convert-safetensors/docs/user-guide.md#adding-your-own-models

The extras.json file offers a few more options now:

{
  "diffusion": [
    {
      "name": "diffusion-knollingcase",
      "source": "Aybeeceedee/knollingcase"
    },
    {
      "name": "diffusion-openjourney",
      "source": "prompthero/openjourney"
    },
    {
      "name": "diffusion-stablydiffused-aesthetic-v2-6",
      "source": "civitai://6266?type=Pruned%20Model&format=SafeTensor",
      "format": "safetensors"
    },
    {
      "name": "diffusion-unstable-ink-dream-onnx-v6",
      "source": "civitai://5796",
      "format": "safetensors"
    }
  ],
  "correction": [],
  "upscaling": [
    {
      "name": "upscaling-real-esrgan-x4-anime",
      "source": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth",
      "scale": 4
    }
  ]
}

I'm reasonably happy with how this is working so far, but if you have any suggestions, lmk. Hope to merge this soon.

[ssube]

This has been merged: #118