SDXL ORT base example produces broken images on Windows 10

[bzlibby]

System Info

OS: Windows 10 Home 22H2 (Build 19045.3324)
Python 3.10.6

pip list --local

Package                 Version
----------------------- ---------------
absl-py                 1.4.0
accelerate              0.21.0
addict                  2.4.0
aiohttp                 3.8.5
aiosignal               1.3.1
antlr4-python3-runtime  4.9.3
Arpeggio                2.0.0
async-timeout           4.0.3
attrs                   23.1.0
basicsr                 1.4.2
beautifulsoup4          4.12.2
boto3                   1.26.69
botocore                1.29.165
cachetools              5.3.1
certifi                 2023.7.22
charset-normalizer      3.2.0
click                   8.1.7
codeformer-perceptor    0.1.2
colorama                0.4.6
coloredlogs             15.0.1
contourpy               1.1.0
controlnet-aux          0.0.2
cycler                  0.11.0
datasets                2.14.4
diffusers               0.20.0
dill                    0.3.7
einops                  0.6.1
facexlib                0.2.5
filelock                3.12.2
filterpy                1.4.5
Flask                   2.2.2
Flask-Cors              3.0.10
flatbuffers             23.5.26
fonttools               4.42.1
frozenlist              1.4.0
fsspec                  2023.6.0
future                  0.18.3
gdown                   4.7.1
gfpgan                  1.3.8
google-auth             2.22.0
google-auth-oauthlib    1.0.0
grpcio                  1.57.0
huggingface-hub         0.16.4
humanfriendly           10.0
idna                    3.4
imageio                 2.31.1
importlib-metadata      6.8.0
invisible-watermark     0.2.0
itsdangerous            2.1.2
Jinja2                  3.1.2
jmespath                1.0.1
jsonschema              4.17.3
kiwisolver              1.4.4
llvmlite                0.40.1
lmdb                    1.4.1
lpips                   0.1.4
Markdown                3.4.4
MarkupSafe              2.1.3
matplotlib              3.7.2
mediapipe               0.9.2.1
mpmath                  1.3.0
multidict               6.0.4
multiprocess            0.70.15
networkx                3.1
numba                   0.57.1
numpy                   1.25.2
oauthlib                3.2.2
omegaconf               2.3.0
onnx                    1.13.0
onnxruntime             1.15.1
onnxruntime-directml    1.15.1
opencv-contrib-python   4.8.0.76
opencv-python           4.8.0.76
optimum                 1.11.2
packaging               23.1
pandas                  2.0.3
piexif                  1.1.3
Pillow                  10.0.0
pip                     23.2.1
protobuf                4.24.1
psutil                  5.9.5
pyarrow                 12.0.1
pyasn1                  0.5.0
pyasn1-modules          0.3.0
pyparsing               3.0.9
pyreadline3             3.4.1
pyrsistent              0.19.3
PySocks                 1.7.1
python-dateutil         2.8.2
pytz                    2023.3
PyWavelets              1.4.1
PyYAML                  6.0
realesrgan              0.3.0
regex                   2023.8.8
requests                2.31.0
requests-oauthlib       1.3.1
rsa                     4.9
s3transfer              0.6.2
safetensors             0.3.2
scikit-image            0.19.3
scipy                   1.11.2
sentencepiece           0.1.99
setproctitle            1.3.2
setuptools              63.2.0
six                     1.16.0
soupsieve               2.4.1
sympy                   1.12
tb-nightly              2.15.0a20230820
tensorboard-data-server 0.7.1
tifffile                2023.8.12
timm                    0.6.13
tokenizers              0.13.3
torch                   2.0.1
torchvision             0.15.2
tqdm                    4.66.1
transformers            4.31.0
typing_extensions       4.7.1
tzdata                  2023.3
urllib3                 2.0.4
waitress                2.1.2
Werkzeug                2.3.7
wget                    3.2
wheel                   0.41.1
xxhash                  3.3.0
yapf                    0.32.0
yarl                    1.9.2
zipp                    3.16.2

Who can help?

@JingyaHuang, @echarlaix

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)

I was using this snippet from the documentation for SDXL:

from optimum.onnxruntime import ORTStableDiffusionXLPipeline

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
base = ORTStableDiffusionXLPipeline.from_pretrained(model_id)
prompt = "sailing ship in storm by Leonardo da Vinci"
image = base(prompt).images[0]
image.save("../outputs/sdxl-test.png")

and got this output:

I originally ran into this while trying to use SDXL with onnx-web, but was able to reproduce it with the above example and the model that you've already converted. I tried it with the DirectML provider as well and get similar results.

Expected behavior

I would expect to get a picture of a boat, similar to the one from the documentation:

[fxmarty]

Thank you, will have a look asap!

[fxmarty]

@bzlibby I can not reproduce the issue on CUDA EP on Linux. I have to say I'm not super keen to run on CPU EP because it is super slow (I'll try next week).

from optimum.onnxruntime import ORTStableDiffusionXLPipeline

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
base = ORTStableDiffusionXLPipeline.from_pretrained(model_id, provider="CUDAExecutionProvider")
prompt = "sailing ship in storm by Leonardo da Vinci"

print("start inference")
image = base(prompt).images[0]
image.save("./sdxl-test.png")

getting:

Could you maybe try updating optimum to 1.12? Could you try to run on CUDA EP to see if the issue is there as well? You would need to install onnxruntime-gpu instead of onnxruntime.

[ssube]

Hi, I can reproduce this as well, with very similar results. Notably, I also have a Linux test machine with both the CUDA and ROCm providers set up, and I cannot reproduce the issue there. This is specific to Windows.

a very blocky image:

I'm using:

accelerate                0.22.0
diffusers                 0.20.1
onnx                      1.13.1
onnxruntime               1.15.1
onnxruntime-directml      1.15.1
opencv-contrib-python     4.7.0.72
opencv-python             4.7.0.68
optimum                   1.12.0
torch                     1.13.1+cpu
torchmetrics              0.11.4
torchvision               0.14.1+cpu
transformers              4.32.1

@bzlibby originally found this while helping me test SDXL support for my ONNX tool, which is primarily used by people on Windows machines with AMD video cards, where the CUDA provider is not available. I understand that using the CPU provider is very slow, but I wanted to rule out issues with the AMD drivers or DirectML, and CPU is the fallback provider for ORT.

We are both able to reproduce the issue using the pre-converted model on HF hub and the example from the documentation, no other code needed. This does not happen on Linux on any version of optimum that I have tested, with any provider. It does happen with both the CPU and DirectML providers on Windows.

[fxmarty]

Yes that's bad. Is the issue only with SDXL or e.g. with SD 1.2 as well? As soon as I can get my hands on a Windows machine I can have a look. It could possibly be a bug in ORT.

[bzlibby]

@fxmarty I updated optimum like you suggested and tested both SDXL and SD 1.5 (I didn't see an example in the documentation for 1.2, but I can test that too if it'd be helpful!) - I'm getting the same results with SDXL still, but SD 1.5 worked just fine for me:

from optimum.onnxruntime import ORTStableDiffusionPipeline

model_id = "runwayml/stable-diffusion-v1-5"
pipeline = ORTStableDiffusionPipeline.from_pretrained(model_id, revision="onnx")
prompt = "sailing ship in storm by Leonardo da Vinci"
image = pipeline(prompt).images[0]
image.save("../outputs/sdxl-test.png")

output:

using:

accelerate              0.22.0
diffusers               0.20.2
onnx                    1.13.0
onnxruntime             1.15.1
onnxruntime-directml    1.14.0
opencv-contrib-python   4.8.0.76
opencv-python           4.8.0.76
optimum                 1.12.0
torch                   1.13.1
torchmetrics            0.11.4
torchvision             0.14.1
transformers            4.32.0