DirectML 1.10 is 2x as slow as CUDA for onnxruntime on NVidia Quadro P5000

[elephantpanda]

I am using onnxruntime to run stable diffusion float16 onnx. And I am getting times of 1.5 seconds per cycle for DirectML and 0.7 seconds per cycle when I switch to CUDA runtime.

I have Nvidia 1080 GTX equivalent (actually is is Quadro P5000).

Is this as expected? I am using latest DirectML.dll 1.10 from the nuget package.

[Amblyopius]

Are you using ONNX Runtime Nightly? For other hardware the difference between the official release and nightly is close to 2x. Note though that to my knowledge a P5000 does not have the features to do proper FP16, your card is faster in FP32 because the chip is crippled in FP16. My laptop GTX 1070 on CUDA does 0.55s per iteration on FP32 hence outperforming your P5000 despite how weak it is in comparison.

[elephantpanda]

Are you using ONNX Runtime Nightly? For other hardware the difference between the official release and nightly is close to 2x. Note though that to my knowledge a P5000 does not have the features to do proper FP16, your card is faster in FP32 because the chip is crippled in FP16. My laptop GTX 1070 on CUDA does 0.55s per iteration on FP32 hence outperforming your P5000 despite how weak it is in comparison.

Hello, I am not using the nightly build. I will try that. (I assumed that the problem bust be in the DirectML implementation rather than the onnxruntime since CUDA is so fast).

My FP16 version is slightly faster on CUDA than the FP32 version. (BTW there are more reasons to use FP16 such as smaller file sizes and load times). But that might be other reasons such as time spent pushing data from CPU to GPU.

I would prefer to use DirectML because the CUDAnn runtime is about 2GB!

Have you tried DirectML on your GTX 1070? Can you get similar times to CUDA?

Also I am getting about 0.38 seconds per cycle (CUDA) if I set guidance=1, since that only needs to do half the calculation.

[Amblyopius]

I maintain a repo focused on ONNX FP16 here: https://github.com/Amblyopius/AMD-Stable-Diffusion-ONNX-FP16. Mostly to support users with AMD cards (as they don't have the CUDA option) but happy to support Nvidia. My main card is a 6700XT so haven't tried DirectML on the GTX 1070. My understanding is that we're getting an updated ONNX Runtime in February, until then nightly is the way to go. You can always check the repo for what is the best (known to me) state of affairs.

[elephantpanda]

I did a pip install of ort-nightly-directml and swapped over the DirectML.dll files but now I get an error:

Non-zero status code returned while running Trilu node. Name:'/text_model/Trilu' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1866)\onnxruntime.dll!00007FFC6FBE2A90: (caller: 00007FFC6FBE1F93) Exception(3) tid(ad0) 80070057 The parameter is incorrect.

I didn't see any new onxxruntime.dll installed anywhere.

I guess I need a new onnxruntime.dll from somewhere. I'll be honest I don't really understand how these ort-nightly things work. I downloaded all my dot net packages from visual studio nugets. (Oh I forgot to mention I'm using c#)

[Amblyopius]

I was typing away and then you updated that you were using c#. You can get the nightly for DirectML in the same place: https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly

[elephantpanda]

I was typing away and then you updated that you were using c#. You can get the nightly for DirectML in the same place: https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly

Thanks! It worked!!!! 🥳🎉🎉🎉🥳 It's a miracle.

For anyone else with this problem:
I didn't need to replace the DirectML.dll file
I just needed to get the 1.14 onnx runtime developer release.

Now my DirectML times are about 0.42 seconds (compared to 0.38 for CUDA - an I'm not going to quibble about 10%). That's good enough for me 😁 (With the released runtime it was 0.9 seconds so over 2x improvement!) Now I can delete all those 2GB CUDAnn files.

[Amblyopius]

Cool, happy to help! To my knowledge (but for clarity: not associated with MS or the team) the next official release is planned for February. So then you can go to the mainline and get the benefits without having to use the dev version.

[elephantpanda]

It's good to be ahead of the curve.