[Bug]: Optimization of Unet fails 6950 XT

[captroper]

What happened?

This appeared to me to be the same issue as 510 and 301, though I know nothing. I ran the following commands:

• conda create --name olive python=3.9
• conda activate olive
• pip install olive-ai[directml]==0.3.1
• git clone https://github.com/microsoft/olive --branch v0.3.1
• cd (to relevant directory)
• pip install -r requirements.txt
• python stable_diffusion_xl.py --optimize

I've attached the log, as well as a DXDIAG, but it errors out when optimizing unet saying "failed to run olive on gpu-dml".... "887a0006 the gpu will not respond to more commands".

DxDiag.txt
ErrorLog.txt

Version?

0.3.1

[guotuofeng]

The following error message seems be related to DirectML EP.

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\ExecutionProvider.cpp(896)\onnxruntime_pybind11_state.pyd!00007FFE31C80201: (caller: 00007FFE31C80C2F) Exception(2) tid(3c14) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

[guotuofeng]

@jstoecker, do you have any insight?

[guotuofeng]

seems similar with #510

[jstoecker]

This is DXGI_ERROR_DEVICE_HUNG during inference/evaluation, which typically happens when some GPU work is taking excessively long. The recent AMD driver optimizations for stable diffusion / multi-head attention target the RDNA 3 architecture (e.g., the 7000 series, like the Radeon RX 7900 XTX) but not the RDNA 2 (6000 series). Still, we can try to repro this on an RDNA card to see if anything jumps out.

[CellerX]

6800xt has same err

[vibbix]

Error on my 6900XT as well, on 0.4.0

[Jerry-zirui]

Same Error occurred in AMD Ryzen 7 7840U w/ Radeon 780M Graphics.
I increased the dedicated GPU memory as #510 mentioned, but the error still.