[Bug]: RuntimeError: ADL2: Failed to get DedicatedVRAMUsage

[amegaara]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

i followed a video guide to run SD on AMD GPUs and i got SD to load on the webui, but whenever i enter a prompt and click generate it gives this error "RuntimeError: ADL2: Failed to get DedicatedVRAMUsage" i searched multiple times on google and reddit but no one seems to have this issue, can anyone please help me with this?

i have these arguments turned on" Launching Web UI with arguments: --opt-sub-quad-attention --lowvram --disable-nan-check " because im using an RX580 8gb

i have updated the GPU drivers

Steps to reproduce the problem

1. install stablediffusion amd
2. run webui
3. try to generate any image

What should have happened?

should generate an image

Version or Commit where the problem happens

v1.4.1

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

AMD GPUs

Cross attention optimization

Automatic

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--opt-sub-quad-attention --lowvram --disable-nan-check

List of extensions

none

Console logs

Startup time: 273.9s (import torch: 39.7s, import gradio: 17.1s, import ldm: 9.8s, other imports: 147.4s, opts onchange: 0.2s, setup codeformer: 5.4s, list SD models: 1.2s, load scripts: 10.5s, load upscalers: 0.7s, refresh textual inversion templates: 0.3s, reload hypernetworks: 1.3s, initialize extra networks: 2.8s, scripts before_ui_callback: 0.4s, create ui: 19.6s, gradio launch: 17.5s).
Creating model from config: C:\sd\webui\stable-diffusion-webui-directml\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: InvokeAI... done.
Textual inversion embeddings loaded(0):
Model loaded in 201.1s (load weights from disk: 44.4s, create model: 1.9s, apply weights to model: 150.2s, apply half(): 1.5s, move model to device: 1.4s, load textual inversion embeddings: 0.3s, calculate empty prompt: 1.3s).
  0%|                                                                                           | 0/20 [00:10<?, ?it/s]
*** Error completing request
*** Arguments: ('task(a6r4kjzypgwzk9a)', 'a cute cat', '', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0) {}
    Traceback (most recent call last):
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\call_queue.py", line 55, in f
        res = list(func(*args, **kwargs))
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\call_queue.py", line 35, in f
        res = func(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\txt2img.py", line 64, in txt2img
        processed = processing.process_images(p)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\processing.py", line 623, in process_images
        res = process_images_inner(p)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\processing.py", line 742, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\processing.py", line 995, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 439, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 278, in launch_sampling
        return func()
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 439, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 158, in forward
        x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict([cond_in], image_cond_in))
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_unet.py", line 91, in UNetModel_forward
        return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
        h = module(h, emb, context)
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
        x = layer(x, context)
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
        x = block(x, context=context[i])
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
        return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
        return CheckpointFunction.apply(func, len(inputs), *args)
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
        return super().apply(*args, **kwargs)  # type: ignore[misc]
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
        output_tensors = ctx.run_function(*ctx.input_tensors)
      File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 272, in _forward
        x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x
      File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 361, in split_cross_attention_forward_invokeAI
        r = einsum_op(q, k, v)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 336, in einsum_op
        return einsum_op_dml(q, k, v)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 321, in einsum_op_dml
        mem_free, mem_total = torch.dml.mem_get_info(q.device)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\dml\backend.py", line 68, in mem_get_info
        return memory_control.mem_get_info(device.index)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\dml\memctl\amd\__init__.py", line 7, in mem_get_info
        usage = AMDMemoryControl.driver.get_dedicated_vram_usage(index) * (1 << 20)
      File "C:\sd\webui\stable-diffusion-webui-directml\modules\dml\memctl\amd\driver\atiadlxx.py", line 36, in get_dedicated_vram_usage
        raise RuntimeError("ADL2: Failed to get DedicatedVRAMUsage")
    RuntimeError: ADL2: Failed to get DedicatedVRAMUsage

Additional information

No response

[lshqqytiger]

You are not on the latest version of the repository.
git pull and try again.

[amegaara]

You are not on the latest version of the repository. git pull and try again.

thanks for the reply! i just did that and now im getting this error "PDHError: Something went wrong." ill be searching online if someone has the same error

[lshqqytiger]

When do you get that error? After launching WebUI or before?

[rexorp]

I just updated to the latest version and now I am getting the same "RuntimeError: ADL2: Failed to get DedicatedVRAMUsage" when I attempt to generate an image. I am also running an AMD RX580 8gb.

[lshqqytiger]

Go to Settings > Optimizations and change DirectML memory stats provider to Performance Counter or None.

[e2heintz]

Go to Settings > Optimizations and change DirectML memory stats provider to Performance Counter or None.

I also get the PDHError. It started after a recent git pull. The above setting did not help. The error appears after hitting 'Generate'.

[lshqqytiger]

Do you get same error with None?

[amegaara]

i tried both settings and still nothing unfortunately

[amegaara]

`---
0%| | 0/20 [00:00<?, ?it/s]
*** Error completing request
*** Arguments: ('task(2yrcia0ivma100i)', 'a car', '', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000002646C73EA70>, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0) {}
Traceback (most recent call last):
File "C:\sd\webui\stable-diffusion-webui-directml\modules\call_queue.py", line 58, in f
res = list(func(*args, **kwargs))
File "C:\sd\webui\stable-diffusion-webui-directml\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\txt2img.py", line 69, in txt2img
processed = processing.process_images(p)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\processing.py", line 680, in process_images
res = process_images_inner(p)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\processing.py", line 797, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\processing.py", line 1057, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 464, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 303, in launch_sampling
return func()
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 464, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_samplers_kdiffusion.py", line 183, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_utils.py", line 17, in
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_utils.py", line 28, in call
return self.__orig_func(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_unet.py", line 91, in UNetModel_forward
return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
h = module(h, emb, context)
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
x = layer(x, context)
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
x = block(x, context=context[i])
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
return checkpoint(self.forward, (x, context), self.parameters(), self.checkpoint)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
return CheckpointFunction.apply(func, len(inputs), *args)
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
output_tensors = ctx.run_function(ctx.input_tensors)
File "C:\sd\webui\stable-diffusion-webui-directml\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 272, in forward
x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x
File "C:\sd\webui\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 389, in split_cross_attention_forward_invokeAI
r = einsum_op(q, k, v)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 363, in einsum_op
return einsum_op_dml(q, k, v)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\sd_hijack_optimizations.py", line 347, in einsum_op_dml
mem_free, mem_total = torch.dml.mem_get_info(q.device)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\dml\backend.py", line 16, in pdh_mem_get_info
mem_info = DirectML.memory_provider.get_memory(get_device(device).index)
File "C:\sd\webui\stable-diffusion-webui-directml\modules\dml\memory.py", line 17, in get_memory
paths_dedicated = expand_wildcard_path(f"\GPU Process Memory(pid{pid}phys{device_id})\Dedicated Usage")
File "C:\sd\webui\stable-diffusion-webui-directml\modules\dml\pdh_init.py", line 23, in expand_wildcard_path
raise PDHError("Something went wrong.")
modules.dml.pdh.errors.PDHError: Something went wrong.

---

`

this is the cmd line if it helps

[oinaskovs]

Have the same error when trying to run on a second GPU attached via a PCI 1x slot. First GPU (they're both 6600 XT) that's running the display is working fine.

[asthomas]

I have the same problem. In the above stack trace, the call to "expand_wildcard_path" is trying to find the performance counter for the VRAM usage of the python process by expanding this string:
\GPU Process Memory(pid_11696_*_phys_0)\Dedicated Usage
where the python process has PID 11696 (I confirmed this in task manager).

When I open the Windows performance monitor and look at the GPU Process Memory category, I can see a number of performance counters of the form "pid__*_phys_0", where PID is an existing process ID. The PID of the python process is not among them. For whatever reason, python does not have access to the GPU.

What am I missing?

[Ronin1mil]

In stable-diffusion-webui-directml\modules\dml open memory.py and change device_id in lines 20 and 21 to 0. I tested on my rx 580 and 3060 and it allows the 580 to generate images.

Edit: being more specific
paths_dedicated = expand_wildcard_path(f"\GPU Process Memory(pid_{pid}*phys{device_id})\Dedicated Usage")
paths_committed = expand_wildcard_path(f"\GPU Process Memory(pid{pid}_*phys{device_id})\Total Committed")

[dfmjndm]

I also get the PDHError. It started after a recent git pull.