MIOpen(HIP): Error [EvaluateInvokers] & Potential VAE Decoder Instability with Large Inputs

[Cynthia-lxx]

Problem Description

Environment

OS: Windows 11 24H2
Hardware: AMD RX 7800M (gfx1101)
Software: PyTorch with ROCm support (2.7.0a0+git3f903c3)

Description

Two related observations while running Stable Diffusion inference (specifically image upscaling with large tile sizes using SD1.5 models):

1. VAE Decoder Instability: Under specific conditions (handling large latent tensors, e.g., ~1024x1024+), the VAE decoder produces output tensors containing NaN (Not a Number) or Inf (Infinity) values. This seems more prevalent with SD 1.5 models and correlates with specific large input sizes.
2. MIOpen Errors: The following error message appears frequently in the logs during these operations, often coinciding with performance hiccups (delays of a few seconds):
   MIOpen(HIP): Error [EvaluateInvokers] ... Invalid elapsed time detected

The VAE instability is the primary functional problem, leading to corrupted output. The MIOpen error might be a symptom or a separate underlying performance/validation issue.

Root Cause

The root cause is suspected to be deep within the ROCm software stack (MIOpen/HIP). The combination of large tensor dimensions specific to SD1.5 model architecture and the ROCm backend might trigger numerical instability or validation errors in certain convolution operations within the VAE decoder.

Proposed Solution

Investigate the potential root causes for:

1. The VAE decoder producing NaN/Inf values on large inputs specifically under ROCm.
2. The MIOpen(HIP): Error [EvaluateInvokers] ... Invalid elapsed time detected error.
   A fix at this level would prevent the invalid values from being generated in the first place, which is preferable to handling them downstream in user code.

Impact

The NaN/Inf values corrupt the output of applications (like ComfyUI & USDU) relying on stable VAE decoding. The MIOpen errors cause non-fatal but noticeable performance interruptions. Resolving this would improve the stability and user experience of PyTorch ROCm users in the generative AI space.

Operating System

Windows 11 24H2

CPU

Intel i5-13500H

GPU

AMD RX 7800M (gfx1101)

ROCm Version

ROCm 6.2

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[tcgu-amd]

Hi @wjz-space, thanks for reaching out to us, and sorry that you are experiencing issues with ROCm. The Nah/Inf issue sounds like it could be caused by numerical precision of the datatype; do you know if your VAE decoder is running in fp16/bf16 or fp32 mode? Also, how often does this error occur?

For MIOpen(HIP): Error [EvaluateInvokers] ... Invalid elapsed time detected I have a theory that it might be related to how the Windows driver interacts with the tuning process in MIOpen, but will need to look deep to confirm.

Thanks!

[tcgu-amd]

HI @wjz-space, so far, it doesn't seem like I was able to reproduce either of the issues. I was using the latest rocm/pytorch wheels from TheRock. How did you install ROCm on your system? Thanks!

[tcgu-amd]

@wjz-space While I still wasn't able to reproduce the numerical instability, I was able to reproduce the time out issue, although it looked slight different on my end:

I believe they belong to the same class issues, since they are both timer related. Will begin root-cause. Thanks!

[Cynthia-lxx]

Hi @tcgu-amd, thanks for looking into this!
To answer your questions:

• The issue occurs with a Stable Diffusion 1.5-based model in fp16 precision.
• The NaN/Inf problem consistently appears when the tile size is 720×480 or larger (on a 12 GB VRAM GPU), and it seems to happen during the VAE decode step.
• I can reliably reproduce the issue using:
  ComfyUI (2025.09.12 update) + ROCm 6.2 + RX 7800M + UltimateSDUpscale (latest version).

I've also cross-reported this issue in two other repositories with additional context, which may be helpful:
Comfy-Org/ComfyUI#9871
ssitu/ComfyUI_UltimateSDUpscale#159
Let me know if you need a minimal reproduction script or logs. Happy to help!

[tcgu-amd]

Hi @tcgu-amd, thanks for looking into this! To answer your questions:

• The issue occurs with a Stable Diffusion 1.5-based model in fp16 precision.
• The NaN/Inf problem consistently appears when the tile size is 720×480 or larger (on a 12 GB VRAM GPU), and it seems to happen during the VAE decode step.
• I can reliably reproduce the issue using:
  ComfyUI (2025.09.12 update) + ROCm 6.2 + RX 7800M + UltimateSDUpscale (latest version).

I've also cross-reported this issue in two other repositories with additional context, which may be helpful: comfyanonymous/ComfyUI#9871 ssitu/ComfyUI_UltimateSDUpscale#159 Let me know if you need a minimal reproduction script or logs. Happy to help!

Thanks for getting back! I see that you are using quite an outdated version of ROCm. Can you try with the latest versions? We recommend trying the latest ROCm + pytorch with our new build system The Rock with from here https://github.com/ROCm/TheRock/blob/main/RELEASES.md. (it is much easier to install now, the package is much smaller and you can install from pip directly). If you can give that a try and let us know that would be great. Thanks!

[Cynthia-lxx]

Hi @tcgu-amd, thanks for the suggestion!
I’m currently stuck with the HIP SDK for Windows, so I can’t easily move to the latest ROCm/TheRock builds.

However, I did find a clue shortly after my last post:
when I launch ComfyUI with --use-pytorch-cross-attention instead of the default sub-quadratic attention path, both the NaN/Inf artifacts and the MIOpen(HIP): Error [EvaluateInvokers] messages disappear.
This seems to indicate that the instability is triggered by the sub-quadratic attention implementation rather than the VAE decoder itself.

Hope this helps narrow the root-cause :D

[tcgu-amd]

Hi @wjz-space, sorry for not updating in a while. Glad that you are able to find a work-around!

So after some digging, it appears that the first invalid time elapsed error is actually, in spite of its name, a catch-all error on Windows for failed solvers/kernels during the tuning process. Basically, if a solver fails/crashes due to unhandled errors (which shouldn't and almost never happens on Linux), the elapsed time for the solver will be zero, and the error you are seeing then catches and mark the solver as invalid. In other words, if you see this error, it usually means that some solvers that are not supported/tested on windows are being included in the tuning process, which is normal since Windows support for MIOpen is still relatively young. We are expanding our coverage as we speak, so this issue should fizzle out within a few updates. You shouldn't need to worry about it unless you are experiencing significant performance drops.

For the sub-quadratic attention issue, given that you are using an older version of ROCm and I cannot reproduce it on the latest, the issue was probably fixed somewhere in-between. I will leave it as it is for now, and if you still observe the same thing in a newer version when you do get the chance to update, let me know!

Otherwise, I will close this issue for now, but please let me know if you have any follow-up questions or would like me to keep the issue open.

Thanks!!