Memory access fault by GPU node-1

[Schnurzel700]

OS: Arch Linux
CPU: Ryzen 7 5700x
Kernel: 6.18.3 and 6.12.63-lts (same result)
GPU: AMD RX 6800 XT (gfx1030, 16 GB VRAM)
RAM: 32 GB
Python: 3.12, 3.13 (same result)
ComfyUI: 0.8.2
Frontend: 1.36.13
PyTorch: 2.11.0.dev + ROCm 7.1
ROCm tested: 6.2, 6.4, 7.1

Model:
Model: sdxl_turbo_fp16
Resolutions: 512×512, 768×768, 1024x1024x

Reproduction:

python3 -m venv comfy-env
source comfy-env/bin/activate
pip install comfy-cli
pip install torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/rocm6.4
comfy --workspace=/home/user/repos/ComfyUI/comfy install

PyTorch and Python Versions i tried:

❯ python --version                                                                                 
Python 3.13.11
❯ python -c "import torch; print('PyTorch:', torch.__version__); print('ROCm:', torch.version.hip)"
PyTorch: 2.11.0.dev20260108+rocm7.1
ROCm: 7.1.52802

❯ python --version                                                                                 
Python 3.12.12
❯ python -c "import torch; print('PyTorch:', torch.__version__); print('ROCm:', torch.version.hip)"
PyTorch: 2.5.1+rocm6.2
ROCm: 6.2.41133-dd7f95766

Error:
Memory access fault by GPU node-1 (Agent handle: 0x577d0d3344f0) on address 0x730e41a5f000. Reason: Page not present or supervisor privilege.

ComfyUI runs fast, but reliably crashes the GPU with a memory access fault after repeated generations, especially when switching resolutions.
After this, GPU execution is broken until ComfyUI is restarted.
I also tried the manual Installation and Installtion with comfy-cli.

How can i solve this problem?

[Ultra119]

I use a 6800 xt and have been dealing with this problem on Arch for a long time (since rocm 5).
Just use officially supported distros. Preferably, of course, based on Debian/Ubuntu. I don't have this problem there.

[basicist]

Same issue here using Arch with a AMD 7800 XT

[Schnurzel700]

I just tested the issue again on a supported distribution (Ubuntu 24.04.3) and the problem does not appear there. Seems to be related to a too new kernel, which maybe changed VRAM Management Control.

[popsUlfr]

Not exactly sure if at all related but I got comfyui working smoothly again on my Strix Halo with CachyOS by using following rocm url: https://rocm.nightlies.amd.com/v2/gfx1151/
I wiped my venv and first thing I did after recreating it:

pip3 install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx1151/

Then install all requirements: pip3 install -r requirements.txt

And no crashes, memory access faults or segfaults anymore.

For gfx1030 it is most likely this:

pip3 install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx103X-dgpu/

[alexheretic]

I've been seeing this on Arch with my 7900gre. However, I found reverting to kernel 6.17.9 stopped it occurring so may be a different issue (or perhaps try that).

On arch you can downgrade the kernel using the archive, and for the kernel package this is a fairly ok thing to do unlike doing so with many other packages.

pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.17.9.arch1-1-x86_64.pkg.tar.zst

Related issues:

• [Issue]: Memory access fault by GPU node-1: amdgpu: [gfxhub] page fault ROCm/amdgpu#204
• [Issue]: Memory access fault by GPU node-1 (Agent handle: 0x5de975b3cc80) on address 0x799bfc063000 ROCm/legacy-rocm-build#5051 (comment)

[waydxd]

Downgrading the Arch Linux kernel to an older version (in my case linux-lts 6.12.66-1) resolved my memory access fault issues with the RX 6700 XT.

I also attempted @popsUlfr’s workaround by installing the ROCm nightly builds:

pip3 install --pre torch torchvision torchaudio \
  --index-url https://rocm.nightlies.amd.com/v2-staging/gfx103X-dgpu/

However, the memory access faults persisted on the latest kernel. The issue only disappeared after switching to the LTS kernel, so it seems both steps are necessary.

[Schnurzel700]

I already tried downgrading to Linux Kernel LTS-6.12.63, in my case it did not resolve the issue with my RX 6800 XT.

[Schnurzel700]

Which python version do you use?

[Schnurzel700]

I tried running the ROCm PyTorch installation command @popsUlfr shared, but it failed with a "no matching distribution" error. Here’s the output:
pip3 install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx103X-dgpu/ Looking in indexes: https://rocm.nightlies.amd.com/v2/gfx103X-dgpu/ ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch

[Ultra119]

I tried running the ROCm PyTorch installation command @popsUlfr shared, but it failed with a "no matching distribution" error. Here’s the output:
pip3 install --pre torch torchvision torchaudio --index-url https://rocm.nightlies.amd.com/v2/gfx103X-dgpu/ Looking in indexes: https://rocm.nightlies.amd.com/v2/gfx103X-dgpu/ ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch

There is no torch or other components for gfx1030 on rocm.nightlies.amd.com. They are distributed via https://download.pytorch.org/whl/rocm6.4 or https://download.pytorch.org/whl/nightly/rocm7.1/ if you need latest

[Schnurzel700]

Just tried Linux Kernel 6.17.9-arch1-1 as @alexheretic and @waydxd suggested, but still same Issue for me

[alexheretic]

Just tried Linux Kernel 6.17.9-arch1-1 as @alexheretic and @waydxd suggested, but still same Issue for me

Could be another issue with the same symptoms I guess. You definitely restarted after changing kernel I assume? (Can confirm with uname -r).

[TomatoLoli]

For those still experiencing this issue. Try the following launch options. python main.py --use-quad-cross-attention --disable-smart-memory --cache-none

[feffy380]

For those still experiencing this issue. Try the following launch options. python main.py --use-quad-cross-attention --disable-smart-memory --cache-none

I had luck with --gpu-only and I've never encountered this issue with A1111 or reForge, so it looks like comfyui's partial offload is triggering a bug in rocm 7

[Schnurzel700]

@alexheretic Yes, I rebooted every time after changing kernel
@TomatoLoli The Launch Command you provided did unfortunately not work for me. Same as --gpu-only. Still same issue.
I tried Python v3.14.2 and v3.12.12.