Blender Cycles engine causes "Memory access fault by GPU node-1"

[widarr]

Problem Description

Error:
Blender crashes with Memory access fault by GPU node-1 (Agent handle: 0x7c1aea898400) on address 0x5fd000. Reason: Page not present or supervisor privilege.

Operating System

6.7.4-2-MANJARO Linux

CPU

AMD Ryzen 9 5950X

GPU

AMD Radeon Pro W7900

ROCm Version

ROCm 6.0.0

ROCm Component

No response

Steps to Reproduce

Reproducing error: Just opening Blender (tested 4.0-4.1), switching Render Engine to Cycles (Feature Set: Supported, Device: GPU compute) Activate Viewport shading (and maybe rotate the camera around the default cube a few times) causes the crash. Also trying to render anything instantly crashes Blender.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module is loaded

HSA System Attributes

Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES

==========
HSA Agents

---

Agent 1

---

Name: AMD Ryzen 9 5950X 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD Ryzen 9 5950X 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3400
BDFID: 0
Internal Node ID: 0
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131808628(0x7db3d74) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 131808628(0x7db3d74) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131808628(0x7db3d74) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 2

---

Name: gfx1100
Uuid: GPU-9e8b159eef31d025
Marketing Name: AMD Radeon Pro W7900
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 6144(0x1800) KB
L3: 98304(0x18000) KB
Chip ID: 29768(0x7448)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1760
BDFID: 3584
Internal Node ID: 1
Compute Unit: 96
SIMDs per CU: 2
Shader Engines: 6
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 528
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 47169536(0x2cfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 47169536(0x2cfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1100
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***

Additional Information

OS:
NAME="Manjaro Linux"

CPU:
model name : AMD Ryzen 9 5950X 16-Core Processor

GPU:
Name: AMD Ryzen 9 5950X 16-Core Processor
Marketing Name: AMD Ryzen 9 5950X 16-Core Processor
Name: gfx1100
Marketing Name: AMD Radeon Pro W7900
Name: amdgcn-amd-amdhsa--gfx1100

Mobo: ASUSTeK model: ROG STRIX X570-E GAMING WIFI II

Graphics:
Device-1: AMD Navi 31 [Radeon Pro W7900] driver: amdgpu v: kernel
Display: x11 server: X.Org v: 21.1.11 with: Xwayland v: 23.2.4 driver: X:
loaded: amdgpu unloaded: modesetting,radeon dri: radeonsi gpu: amdgpu
resolution: 3840x2160~144Hz
API: EGL v: 1.5 drivers: radeonsi,swrast platforms: x11,surfaceless,device
API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 23.3.5-manjaro1.1
renderer: AMD Radeon Pro W7900 (radeonsi navi31 LLVM 16.0.6 DRM 3.57
6.7.4-2-MANJARO)
API: Vulkan v: 1.3.276 drivers: amd surfaces: xcb,xlib

RAM (DDR4)
Memory: total: 128 GiB

PSU: 1000W

Case: EATX with 9 Fans

[twoexem]

I have this issue too on my AMD RX6600XT.

[Jmvars]

I have this issue on my RX 7900 XT. Viewport shading works fine, Rendered View or trying to render anything causes instant crash with this error message.

[Kagukara]

Getting the same on 7900 XTX, ran gdb blender-4.2 and it gave me:

Thread 130 "blender-4.2" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	     return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;

backtrace:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007fffe98ab393 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007fffe985a6c8 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007fffe98424b8 in __GI_abort () at abort.c:79
#4  0x00007fff1f023514 in rocr::core::Runtime::VMFaultHandler (val=<optimized out>, arg=<optimized out>)
    at /usr/src/debug/hsa-rocr/ROCR-Runtime-rocm-6.0.0/src/core/runtime/runtime.cpp:1429
#5  0x00007fff1f07f642 in rocr::core::Runtime::AsyncEventsLoop () at /usr/include/c++/13.2.1/bits/stl_vector.h:1125
#6  0x00007fff1f027a6c in rocr::os::ThreadTrampoline (arg=<optimized out>) at /usr/src/debug/hsa-rocr/ROCR-Runtime-rocm-6.0.0/src/core/util/lnx/os_linux.cpp:80
#7  0x00007fffe98a955a in start_thread (arg=<optimized out>) at pthread_create.c:447
#8  0x00007fffe9926a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[fililip]

Does running Blender with HSA_ENABLE_SDMA=0 work?

[Kagukara]

Running HSA_ENABLE_SDMA=0 blender-4.2 gives me the same result.

Memory access fault by GPU node-1 (Agent handle: 0x7c32f76c4e00) on address 0x7c315dde2000. Reason: Page not present or supervisor privilege.
Aborted (core dumped)

[twoexem]

Yeah, same here:
Memory access fault by GPU node-1 (Agent handle: 0x76eb18475600) on address 0x984000. Reason: Page not present or supervisor privilege.

[twoexem]

My dmesg output shows the following:

[18786.415357] amdgpu 0000:2d:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32799, for process blender pid 36725 thread blender:cs0 pid 36744)
[18786.415359] amdgpu 0000:2d:00.0: amdgpu:   in page starting at address 0x0000000000989000 from client 0x1b (UTCL2)
[18786.415361] amdgpu 0000:2d:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000000
[18786.415362] amdgpu 0000:2d:00.0: amdgpu:      Faulty UTCL2 client ID: CB/DB (0x0)
[18786.415364] amdgpu 0000:2d:00.0: amdgpu:      MORE_FAULTS: 0x0
[18786.415365] amdgpu 0000:2d:00.0: amdgpu:      WALKER_ERROR: 0x0
[18786.415366] amdgpu 0000:2d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[18786.415367] amdgpu 0000:2d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[18786.415369] amdgpu 0000:2d:00.0: amdgpu:      RW: 0x0

[TNT3530]

Can confirm this also happens on Ubuntu 22.04 with kernels 6.5.0-14, 6.5.0-28, 6.8.1-060801, 6.7.10-060710, and 6.6.22-060622 when using Instinct MI100s and ROCm 6.0

[GZGavinZhao]

Hi, I think this may be a compiler bug, further investigation still needed: https://projects.blender.org/blender/blender/issues/112084#issuecomment-1157532

[GZGavinZhao]

If you're getting this only on ROCm 6, git bisect shows it's likely a compiler bug. Filed ROCm/llvm-project#58.

[alexxu-amd]

Hi all, can you check if this issue can still be reproduced with the latest ROCm 6.1?

[GZGavinZhao]

You can still reproduce the crash with a slight caveat. The reproducing step with Docker is as follows:

1. Pull the rocm/rocm-terminal:6.1 docker image.
2. Enter the docker image: podman run -it --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/rocm-terminal:6.1. I personally use podman, if you use docker replace podman with sudo docker.
3. Create a work directory for convenience. mkdir work && cd work
4. Install runtime dependencies for Blender: sudo apt-get update && sudo apt-get install build-essential git git-lfs subversion cmake libx11-dev libxxf86vm-dev libxcursor-dev libxi-dev libxrandr-dev libxinerama-dev libegl-dev libwayland-dev wayland-protocols libxkbcommon-dev libdbus-1-dev linux-libc-dev unzip libsm6 libxext6 -y. (yes, not all of these dependencies are needed, I'm just being safe here)
5. Fetch the Blender 4.1 binary: curl -O https://download.blender.org/release/Blender4.1/blender-4.1.0-linux-x64.tar.xz
6. Extract Blender: tar xf blender-4.1.0-linux-x64.tar.xz. You should now have a blender-4.1.0-linux-x64 directory in you current directory.
7. Fetch the BMW27 blender project for testing: curl -O https://download.blender.org/demo/test/BMW27.blend.zip
8. unzip BMW27.blend.zip. You should now have a BMW27.blend file in your current directory.
9. Render the file. blender-4.1.0-linux-x64/blender -b BMW27.blend -f 0 -- --cycles-device HIP. Note that by default Blender will use your GPU with id 0 to render; if you want to use another GPU, run export HIP_VISIBLE_DEVICES=<gpu-id-you-want-to-use> before rendering with Blender. You can find the available GPU ids by running rocm-smi (under the Device column).
10. You should now almost immediately get Memory access fault by GPU node-1 (Agent handle: 0x7f5f2af98000) on address 0x7f5afa7cb000. Reason: Page not present or supervisor privilege.

The caveat is that this issue is probably on the compiler side, not ROCm's runtime. Blender ships pre-compiled GPU fatbins that is compiled with some ROCm version that I don't know (I suspect the fatbins that come with 4.1 are compiled with ROCm 6.0, but even the Blender folks aren't sure). To verify that it's probably a compiler issue, continue:

11. Clone Blender. To save space and ensure consistency, we're just going to clone the 4.1.0 version: git clone https://projects.blender.org/blender/blender.git --depth 1 --branch v4.1.0
12. Enter the Blender folder and compile the fatbins with ROCm 6.1: pushd blender && hipcc --offload-arch=$arch --genco intern/cycles/kernel/device/hip/kernel.cpp -D CCL_NAMESPACE_BEGIN= -D CCL_NAMESPACE_END= -D HIPCC -I intern/cycles/kernel/.. -I intern/cycles/kernel/device/hip -ffast-math -o kernel_$arch.fatbin && popd. Replace $arch with the architecture of the GPU you want to run on, e.g. gfx900. If you want to compile against multiple architectures, you must repeat step 12 and 13 for each architecture.
13. Put the blender/kernel_$arch.fatbin file at blender-4.1.0-linux-x64/4.1/scripts/addons/cycles/lib/kernel_$arch.fatbin. Once again, replace $arch with your desired GPU architecture.
14. Now run step 9 again. Blender shouldn't crash anymore.

If you're unable to reproduce the crash with the BMW27 example, try any one of the demo files under the "Cycles" section on the Blender demos page. One common test file we used was the Classroom example.

[alexxu-amd]

@GZGavinZhao Thanks for the detailed description. We will look into it

[jeancf]

I am affected by this bug also with 6.8.8-2-MANJARO + ROCm 6.0.2 + Blender 4.0 + Radeon RX 6800XT

inxi -SCG
System:
  Host: desktop1 Kernel: 6.8.8-2-MANJARO arch: x86_64 bits: 64
  Desktop: KDE Plasma v: 5.27.11 Distro: Manjaro Linux
CPU:
  Info: quad core model: Intel Core i7-6700K bits: 64 type: MT MCP cache:
    L2: 1024 KiB
  Speed (MHz): avg: 800 min/max: 800/4200 cores: 1: 800 2: 800 3: 800 4: 800
    5: 800 6: 800 7: 800 8: 800
Graphics:
  Device-1: AMD Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] driver: amdgpu
    v: kernel
  Display: x11 server: X.Org v: 21.1.12 driver: X: loaded: amdgpu
    unloaded: modesetting,radeon dri: radeonsi gpu: amdgpu resolution:
    1: 2560x1440 2: 1280x1024
  API: EGL v: 1.5 drivers: kms_swrast,radeonsi,swrast
    platforms: gbm,x11,surfaceless,device
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.0.2-manjaro1.1.1
    renderer: AMD Radeon RX 6800 XT (radeonsi navi21 LLVM 16.0.6 DRM 3.57
    6.8.8-2-MANJARO)
  API: Vulkan v: 1.3.279 drivers: radv,llvmpipe surfaces: xcb,xlib