-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Issue]: Atomic optimizer reorder causes memory access fault in Blender #58
Comments
I can confirm that reverting that commit (30a3adf) fixes the crash with other scenes like classroom but not with blender-3.2 (https://cloud.blender.org/p/gallery/629f23f908e12d4ff15241d3) which still crashed with the similar error (happens only with rocm-6.x):
|
@LAKostis this is weird, because the blender-3.2 splashscreen renders fine for me on all the devices I've tested. Are you sure your Blender is not using your cycles cache? Maybe try clearing I also see that you seem to be using the Blender provided by your distro. For consistency, can you download the official binaries from Blender and try testing with that instead? |
Yes, I've specially tested this before rocm-6.0.x migration and this demo started failing only with rocm-6.0.x. Cache clearing doesn't help. It can be device specific issue - this demo crashes on my RX 6700 XT (gfx1031) but works on gfx900 (with rendering artifacts and only after setting HSA_ENABLE_SMDA=0) Regarding the blender build - I'm the blender package maintainer in this distro, so I'm sure what build options where used there :) For the sake of clarity official blender builds crash exactly the same way. I can provide any additional information or logs if you need any. |
Thanks for the quick response!
Just curious, does compiling against |
No, with HSA_OVERRIDE_GFX_VERSION=gfx1030 it doesn't start with error But interesting, it works with lowering -O level:
with
I'm using 6.6.25 kernel + patches up to v6.5-2638-gbf901afac5d5f from amd-staging-drm-next |
UPDATE: more funny things with compiler:
This command produces workable kernel despite of warnings about deprecated commands. But this command:
Produces kernel which crashes. And those kernels are not equal:
|
- Applied fixes for cycles: + cycles/hip: reduce opt level and enable hipcc-func-supp for gfx1031 kernel (see ROCm/llvm-project#58)
Problem Description
Following https://projects.blender.org/blender/blender/issues/112084, I've bisected the
rocm-6.0.x
branch and found that commit 30a3adf caused any Blender render (using HIP, of course) to crash with message along the lines of "Memory access fault by GPU node-1 (Agent handle: 0x7f1db8337e00) on address 0x7f1bf177e000. Reason: Page not present or supervisor privilege."Operating System
Solus 4.5 Resilience
CPU
AMD Ryzen 7 5800H with Radeon Graphics
GPU
AMD Instinct MI250, AMD Radeon VII
ROCm Version
ROCm 6.0.0
ROCm Component
llvm-project
Steps to Reproduce
Build this project at the commit mentioned.
Download the Blender 4.1 release binaries:
curl -O https://download.blender.org/release/Blender4.1/blender-4.1.0-linux-x64.tar.xz
,tar xf blender-4.1.0-linux-x64.tar.xz
. You should now have a folderblender-4.1.0-linux-x64
.Clone Blender. Just cloning the
v4.1.0
tag is enough:git clone https://projects.blender.org/blender/blender.git --depth 1 --branch v4.1.0
.In the Blender repo, compile the HIP fatbin used to run Blender render:
hipcc --offload-arch=$arch --genco intern/cycles/kernel/device/hip/kernel.cpp -D CCL_NAMESPACE_BEGIN= -D CCL_NAMESPACE_END= -D HIPCC -I intern/cycles/kernel/.. -I intern/cycles/kernel/device/hip -ffast-math -o kernel_$arch.fatbin
. AdjustHIP_ROCCLR_HOME
,HIP_CLANG_PATH
as necessary to point to the Clang you just compiled. Replace$arch
with the GPU architecture to run on, e.g.gfx900
orgfx1030
. Don't add extra attributes like:xnack-
.If you want to run on multiple architectures, repeat step 4 and 5 for each architecture.
Put this file into
blender-4.1.0-linux-x64/4.1/scripts/addons/cycles/lib/kernel_$arch.fatbin
.Get the BMW27 Blender demo file.
curl -O https://download.blender.org/demo/test/BMW27.blend.zip
,unzip BMW27.blend.zip
. You should have a fileBMW27.blend
.Now run Blender render.
blender-4.1.0-linux-x64/blender -b <path-to-BMW27.blend> -f 0 -- --cycles-device HIP
. By default it runs on GPU with device ID 0, so adjustHIP_VISIBLE_DEVICES
as necessary to run on the desired GPU.You should almost immediately see Blender crash with an error message similar to "Memory access fault by GPU node-1 (Agent handle: 0x7f1db8337e00) on address 0x7f1bf177e000. Reason: Page not present or supervisor privilege."
Now, build LLVM at 1 commit prior, e.g.
git switch --detach 30a3adf50e2d49dfc97c1b614d9b93638eba672d~1
. Repeat step 4-7, and Blender should render normally.All of this is on ROCm 6.0.0. If you get a hang instead of a crash when running Blender (likely your on an APU), Ctrl+C and run again with environment variable
HSA_ENABLE_SMDA=0
.(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
rocminfo --support
Additional Information
This behavior has been reproduced on MI250, RX6600M, Vega 10, and Ryzen 7 5800H. (gfx90a, gfx1032, gfx900, gfx90c, respectively)
Kernel version:
6.6.22-281.current
, with torvalds/linux@96c211f reverted (ref: https://lists.freedesktop.org/archives/amd-gfx/2023-October/100298.html and ROCm/ROCm#2596 (comment))The text was updated successfully, but these errors were encountered: