Misc. bug: Ryzen 395 AI Max, vulkan is seeing vram from system memory and not dedicated gpu memory

[Goldenkoron]

Name and Version

Version: llama-b6869-bin-win-vulkan-x64 (and other recent versions at least)

When UMA is set to 64gb GPU memory, Llama-cpp sees 64GB, however when UMA is set to 96gb for iGPU, Llama-cpp can only see and use 32gb.

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

No response

Command line

Problem description & steps to reproduce

With a Ryzen 395 AI Max on windows, test the command 'llama-server --devices-list' and see how much vram it shows for Radeon 8060S.

First Bad Commit

No response

Relevant log output

[0cc4m]

Please upload the output of vulkaninfo on your system to a gist and link it here.

[Goldenkoron]

Please upload the output of vulkaninfo on your system to a gist and link it here.

https://gist.github.com/Goldenkoron/c4077621e935bf38358c7131325e816e

[0cc4m]

I assume that vulkaninfo output is when the VRAM size is set to 64 GB. Can you do another one with 96GB?

[Goldenkoron]

I assume that vulkaninfo output is when the VRAM size is set to 64 GB. Can you do another one with 96GB?

Here is with 96gb dedicated https://gist.github.com/Goldenkoron/ccb7e7fcf92de7ddd3c6208765d06480

Here is also the result of --list-devices in llama-server, notice the mismatch with rocm0. 110gb is actually correct when including system shared memory, and it goes around 96gb with 64gb set since 32gb will be allocated to shared memory.

[0cc4m]

Thank you, now I see the problem. With your 64 GB setup AMD provides one big 64 GB memory heap:

	memoryHeaps[1]:
		size   = 68719476736 (0x1000000000) (64.00 GiB)
		budget = 65283502080 (0xf33333000) (60.80 GiB)
		usage  = 0 (0x00000000) (0.00 B)
		flags: count = 2
			MEMORY_HEAP_DEVICE_LOCAL_BIT
			MEMORY_HEAP_MULTI_INSTANCE_BIT

But when you set it to 96GB, it gets split up:

	memoryHeaps[0]:
		size   = 34359738368 (0x800000000) (32.00 GiB)
		budget = 32641751040 (0x799999800) (30.40 GiB)
		usage  = 0 (0x00000000) (0.00 B)
		flags: count = 2
			MEMORY_HEAP_DEVICE_LOCAL_BIT
			MEMORY_HEAP_MULTI_INSTANCE_BIT
[...]
	memoryHeaps[2]:
		size   = 68719476736 (0x1000000000) (64.00 GiB)
		budget = 65283502080 (0xf33333000) (60.80 GiB)
		usage  = 0 (0x00000000) (0.00 B)
		flags: count = 2
			MEMORY_HEAP_DEVICE_LOCAL_BIT
			MEMORY_HEAP_MULTI_INSTANCE_BIT

We currently only consider the first one we find, that's why it reports 32GB. I'll think about a good way to solve that.

[arch-btw]

I'm having a similar issue but with a different 96GB ram linux system, I'll just add my output maybe that helps:

./llama-cli --list-devices

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  Vulkan0: AMD Radeon 780M Graphics (60016 MiB, 57015 MiB free)

vulkaninfo

VkPhysicalDeviceMemoryProperties:
=================================
memoryHeaps: count = 2
        memoryHeaps[0]:
                size   = 31465668608 (0x753800000) (29.30 GiB)
                budget = 29892384768 (0x6f5b99800) (27.84 GiB)
                usage  = 0 (0x00000000) (0.00 B)
                flags:
                        None
        memoryHeaps[1]:
                size   = 62931337216 (0xea7000000) (58.61 GiB)
                budget = 59784769536 (0xdeb733000) (55.68 GiB)
                usage  = 0 (0x00000000) (0.00 B)
                flags: count = 2
                        MEMORY_HEAP_DEVICE_LOCAL_BIT
                        MEMORY_HEAP_MULTI_INSTANCE_BIT

dmesg | grep -i "amdgpu.*memory"

amdgpu: amdgpu: 1024M of VRAM memory ready
amdgpu: amdgpu: 89000M of GTT memory ready.

[0cc4m]

Please check if #17110 fixes your problems.

[arch-btw]

Thank you so much @0cc4m , it's correct now:

Available devices:
  Vulkan0: AMD Radeon 780M Graphics (90024 MiB, 85522 MiB free)

However, it does still say this: shared memory: 32768 is that normal? Sorry, I'm just confused with the terminology and where that 32gb number comes from.

[0cc4m]

It's actually 32kb and an internal number relevant to kernel launches.