eGPU Optimizations

[rc345ct243tc]

I'm running a RX 6700XT on a thundebrolt eGPU dock (Razer Core X, PCIe 3.0 x4). My rest of my laptop's specs are i7 8550U, 16GB RAM, 1TB SSD. I'm running Fedora 35 with kernel 5.16 and Mesa 21.1. I'm using the latest version of Proton-GE.

I'm seeing extremely bad performance on some DX12 games compared to Windows. I've tested Forza Horizon 4 and Forza Horion 5 so far. On FH4, at 1080p on the very low preset, I get around 120 fps in the benchmark in windows, but I was getting around 50 fps on linux.

After doing some digging around on the internet, I found that disabling resizable bar/smart access memory based optimizations would help. A couple of related posts by another user that helped me 1, 2 and 3. My laptop doesn't support resizable bar and doesn't have any options in the bios about it either.

RADV_PERFTEST=nosam didn't help me, but then setting VKD3D_CONFIG=no_upload_hvv boosted the performance in the FH4 benchmark to 70fps. This is an improvement, but is still lagging behind windows by a lot.

I have a couple of question

1. Are there any other tricks/optimizations I can try to help my situation
2. Is there too much overhead from Proton/VKD3D that my system isn't able to handle? I doubt this is the case cause my CPU usage isn't anywhere close to 90% while running the game.
3. Is the way in which VKD3D works inherently going to cause problems on a thunderbolt eGPU setup.

[Blisto91]

Would you be able to update your drivers? Mesa 21.1 is getting quite old by now and optimizations might have made it in since.

Recently on vkd3d-proton master the VK_KHR_dynamic_rendering vulkan extension, which was first introduced in mesa 22.0, became a requirement.
All versions after GE-Proton7-9 have this. So you could try to see if GE-Proton7-9 makes a difference since it doesn't include those changes. (Tho i don't know if they would make performance worse for people with old drivers, i thought you wouldn't even be able to launch)

[rc345ct243tc]

I have tried it out on arch also, and that is running Mesa 22. Same performance issues on it. I seem to be using vkd3d 2.6, which doesn't have this.

Are you saying that this vulkan extension will help my situation or make it worse?

If you have any other thoughts about things I can try, please do let me know.

[Blisto91]

I have tried it out on arch also, and that is running Mesa 22. Same performance issues on it. I seem to be using vkd3d 2.6, which doesn't have this.

I don't know if vkd3d shows the git commit when using a non release version. But atleast a lot of ge protons use the latest git at time of release.
Roger to the same performance.

Are you saying that this vulkan extension will help my situation or make it worse?

Don't know. Don't have insight into that. I just know it is marked as a required extension after the stable 2.6 release for proper usage.

I'll let the experts take over from here as i don't have much input 😁

[doitsujin]

Are you saying that this vulkan extension will help my situation or make it worse?

It will not change anything specific to external GPUs.

In general, we cannot test this kind of setups, but as a D3D12 implementation we don't have much freedom in deciding how to move data around since the app is in charge of managing memory and uploading data, so I'm not sure if there's anything we could optimize on our end, and how much of this is up to the RADV side of things. We'll be making no_upload_hvv the default again however since the current behaviour causes more problems than it solves even on regular desktop PCs.

That said, DXVK is also expected to perform very poorly on eGPUs since we heavily rely on a low-latency data path between CPU and GPU there, and reengineering the whole thing to asynchronously DMA everything on the GPU is not a priority - too much work, can't test if it even improves anything, and the user base for which this is relevant is is probably in the double digits at best. With a setup like that, you'll want to stay on Windows for gaming purposes.

[rc345ct243tc]

Your explanation seems to make sense. Do you know if there is any test application that can be used to measure the memory copy overhead that these translation layers bring? I know windows has some standard apps to measure GPU memory read/write speeds, but is there something more specific to the DXVK/VKD3D way of doing things?

I wonder how much of this is due to vkd3d/dxvk vs the radv driver. I tried testing out amdvlk/amdvlk-pro, but things didn't seem to run properly with it at all. I will probably investigate this a bit further.

I wonder how things are on the nvidia side of things. I don't have an NVIDIA GPU currently, so can't really test that out.

On a related note, do you think force enabling GPU/CPU P2P DMA would help? I get a warning similar to this in dmesg

i915 0000:00:02.0: cannot be used for peer-to-peer DMA as the client and provider (0000:04:00.0) do not share an upstream bridge or whitelisted host bridge

It appears with both i915 and amdgpu listed as the driver logging the message. It seems like I can recompile the kernel by adding in the necessary PCI host bridge IDs over here. Tbh, I know nothing about what this actually does or whether it would actually work in my setup. But one can wonder.

[rc345ct243tc]

@doitsujin I have another data point and was wondering if you could explain this/give me your thoughts on this.

I ran the DX11 version of the Unigine Superposition benchmark at 1080p extreme. These are the (approx) scores I got.

Windows 10 DX11 - 6000
Windows 10 DXVK - 5600
Arch Linux (5.17, Mesa 22) DXVK - 2400

I am trying to think about how they relate to the overhead that you described that DXVK has. Shouldn't the same overhead be there on windows also then? Does this suggest that the problem in my case might be with the driver instead?

[K0bin]

Different Vulkan drivers, different compositors, there's the possibility that something in Wine regressed and makes this very slow...

[Blisto91]

Having googled a bit it does maybe seem like eGPU support isn't the greatest on linux. Not sure what is going on tho.
I'm guessing wine cpu or other overhead probably doesn't help in this regard and maybe mesa (if anything can be done there) haven't had much focus on optimizing for them. Dunno.

[rc345ct243tc]

I could theoretically test the mesa vulkan drivers on windows too, cause those exist too. That would rule out a few more variables. Not sure how good those drivers are though.

[rc345ct243tc]

Ok, apparently the radv mesa drivers on windows aren't in a functional state according to this, so I can't test that.

[shvetsnikita]

@doitsujin

DXVK is also expected to perform very poorly on eGPUs since we heavily rely on a low-latency data path between CPU and GPU there, and reengineering the whole thing to asynchronously DMA everything on the GPU is not a priority - too much work, can't test if it even improves anything, and the user base for which this is relevant is is probably in the double digits at best.

I wonder if anything has changed in your opinion regarding this topic, as handheld vendors have been pushing the eGPU angle for their products with the introduction of OCuLink ports and four-lane USB4 ports over the last few years.

Besides that, even regular consumer media (such as The Verge and LTT) are now pushing Linux as the default recommendation not only for those handhelds but even for gaming in general. So I do not think the eGPU user base interested in good vkd3d performance is in the “double digits at best” range anymore.

So, is there any chance someone could look into this further? Are there perhaps a few low-hanging fruits?

P.S. Come here because I have huge frame pacing issue in Resident Evil 4 Remake and Forza Horizon 4 on GPD Win Mini 2024 with Radeon RX 9070 (also Razer Core X). Proton 11 (Beta) and Mesa 26, but that probably doesn't matter.