Memory issues in Final Fantasy XIV

[exolyte]

When there are a lot of other player characters or when LOD settings are turned off, framerate drops from 60 to ~10 and the game sometimes crashes. Running it with a debug build shows nine:device9:SetRenderTarget: User assertion failed: 'i != 0 || pRenderTarget'.
I've added some debug output and it seems like createrendertarget returns E_OUTOFMEMORY here. (It's called with D3DFMT_NULL) This causes NULL to be passed to setrendertarget.

GPU:RX570
Mesa version: git

Windows Apitrace

[exolyte]

Since the game doesn't allow me to login with a modified binary, I've modified wine to always use the LARGEADDRESSAWARE bit. This fixes all out of memory errors, framedrops and crashes so it does seem like malloc failing is the culprit here.
Weird because the game starts lagging/crashing on unmodified wine with only 1GB ram used so there should be enough left over even for a 32 bit app. I'm not really familiar with memory management, but could this be heap fragmentation?

[siro20]

Nine does allocate memory wine (and the game) doesn't know about. A long term goal is to use wine's memory allocator inside Mesa, to make the game aware that it runs out of memory soon.
Try to use the 64bit game version, as it's not limited to 2GiB of memory.

[exolyte]

The 64bit version is d3d11 only, which doesn't work very well. I played for 6 hours continously with LAA enabled and did not have any problems so that does seem to be a solution for now, albeit a hacky one.

If anyone else wants to try this, here is a wine patch to always use LAA. Note that this is not a proper implementation and will probably break other 32 bit apps so be careful.
https://gist.github.com/exolyte/a7761778da953b637e33e29c26814536

[iiv3]

@exolyte , would you explain in a little bit more detail how and where do you set LARGEADDRESSAWARE bit in wine?

I'm not really sure how changing wine allocation makes Nine native allocator not fail...
Is it possible that wine somehow claims most of the virtual space so native allocator cannot get what it needs? Linux malloc usually overcommits memory so it would never report failure...

[exolyte]

Only thing I did was compile wine with the patch linked above which removes the parts that restrict memory allocation to 2GB. Maybe this applies to nine as well? I have no idea how nine allocation limits work.