Running out of memory (and crash) by loading content on demand

[RatajVaver]

Describe the bug

This bug has been already documented in this issue: #966. Unfortunately author of that issue marked it as completed (because he found a workaround), although the bug was never fixed.

This bug is persisting for years and happens for every player on some servers that have this issue after longer sessions of playtime (i.e. 3 hours, but it differs mainly server to server and also client to client). I'm convinced this is not an issue for most servers but only a few, because otherwise I suppose it would have been solved a long time ago. However it's not a small issue, because for those few servers every single player experiences these crashes (besides for a few very unusual cases).

The issue is, when you use dxCreateTexture and similar functions very often and you're also unloading these textures properly later on (to avoid memory leaks), there are probably some leftovers in the memory that in the end cause memory leak by re-creating new and new textures, sounds, importing models, etc.

Only workaround (althought not very viable) is doing a memory leak intentionally (by not unloading textures and keeping them in memory even when they're not in use), which helps some players (crash happens later on), but also harms other players (crashes even more often).

There might be a memory leak in my server, althought I doubt it, since I was trying to find it for a very long time and I wasn't able to replicate this bug on my dev server with same scripts but different data (there just wasn't 100 players driving around with textured vehicles, as it is on my production server).

I've tried to find the problem in both showmemstat and performancebrowser, all with no success. Therefore I believe this is bug in MTA and how it handles freed memory. Maybe it's something with memory fragmentation? I have no idea, I don't know much about that. There's probably someone who can track this down and explain it better than me.

Steps to reproduce

Might be hard to reproduce, because it usually happens only on servers with a lot of custom player content. For example on RolePlay servers with a lot of custom interior textures, vehicle textures (remap), player radio stations, ped models. These things are loaded only when needed (e.g. vehicle streams in -> shader+texture is created, vehicle streams out -> shader+texture is deleted). When you do this a lot of times, you get to a point where debug outputs "not enough memory" and no new textures are being loaded, no new sounds from URL are being played, no new models are imported, even screenshots can't be taken. Then after few minutes there's the crash (well documented in #966).

It might however be reproducable by creating lots of textures, unloading them and loading them again for a while.

I've did some testing year ago, but I wasn't able to reproduce it on dev server at all. Maybe someone can. I'm creating this issue mainly to get some attention to it, since I can't find out enough information that could lead to solution.

Version

Every version to date (including nightly).

Additional context

No response

Relevant log output

No response

Security Policy

• I have read and understood the Security Policy and this issue is not security related.

[ffsPLASMA]

Can you give us an example on how you load and unload textures?
Also how many textures do you load at once, what sizes are they?

[RatajVaver]

Textures are loaded using dxCreateTexture and unloaded using destroyElement.
As an open-source example, this script uses very similar way of dealing with textures and is also causing this issue: https://github.com/OwlGamingCommunity/MTA/tree/main/mods/deathmatch/resources/item-texture

I believe it is however combination of more things that has to be loaded in memory, than just textures. If I was able to reproduce it on dev server, I'd try to pinpoint the problematic function, although I believe this is not really problem with any particular script or function, but maybe problem with how MTA handles memory. Of course I could be mistaken, this is just what I suspect.

Maybe there's a problem elsewhere on my server (and few others) and I'm just focusing too much on textures because it works with memory a lot. That's why I suspect it the whole time, also because that's what @brzys is saying in #966.

[Lpsd]

#2672 - related

[brzys]

Yup, dxCreateTexture has some kind of leak. My gamemode has DX-based UI destroying textures when they're not used anymore, so after 2-3 hours of gameplay MTA would crash at some point. The workaround was to cache textures without destroying them.

[JeViCo]

Yup, dxCreateTexture has some kind of leak. My gamemode has DX-based UI destroying textures when they're not used anymore, so after 2-3 hours of gameplay MTA would crash at some point. The workaround was to cache textures without destroying them.

Have you tried clearing cached textures over time? Does the problem still occur?

[JeViCo]

I'm assuming that it might be related to the image size as well. For example, we use 300x300 image and it stores as 512x512 tex (nearest power of two). That difference may break something at one point

[brzys]

Yup, dxCreateTexture has some kind of leak. My gamemode has DX-based UI destroying textures when they're not used anymore, so after 2-3 hours of gameplay MTA would crash at some point. The workaround was to cache textures without destroying them.

Have you tried clearing cached textures over time? Does the problem still occur?

I haven't really made any tests for reproducing this. My resources used a lot of ped renders with 512x384 textures, lets say players could create/destroy them ~5000 times an hour. Without destroying them overall client stability increased.

[JeViCo]

The only texture-related leak i know is shader+texture destroying order. You can make leak happen by creating both shader and texture, passing texture to any kind of shader as material and destroying the texture first. It happened to me several times on our dev server after 10+ resource restarts. Not sure that this happens on resource restart only. There's no way to log it on prod

[JeViCo]

The main problem of this issue is lack of stats/logger/counter that will allow to track down the problem source for sure. Current state of memstats does NOT answer these questions:

• Is that a model replacement issue?
• Is that a texture creation/deletion issue?
• Is that a streaming related issue?
  And so on

[Lpsd]

Adding proper memory leak diagnostics requires the client projects to be built with specific configurations, including its dependencies matching these configs (especially when it comes to runtime variations). This can be a struggle depending on the project you're trying to investigate (for example, try this on Client Deathmatch and see how far you get).

Reference: https://learn.microsoft.com/en-us/visualstudio/debugger/finding-memory-leaks-using-the-crt-library?view=vs-2022

FYI; whilst it doesn't help currently, this is something I have been working on for #2672 and I think a separate issue would be beneficial to discuss this topic overall.

[RatajVaver]

What we recently found out is that when we set Streaming memory to a third or half instead of max, it helps a lot.
I went from crashes every 2-3 hours to no crash at all during 11 hours playtime.
I find that interesting and worth sharing. Maybe this information can help with pinpointing the cause.

Of course this is not ideal solution, since it introduces some other problems.