Importing directory of images over-extends Godot's RAM usage

[CitrusWire]

Tested versions

4.2.2

System information

Windows 7

Issue description

If you import a directory with lots of images, Godot will "helpfully" try and do it in parallel; makes sense for most game assets. However, the import process for even one large image file uses huge amounts of RAM, so when Godot tries to do multiple files in parallel, it can easily end up using far more RAM than the system has.

Godot does this self-induced lack of RAM well. It can crash Godot (not predictably reproduceable), crash Windows (not predictably reproduceable), and do weird stuff to the interface after the import is completed (below; not predictably reproduceable).

Steps to reproduce

1. Create a new Project

2. Import a directory of large images. In my case I'm using about 22 Hubble Pictures
   

3. Godot Analyses the images, using 12GB or so of RAM.

If I double the number of images (copy/paste onto themselves), sure enough, Godot now uses 20GB of RAM, which is more than the 16GB on my system, so lots of paging and Bad Things (tm) can happen.

I expect if I had a directory filled solely with a couple dozen of the very largest of those images, Godot would do awful things to my RAM. Someone else can test that. :-)
You can get some Hubble pictures from here if you're struggling for large images to test against: https://hubblesite.org/images

Minimal reproduction project (MRP)

N/A

[aaronp64]

It looks like most of the memory usage is related to Godot saving its own compressed version of the images when importing. By default, its compressing to WebP format. If you enable Project Settings/Rendering/Textures/Lossless Compression/Force PNG while importing, it will use PNG instead. This used a little under half as much memory as WebP when importing one image for me, and even less when loading multiple files at once. There's also Project Settings/Editor/Import/Use Multiple Threads - turning this off will avoid trying to import all of them at once.

I think most of the memory usage (especially for WebP) is from the third party libraries we're using, though it does look like we might be creating an extra copy or two within Godot that we could avoid, I can look more into that side of it.

[CitrusWire]

That definitely sounds like a promising course of action, but I was more wondering if it'd be possible to better balance the import scheduler so it didn't demand to do all of the heavy stuff simultaneously.

Just a guess, but if RAM use can be reduced at least 5x for this processes (again, just a guess!) the balancing may not be necessary, as I guess (yet again) not too many people use lots of over-sized images in their projects?

It also occurs to me: Until it's fixed, given how easy it is to trigger, it may be a good way of testing for the crashes and other weird behaviour I experienced after running out of RAM. But I know nothing about C++ or Godot dev and if that would be worthwhile, just a notion.

[huwpascoe]

How big are those images, in terms of width x height? It'd be useful to know how much memory they use uncompressed (an UHD rgba bitmap for example is like 32MB)

[CitrusWire]

The largest one by filesize (pillars of creation):

So IrfanView uses 168MB for it and loads it in less than a second. Godot uses about 2GB for the import process for the same fine.

[huwpascoe]

Looked with memory profiler and there's a large amount of mystery malloc. Why?

paraphrasing from 🐄 cowdata.h:

// Speed is more important than correctness here, do the operations unchecked
// and hope for the best.
return next_po2(p_elements * sizeof(T))

So, if you ask for a 137mb buffer of memory, it's gonna allocate 256mb.

Whether it's a good design or not, I don't know, I'm no memory expert.

[clayjohn]

There is definitely something wrong here, Godot must be holding on to memory that it doesn't need during the import process.

By my estimate all the textures in the folder in the OP should add up to about 1GB (uncompressed and with mipmaps). Its understandable to have some excess memory usage (perhaps 3-5 times the combined size of the images), but 12x is a bit much.

I tested just importing https://stsci-opo.org/STScI-01EVT2R4M58JN5WT7CQ998A5VD.jpg alone and it used about 1.5 Gbs of memory by itself. The uncompressed texture with mipmaps is 187mb.

As other comments have said, this overhead seems to come almost entirely from WebP as using the lossy method uses much less memory and using the VRAM uncompressed option uses very little.

Here is a graph of memory use:

The very small blip on the left is from importing as VRAM uncompressed. The bump in the middle is using WebP-lossy (~600 mb). The much longer and bumpier bump (with three separate levels) comes from WebP-lossless. You can see it allocates a similar amount of memory to lossless and then spikes near the end.