ResourceLoader.load_threaded_request use_sub_threads = true causes Editor + Player to hang.

[cosmoddd]

Godot version

4.1

System information

Godot v4.1.beta3 - Windows 10.0.22621 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2060 (NVIDIA; 31.0.15.3203) - Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (12 Threads)

Issue description

When I use ResourceLoader.load_threaded_request(_scene, "", true) in 4.0.3 Stable, my set of three scenes loads in about five seconds.

When I use the same command in 4.1b3, to load the same data, the program hangs for roughly 3-5 minutes, or crashes. After spending time troubleshooting, I am left with the impression that the use_sub_threads feature seems to slow down the editor when there are heavier scenes to load.

I tried moving the level loading thread to a new thread group (the new 4.1 feature), but this threw a ton of errors dealing with the level loading being on the wrong thread.

Other things I've done to try to alleviate the issue:

• clean reimport of .godot project file
• clean export to standalone build in Windows
• disabled physics where possible

All to no avail. I'm including a smaller project so you can try to see what's happening during the ResourceLoader.load_threaded_request(level, "", true) command yourself.

I didn't change anything in my project - except upgrade to 4.1.

Possible regression?

Steps to reproduce

Download and extract the project: here

Import the project.

Load in a series of models that test the engine's ability to load a large scene.

Play the scene and notice the load speed timer in the output window.

On line 16 of BaseLevel.gd, compare the editor behavior for
ResourceLoader.load_threaded_request(level, "", true)
to
ResourceLoader.load_threaded_request(level, "", false)

Notice how there's a serious bottleneck when the use_sub_threads setting is set to 'true'.

Minimal reproduction project

Download and extract the project: here

[RandomShaper]

[...] compare the editor behavior [...]

From the GDScript in the MRP:

# setting the threaded_request ot true causes a bottleneck in the editor

I'm not sure I'm understanding the issue. The editor is not affected by that GDScript code, which is only executed at runtime.

I've tried anyway to trigger problems in multi-threaded loading, to no avail. I've even forced lower number of threads than actually on my system (12, to match yours; and even 4) and added an artificial delay to the load of each individual resource, to simulate a slower hard disk, to see if I could create some kind of deadlock.

So, I need more clues. It would be great to get minidumps at the moments of both the reported situations (locked for a long time and on the crash).

[cosmoddd]

Thank you for taking the time to prioritize this. Since it doesn't seem to be happening for you on the Minimal Reproduction Project, I've done two things.

The first is that I've included a crash dump from my most recent attempt to run my full project. I'm attaching it as a .zip below.

Error code:

Faulting application name: Godot_v4.1-rc1_win64.exe, version: 4.1.0.0, time stamp: 0x00000000
Faulting module name: ntdll.dll, version: 10.0.22621.1848, time stamp: 0x48d14984
Exception code: 0xc0000374
Fault offset: 0x000000000010be19
Faulting process id: 0x0x8238
Faulting application start time: 0x0x1D9A9BD89CFF0B2
Faulting application path: C:\Users\gregh\AppData\Roaming\Godot\app_userdata\Godot Version Manager\versions\Godot_v4.1-rc1_win64.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 366515d4-e472-4fb9-98be-6f9280d5634b
Faulting package full name: 
Faulting package-relative application ID: 

The file format is in Windows Event Viewer, .evtx and I've saved it inside a .zip file.

Download godot level load crash event files here

The second thing is I've included my project in full so you can see the issue happening directly.

Steps to reproduce:

1. Download and extract the project: here (Currently in 4.1Rc1)

2. Import the project.

3. Press F5 to start a new game.

4. Press two colors and enter your name and click "Proceed"

5. Notice the load speed timer in the output window.

6. Navigate to res://Scripts/SceneManagement/SceneManager.gd

7. On line 63, change
   ResourceLoader.load_threaded_request(level, "", false)
   to
   ResourceLoader.load_threaded_request(level, "", true)

8. Compare the speeds of how the level loads between the two modes.

I hope this provides more clarity to the issue. If this is still not giving you useful information, let me know.

[cosmoddd]

Here's a visual indicator of what happens when I load the game using both methods.
ResourceLoader.load_threaded_request(level, "", false)

ResourceLoader.load_threaded_request(level, "", true)

If you have a preferred method of getting the minidumps of these processes, let me know and I'll provide them.

[RandomShaper]

Thank you very much for the extra info, but sadly I'm still not able to reproduce the issue.

The best way of providing .dmp files would be to use WinDbg (unless you have it already installed, you can get it from https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/).

Once installed:

• Start the game.
• Start WinDbg (X64) from the Start menu.
• In the menu bar, File -> Attach to a Process. It's easier if you select By Executable and find the one starting with godot in the list (be sure to pick the one corresponding to the running game, not the editor).
• Exercise the game until you run into one of the pathological situations.
• If it's the crash, the debugger will likely break itself.
• If it's the deadlock, you'll have to choose Debug -> Break from the menu bar.
• Now, in the Command window, at the botoom, you can type commands. Type this one: .dump /mA C:\Users\<your_windows_user>\Desktop\godot.dmp and hit Enter.
• Now you should have the roughly 1.5 GiB file there. Please provide it to me via some appropriate file sharing method (ideally, one for the crash and one for the deadlock).

I know it's a lot to ask, but it will help immensely. If someone from the production team can reproduce, they may help obatining the minidumps.

[cosmoddd]

Thanks for getting back to me with detailed instructions on debugging this issue.

Using the method above, I was not able to get my build to crash, only lock up.

Here is the minidump file, compressed in a .zip. (lmk if you need me to post uncompressed)

Here is a gif of my screen following the steps given, including the errors thrown.

This error only occurred when ResourceLoader.load_threaded_request(level, "", true) was set to "true."

I appreciate the attention on this issue and I hope my information is useful.

[kaist-kihwan]

I have similar issue. When I use ResourceLoader.load_threaded_request(scene, "", true), at certain point, the progress stops lke below.

0.53125
0.54166662693024
0.69014418125153
0.86001598834991
0.86001598834991
0.86001598834991
0.86001598834991
0.86001598834991
0.86001598834991

[RandomShaper]

@cosmoddd, hmm... I was expecting the WinDbg you could download from there was the same I have on my machine (which is part of the Windows SDK), but it turns out it's a different one (maybe just newer?).

Thank you very much for providing the dump! It's actually a crash, by the way. If you can reproduce the stall at some point, please capture it, too.

Unfortunately, I didn't realize that we don't (and can't) have PDB files for official builds. However, I may still have a trick up my sleeve before having to ask you to get the minidumps from a custom build I would provide to you.

Aside, can you please try with the project setting threading/worker_pool/use_system_threads_for_low_priority_tasks being false. (@akien-mga, if that turns out to avoid the issues, we could at least document it as a workaround in the release notes in case I'm not able to do the proper fix, or make a PR that assumes the functional value of the setting.)

[univeous]

I'm having the same issue in Godot 4.1 rc1.
After disabling threading/worker_pool/use_system_threads_for_low_priority_tasks, ResourceLoader.load_threaded_request(level, "", true) can properly load without hanging, but it spits out some error logs about Attempting to initialize the wrong RID.
There are no such logs when ResourceLoader.load_threaded_request(level, "", false).

[cosmoddd]

I'd be happy to use a custom build if it gives you the info you need. Let me know what the preferred protocol is for sending that over.

As for setting threading/worker_pool/use_system_threads_for_low_priority_tasks to false, that did make a difference! My game's loading times behave as they did in version 4.0.3.

One thing of note:

When I start a new game using ResourceLoader.load_threaded_request(level, "", true)
and threading/worker_pool/use_system_threads_for_low_priority_tasks to false, these errors are thrown.

This is not a consistent error. It only seems to happen about 5 out of the past 10 times I load my scenes. Troubleshooting where exactly this happens in my code is proving difficult, since it only tells me the script which is the source of the error, but doesn't give a stack trace beyond the c++ level.

Update: I've traced the code where the error is thrown... ResourceLoader.load_threaded_request(level, "", true)