Instancing certain scenes in threads is extremely slow

[kb173]

Godot version:
3.2.stable.official

OS/device including version:
Ubuntu 19

Issue description:
When calling load(path).instance() in a thread, I would expect the performance to be similar as outside of a thread, maybe slightly slower. With certain scenes, this is the case. For example, instancing a scene with a simple Node or a Spatial only takes 1.5x-2x longer in a thread than on the main thread.

However, with other scenes, this is not the case at all. Instancing a scene with, for example, a Node2D or a MeshInstance in the main thread is completely fine and fast, but doing so in a separate thread takes about 100x as long as in the main thread!

In addition, I have noticed that calling wait_to_finish() on such a thread is completely blocking, seemingly forever.

Steps to reproduce:
Instance a scene with a MeshInstance or a Node2D (likely others too, have only tested a few) in a thread and measure the time it takes with OS.get_ticks_usec().

Minimal reproduction project:
slow-thread-instancing.zip

Start the project and check the logs. On my machine, the second iteration ("Slow scene", "Threaded") takes much longer, about 100x as long.

[volzhs]

similar report with 2.x #16769

[Calinou]

This may be solved in the master branch thanks to the new built-in threaded loading, see #36640.

[Zireael07]

I thought of something else, are all the affected nodes visual (as in, they draw something to screen - both MeshInstance and Node2D do)? In that case, it might be shaders compiling....

[kb173]

@Zireael07 I think you could be onto something there! I did some more testing and noted which nodes were expectedly fast in a thread, and which exhibited the described behavior of being ~100x slower than in the main thread:

Fast:

• AudioStreamPlayer
• ARVRAnchor
• Listener
• Timer
• Tween
• WorldEnvironment
• Position3D
• ResourcePreloader

Slow:

• CanvasLayer
• Viewport
• Sprite3D
• Sprite
• CSGTorus
• Control
• Light2D
• Camera
• DirectionalLight

Seems like everything which has a Material is slow, as are some nodes which interact with visuals, such as cameras and lights. The only thing that surprised me is that the WorldEnvironment is fast when loaded in a thread, even with a PanoramaSky with a custom texture. But I suppose that, as opposed to lights and cameras, the sky doesn't have to interact with a shader?

But even if the reason is shaders compiling - why is it so extremely slow when threaded?

[Calinou]

But even if the reason is shaders compiling - why is it so extremely slow when threaded?

OpenGL isn't thread safe and doesn't offer much in the way of parallelism, so I suppose it has to move back to the main thread constantly or something, causing a slowdown. This should be solved in the master branch thanks to Vulkan, which much more parallelism-friendly. Also, shaders are compiled on startup in the master branch and will be cacheable in the future.

[IllusiveS]

Bump, i am currently trying to move loading scenes to a thread and due to that issue it is insanely slow.
EDIT: using 3.2.1.stable

[norlowski]

I can confirm this issue in 3.2.3 64 bit (non-mono). I made a minimal example repo. I output to a log file to capture timing while instancing scenes. If I should include this information in a different way, please let me know.

https://github.com/norlowski/GodotThreadLoadExample

[norlowski]

I can confirm this issue in 3.2.3 64 bit (non-mono). I made a minimal example repo. I output to a log file to capture timing while instancing scenes. If I should include this information in a different way, please let me know.

https://github.com/norlowski/GodotThreadLoadExample

TLDR, corroborating what @kb173 noticed, sprite3D definitely seems to have issues. I stopped there but it is pretty easy with this minimal repo to try out other nodes. You should be able to just start the project and look at your user:// for a log file. or view the example log file I included.

[MathiasBaumgartinger]

Unfortunately still seems to be broken in 4.0 beta 1, we just ran into the same issue there.

[Calinou]

Unfortunately still seems to be broken in 4.0 beta 1, we just ran into the same issue there.

Can you upload the scene in question here (or at least mention which node types are part of the instanced scene)?

[kb173]

Unfortunately still seems to be broken in 4.0 beta 1, we just ran into the same issue there.

Can you upload the scene in question here (or at least mention which node types are part of the instanced scene)?

(Responding for @MathiasBaumgartinger since we're working on the same project)

It's definitely related to the existence of a SpatialMaterial (or now StandardMaterial3D) in that scene. For example: instancing a scene with a MeshInstance in which the material_override is set is very slow on a thread. However, instancing the exact same scene with an empty material_override, and instead with material_override = preload(material_path) in its _ready function, is much faster. Only when the material is pre-set in the scene is it unexpectedly slow.

I didn't make exact benchmarks but it's really a drastic difference. We could get the loading time (in a thread) from over 10 seconds down to less than 1 second just by removing the StandardMaterial3D from the scene and setting it in _ready instead.

So just to make it clear again if someone stumbles across this issue in the future (until it's fixed): you can work around this problem fairly easily by:

1. Removing materials from the scene in question, and
2. Loading those materials in the scene's _ready function instead