Node2Ds, Node3Ds, relate to each other's hierarchies inconsistently with Viewports

[thygrrr]

Godot version

4.2.dev 4.1.3.stable

System information

Windows 11

Issue description

2D and 3D layer mask handling for viewports is highly inconsistent.

For 2D, a logical AND is performed for hierarchy AND viewport culling mask.
For 3D, hierarchy doesn't matter, as long as the object matches the camera's culling mask.

godot.windows.editor.x86_64.mono_GzW18YAya0.mp4

This means handling child scenes that contain 2D elements that should both render in the viewport and outside of it is very tedious in code (selective reparenting, and subsequent manual transform copying each tick).

It also means 3D objects need to be treated entirely differently in their respective hierarchies.

Consistent behaviour would be intuitive, either that only children of a viewport are visible ("2D behaviour extended to 3D"), or - preferred - viewports render 2D according to their own culling mask, like a 3D camera also does ("3D and 2D culling made consistent")

Note: The font weight/blending bug mentioned in the 2nd label will got its own issue, #84787 (it's probably very unrelated)

Steps to reproduce

1. start sample project
2. observe: viewport is flashing, and with it all the things that are culled by the viewport
3. observe: 3D is flashing whether it's inside or outside the viewport hierarchy
4. observe: the orange label is absent from the running view, it should be flashing (OR the text 3D Layer 2 A should be invisible, as well)

Minimal reproduction project

viewport-chaos-repro.zip

[vpellen]

I think I ran into this today. I was trying to render a tilemap in its entirety on its own to try and deal with some weird inter-tile blending issues. Ran into several convolutions. First, subviewports automatically create their own world2d, which meant that placing the tilemap under the main viewport made it invisible to the subviewport, and making it a child of the subviewport meant that it had no physical interaction with objects in the main viewport. I was able to solve both of these issues by explicitly overwriting the subviewport's world2d to share the same world as the viewport, but I have no idea if this has other negative side-effects. Even when I did that, I ran into the "Logical And" issue, meaning that if I wanted the tilemap to only render to the subviewport, I had to make it a child of that subviewport, which, while not crippling, introduces a bunch of really ugly organizational issues.

I really wish there was a specialized Render-to-texture Camera2D node that I could just pass into a texture explicitly. The subviewport organization is hell because it breaks the hierarchy so badly. Sometimes I really do just want to take a restricted render of a scene without having to completely build my node hierarchy around it.

Incidentally, I've also encountered inverse issues with wishing I could create an isolated world2d or world3d without having to explicitly create viewports. The linking of visuals and physics is extremely obnoxious. It makes me feel like there should be an untangling of physics/hierarchy and rendering. But I guess that's probably a whole other issue.