Add in-editor playback/live preview for running scenes

[jcostello]

Describe the project you are working on:
3D FPS

Describe the problem or limitation you are having in your project:
Unable to inspect graphically how the game is working outside the gameplay view

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
Add the ability to see how the nodes are behaving in the scene editor while the game is running. Juan said that once the vulkan integration were done it would be feasible to implement. godotengine/godot#13354

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
Sync the scene editor view with the game play in order to see objects moving / interacting / being deleted / etc in the editor to let the dev inspect if everything is working fine

If this enhancement will not be used often, can it be worked around with a few lines of script?:
No script can do this

Is there a reason why this should be core and not an add-on in the asset library?:
The scene editor is core to the engine

[Calinou]

Isn't the editor camera replication sufficient for this? godotengine/godot#27742

It's available in 3.2 and later.

[jcostello]

@Calinou that its a helpful feature but for example im working in on a 3d shooter. It would be helpfull if I can play and in the meantime look at the editor to see everything from another perspective to see for example how the enemies are behaving. Do you think im missing something to do that with the current features of godot 3.2.x?

[KoBeWi]

Well, you could add a second camera/viewport to your game that would be monitoring enemies (something like split-screen). I think what is missing is being able to override specific camera. It only works for the main camera, so you'd need some special setup.

[jcostello]

@KoBeWi yeah the idea would be to have this as default. Doing a split screen only for debugging sound like a lot of work in order to have it move also

[Ermiq]

The lack of this functionality is quite disappointing. I can't inspect the objects that are created/instantiated/modified at runtime.
I've been trying to port my little space sim project from the other engine.

Issue number 1.
I randomly generate a solar system in the scene, and I see no planets in-game. Have they been created in the scene actually? I would just take a look at the scene editor to see whether the nodes appeared in the scene hierarchy. But I have to somehow make sure that they're created without an actual visual confirmation of it.
Issue number 2.
Okay, somehow I now certain that I did the scene instantiation properly and the nodes are there. But I still don't see them.
Since I use the procedurally generated meshes for the planets, it might be that there's something wrong with my mesh generation algorithm. But what exactly? Is the mesh even generated? Maybe it has no vertices, triangles, wrong material, shadows_only mode enabled? I would take a look at the scene editor to see the mesh node then and inspect it. But I can't.
Issue number 3.
Okay, now the meshes are generated fine. So, let's implement a LOD system and back face culling. In the other engine I could move the camera around the planet at runtime in the editor to see how's it going or actually fly around the planet in the game window and see how the back faces of the planet are not generated in the scene editor window in real time.
Sidenote: I need the wireframe rendering mode for this test. If I haven't google it already, I wouldn't know that the wireframe mode is somewhat available in the game view in Godot 3.x, but even then, it's kind of a trick rather than an 'officially supporred' functionality.

I've been playing around with Godot for only 2 days now, so I might miss some options available, but still, these are the problems that I have encountered and that wouldn't happen if the scene editor window was able to represent the runtime changes.

[timothyqiu]

@Ermiq

For issue number 1: When running the project in editor, you can click the "Remote" button on the Scene panel to switch the display to runtime scene tree. You can check the runtime hierarchy there.

For issue number 2: Clicking nodes in the runtime/remote scene tree will set the inspector to that node. It shows the current state of node in the running game. You can inspect the properties there.

[Zireael07]

@timothyqiu: Not all properties can be inspected in the remote tree/running game - I often discover that Materials or Meshes aren't. IIRC there's an issue open for that already.

[timothyqiu]

@Zireael07 Yeah, it's a bug if not shown. I fixed some cases that dynamically created resources not shown for 3.x in godotengine/godot#44657, but I'm not sure if it covers all the causes why the property is not shown 🤣

[Calinou]

Sidenote: I need the wireframe rendering mode for this test. If I haven't google it already, I wouldn't know that the wireframe mode is somewhat available in the game view in Godot 3.x, but even then, it's kind of a trick rather than an 'officially supporred' functionality.

This makes me think the live scene editing feature (which is enabled by default) should probably sync the first 3D viewport's debug draw mode with the running project. (I wouldn't tie it to the editor camera replication feature, as you may be willing to use the current project camera but with a debug draw mode enabled.)

[MickL]

I would like to add another point why this should be integrated: Opening a new window when in fullscreen on macOS will leave the editor and go back to the desktop. This has two downsides:

1. You are not able to see the output / debugger
2. It gets you out of the flow / focus that you have with the fullscreen mode (and is kind of annoying)

Therefore I would highly appreciate if it is possible to run the game / scene within the editor, as Unity does.

Video showing the problem: https://youtu.be/kZI56d0ddHo

Screenshot from Unity how it might look like instead:

[geekley]

A simple feature that would help as a first step towards this is improving the Project Camera Override mode to optionally allow to select remote objects by clicking them in game window.
I imagine this would be simple enough that it could even be back-ported to current editor versions with just a tool script.

As a native editor feature, here's some alternative ways this toggle could be implemented:

1. As a button next to the "Project Camera Override" icon (maybe only visible when it's enabled); or
2. As a menu checkbox option when you right-click the "Project Camera Override" icon; or
3. Automatically enabled when both "Project Camera Override" is enabled and "Remote" is selected in the scene tree view

Of course, 2 or 3 would require an appropriate tooltip in "Project Camera Override" button to help discover the feature.

A "live preview" mode like Unity's is still ideal, but this simple feature would help a lot as a stopgap, because at least selecting one among multiple similar-named objects for inspection would be much easier.

[Zireael07]

Project camera override AFAICT has nothing to do with this proposal...

[bodamat]

Would be good to have the option to change something in the remote tree by editor window also then only in the inspector. To move and see it visually is much easier than just in numbers. It shouldn't also remember the changes, the remote tree should work like this now.
It also will help to have 2 tabs: 1st is an editor with an open scene and 2nd is a game tab. As we see the layout of these tabs in the Unity picture above, will be also so convenient for making mobile games, especially in a vertical orientation.
This feature shouldn't replace running the game in the window. It would be better to have 2 different options like in Unreal.

[Norrox]

As @sosasees wrote in the other thread. Important to have the game window setting toggable in the settings if users want it in a tab or stand alone 🙂

[geekley]

Project camera override AFAICT has nothing to do with this proposal...

@Zireael07 My idea is a different method for achieving the same intent: better editor integration with running game by letting you select things in game window. I suggested it because it's (presumably) simpler to implement considering Godot's way of doing things. Project camera override is currently the closest thing we have to this feature request, since it allows the editor to navigate the running scene freely (although it uses the game window). The only problem is that it doesn't allow selecting when enabled. If it did, you'd have a way to pick runtime-instantiated nodes by clicking, which isn't possible now, since remote nodes don't appear visually in the editor. This would be equivalent of Unity's Scene View when play mode is enabled (you can click to select runtime-instantiated objects).

Sure, it's not ideal, but I suggested because it seems easier to implement and it would help a lot to at least have SOME way to do this. It's not quite what this proposal is asking, but it would be a step in that direction.
Intention is somewhat close enough (though you'd use game window instead of scene editor):

Add the ability to see how the nodes are behaving in the scene editor while the game is running

[Calinou]

@HgGamer Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.

[BrandonGillis]

Any news on implementing this feature ? Is there any blocker in the Core that make this feature not possible at the moment ?

[Calinou]

Any news on implementing this feature ? Is there any blocker in the Core that make this feature not possible at the moment ?

Implementing this with "in-process" running is probably not going to happen, as it's very difficult to do right and has many implications. For example, the game process crashing would also crash the editor, leading to possible data loss.

I think the only way to implement this while following the Godot philosophy is to perform fake run embedding. In other words, spawn a borderless window that matches the 3D viewport's size and is automatically hidden when the editor window isn't focused anymore. This way, the game process remains entirely separate from the editor like it is currently. This is non-trivial to do correctly, as you need to move/resize the window when the editor dock is resized (and when the editor window is moved/resized).

The alternative way to do this without sharing process space is to implement an IPC mechanism and send the running project's image to the editor. While you don't need to perform window manipulation anymore, there's a lot of complexity involved:

• You need to render the running project without spawning a window for it (off-screen rendering).
• You need to send the image to the editor at a high enough framerate, without noticeable compression artifacts. Even then, the lower framerate as a result of streaming data can make performance testing misleading.
• You need to send all inputs (mouse, keyboard and touch, gamepad forwarding isn't needed) to the running project.

[BrandonGillis]

Any news on implementing this feature ? Is there any blocker in the Core that make this feature not possible at the moment ?

Implementing this with "in-process" running is probably not going to happen, as it's very difficult to do right and has many implications. For example, the game process crashing would also crash the editor, leading to possible data loss.

I think the only way to implement this while following the Godot philosophy is to perform fake run embedding. In other words, spawn a borderless window that matches the 3D viewport's size and is automatically hidden when the editor window isn't focused anymore. This way, the game process remains entirely separate from the editor like it is currently. This is non-trivial to do correctly, as you need to move/resize the window when the editor dock is resized.

The alternative way to do this without sharing process space is to implement an IPC mechanism and send the running project's image to the editor. While you don't need to perform window manipulation anymore, there's a lot of complexity involved:

• You need to render the running project without spawning a window for it (off-screen rendering).
• You need to send the image to the editor at a high enough framerate, without noticeable compression artifacts. Even then, the lower framerate as a result of streaming data can make performance testing misleading.
• You need to send all inputs (mouse and keyboard, gamepad forwarding isn't needed) to the running project.

Thanks for the detailed answer !

Anyway, I don't really know the internals of godot core, but at the moment there's already the ability to see nodes tree remotely. Wouldn't it be faisible to use that remote node tree that is synced, to build a new "remote scene" inside the editor ?

[neikeq]

Here and in other engines I keep hearing the suggestion about running the game on a separate process and then displaying its image in the editor somehow. But I don't see any explanation on how the editor tools (Inspector, Scene Tree, Animation Dock, and a long etc) would interact with the game. Making them work with the game via IPC sounds like a huge increase in complexity. Even then I'm not sure if the result would be desirable; for example, our current remote scene tree and remote inspector have very noticeably low refresh rates. Would an implementation via IPC be able to achieve high refresh rates?

From my perspective, the two solutions I know are:

• Running the game in the editor process like Unity does. It would require a lot of work and changing bad code practices (e.g.: I don't know if singletons are compatible with this). Even then there are downsides we likely can't solve, like potentially crashing the editor, which might be more likely in our case than in Unity.
• Embedding the editor tools in the running game. This is the solution I gravitate towards for its simplicity. It would still require refactoring editor code to make sure it works in game mode (it likely shouldn't overwrite any project files/scenes/resources).

[BrandonGillis]

In my opinion, if it's done via IPC like the current remote node tree it wouldn't be that bad if the refresh rate is slow. The purpose of this feature for the majority of users (if I'm not wrong) is to make debugging easier.

For a first implementation, I think it would still be beneficial to the user even if the refresh rate is not that good.

I could be wrong though as this is a personal opinion, in that case maybe we could redefine a more indepth scope about what's expected with this feature

[Calinou]

Would an implementation via IPC be able to achieve high refresh rates?

godotengine/godot#29136 only refreshed at 15 Hz. I'm not sure how high it could have been bumped in theory, but I think going above 30 Hz on most hardware will prove challenging (especially at high resolutions). We could support rendering at half resolution to reduce the bandwidth requirement, still.

[BrandonGillis]

Describe the project you are working on: 3D FPS

Describe the problem or limitation you are having in your project: Unable to inspect graphically how the game is working outside the gameplay view

Describe the feature / enhancement and how it helps to overcome the problem or limitation: Add the ability to see how the nodes are behaving in the scene editor while the game is running. Juan said that once the vulkan integration were done it would be feasible to implement. godotengine/godot#13354

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams: Sync the scene editor view with the game play in order to see objects moving / interacting / being deleted / etc in the editor to let the dev inspect if everything is working fine

If this enhancement will not be used often, can it be worked around with a few lines of script?: No script can do this

Is there a reason why this should be core and not an add-on in the asset library?: The scene editor is core to the engine

If I'm not mistaken the initial proposal was to sync the remote scene in the editor.

Sync the scene editor view with the game play in order to see objects moving / interacting / being deleted / etc in the editor to let the dev inspect if everything is working fine

When reading your answers it looks like you speak about rendering the "running game" inside the editor, which is not the same as syncing the scene in the editor.

Remote node tree are already synced to the editor, the only missing pieces is rendering these remote nodes inside an editor scene to have a visual feedback of these remote nodes.

Video example of unity : https://youtu.be/lXunqoE3GPo

As we can see, on the left side we have the running game, and inside the editor we can see the "nodes" being modified in the scene.

[jcostello]

As the OP

Describe the project you are working on: 3D FPS
Describe the problem or limitation you are having in your project: Unable to inspect graphically how the game is working outside the gameplay view
Describe the feature / enhancement and how it helps to overcome the problem or limitation: Add the ability to see how the nodes are behaving in the scene editor while the game is running. Juan said that once the vulkan integration were done it would be feasible to implement. godotengine/godot#13354
Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams: Sync the scene editor view with the game play in order to see objects moving / interacting / being deleted / etc in the editor to let the dev inspect if everything is working fine
If this enhancement will not be used often, can it be worked around with a few lines of script?: No script can do this
Is there a reason why this should be core and not an add-on in the asset library?: The scene editor is core to the engine

If I'm not mistaken the initial proposal was to sync the remote scene in the editor.

Sync the scene editor view with the game play in order to see objects moving / interacting / being deleted / etc in the editor to let the dev inspect if everything is working fine

When reading your answers it looks like you speak about rendering the "running game" inside the editor, which is not the same as syncing the scene in the editor.

Remote node tree are already synced to the editor, the only missing pieces is rendering these remote nodes inside an editor scene to have a visual feedback of these remote nodes.

Video example of unity : https://youtu.be/lXunqoE3GPo

As we can see, on the left side we have the running game, and inside the editor we can see the "nodes" being modified in the scene.

As the OP I can confirm that this is the proposal original intention

[TheBricktop]

* **Embedding the editor tools in the running game.** This is the solution I gravitate towards for its simplicity. It would still require refactoring editor code to make sure it works in game mode (it likely shouldn't overwrite any project files/scenes/resources).

I think this is a winner in terms of achievieing the aforementioned function and avoiding too making too much changes to the core editor.

[YuriSizov]

Remote node tree are already synced to the editor, the only missing pieces is rendering these remote nodes inside an editor scene to have a visual feedback of these remote nodes.

Remote tree is only inspected in the editor. That "only thing missing" is pretty much the most complex part. It's practically the same as running the entire game/playing scene in the editor, which is not a trivial task. If you want to interact with your playing scene from the editor's rendered viewport, then it must be executed in the editor environment. And creating some kind of dumb copy of the remote scene tree that would be "display only" and void of any logic is probably a way harder task than that.

This is why those other proposals have been merged with this one. If we were to implement any of it, it would probably be one solution.

[Calinou]

@Aiden-Potter Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.

[Calinou]

Closing in favor of #7213, which provides a more detailed technical implementation.