Add an ObjectDB Profiling Tool

[AleksLitynski]

Intro

This PR adds a new tab to the debug menu that can help profile a game's memory usage. It's a follow up to this proposal - godotengine/godot-proposals#10628.

Specifically, this lets you save a snapshot of all the objects in a running game's ObjectDB to disk. It then lets you view the snapshot and diff two snapshots against each other. This is meant to work similarly to Chrome's heap shopshot tool or unity's memory profiler.

UI Overview

The column on the left is universal functionality.

1. "Take ObjectDB Snapshot" button - When a game is being debugged, press this button to capture a snapshot
2. List of snapshots - A list of all the snapshots, stored in the user://objectdb_snapshots folder. Click a snapshot to view it. Right click a snapshot to rename, delete, or view in your file browser.
3. Diff menu - A dropdown containing all the snapshots. Select a snapshot to diff against.

The space on the right contains 6 tabs. Each tab shows the contents of the snapshot in a different way. All tabs support diffing in some form.

1. Summary view - Show a few blurbs about the snapshot, and highlight things that might be problems. Information includes:
   a. Total memory used
   b. Name and creation time of the snapshot
   c. Total object count
   d. Total node count
2. Classes view - Shows a tree of all the classes in the game. Select a class to see the instances of the class on the right. Select an instance to open it in the inspector view. Class tree can be filtered and sorted. When diffing, deltas in instances counts are shown next to each class, and instances from both snapshots are shown when a class is selected.
3. Objects view - Shows a table of every object in the game. Can be sorted and filtered. When an object is selected, it is displayed in the inspector and all inbound and outbound references are shown on the right. Click a reference to display the referenced object. When diffing, both snapshots are displayed in the same table. I'm not sure if this is particularly useful, but wanted to make every view work with diffing.
4. Nodes view - Shows the scene tree as well as any orphaned nodes. Can be filtered and sorted. Clicking nodes shows them in the inspector. When diffing, both trees are merged. Added nodes are marked in green, removed nodes are marked in red. Diff can also be set to side-by-side mode to see both trees next to each other.
5. RefCounted view - works basically the same as Objects view, but displays only ref counted objects, and only shows incoming references from other ref counted objects. It will highlight objects that are definitely trapped in reference cycles, but there's a very low hit rate on this because most references are owned by the engine and cannot be introspected on.
6. JSON view - Just dumps the whole snapshot to json. Diffing shows two jsons side by side.

Other thoughts

Other than a few fixes in core, most of the code is in a new module. The biggest change to core is adding a lock to Object::set so that objects cannot be modified when ObjectDB::debug_objects is running. I'm fairly new to C++, so I'm hoping someone can double check I'm using locks correctly there.

Snapshots can be in the 10-100s of megabytes, and the debug network peer has a maximum buffer size of 8mb, so I'm chunking the snapshots into smaller slices and sending them 6mb at a time.

Since snapshots can be pretty big, I store them in an LRUCache and only keep 10 in the editor's memory at a time. If it feels like selecting a snapshot for the first time is slow, that's probably why.

Also along those lines, I don't populate each tab until you actually click on it. It makes loading a snapshot a little faster, but slows down clicking on each tab. The Objects, RefCounted, and JSON tabs can be brutally slow.

In order to show snapshot objects in the inspector, I'm extending the EditorDebuggerRemoteObject class used in the scene tree's remote tab.

Snapshots have the .odb_snapshot file extension. The first line is a version number, and the second like is the output of calling Marshalls::variant_to_base64. I was going to save this as a .tres, but realized 99% of the data I was saving wasn't a resource, and Marshalls::variant_to_base64 was the easiest way I could find to get the data in and out of a string.

• Bugsquad edit, closes: Allow snapshotting the ObjectDB to help check for memory leaks godot-proposals#10628

[Calinou]

Tested locally, it works as expected on Windows 11 23H2. This is a great start 🙂

Some feedback:

• I'd make the snapshot store the exact Godot version (including commit hash) that was used to generate it, so that you can compare it in diffs. Use something like this to generate the stored version string:

godot/editor/project_manager.cpp

Lines 1463 to 1467 in c3e16cd

	String hash = String(VERSION_HASH);
	if (hash.length() != 0) {
	hash = " " + vformat("[%s]", hash.left(9));
	}
	version_btn->set_text("v" VERSION_FULL_BUILD + hash);

• The Snapshot Overview line shouldn't have an underline, as people can mistake it for a clickable link. Just use a bold font to indicate it's a heading.

• I noticed timestamp files are named according to the UTC time, not local time. This is inconsistent with log file naming which always uses local time.

• It might be a good idea to compress ObjectDB snapshot files as they can get quite large in projects with lots of objects. If I put a 3.2 MB snapshot in a ZIP, it's only 224 KB once compressed, which is a more than 10x compression ratio. FileAccess has compression methods available for this, along with the Compression singleton.

• Whenever I save a snapshot, I see this in the terminal output (not the editor Output panel):

Starting to snapshot
WARNING: Property not found: ProjectSettings
     at: ProjectSettings::_get (core\config\project_settings.cpp:351)
snapshot length: 7089

The unknown setting warning should be fixed, and I suggest moving the snapshot-related prints to print_verbose() so that they only appear in verbose mode.

• I got a sporadic crash when clicking the Filter objects field in the ObjectDB debugger. I haven't been able to reproduce it again though:

CrashHandlerException: Program crashed
Engine version: Godot Engine v4.4.dev.custom_build (036846c0284c4176ace06ca38d466bfb167618a1)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] StringName::~StringName (C:\Users\Hugo\Documents\Git\godotengine\godot\core\string\string_name.h:187)
[1] StringName::~StringName (C:\Users\Hugo\Documents\Git\godotengine\godot\core\string\string_name.h:187)
[2] EditorStringNames::~EditorStringNames
[3] EditorStringNames::`scalar deleting destructor'
[4] memdelete<EditorStringNames> (C:\Users\Hugo\Documents\Git\godotengine\godot\core\os\memory.h:140)
[5] EditorStringNames::free (C:\Users\Hugo\Documents\Git\godotengine\godot\editor\editor_string_names.h:45)
[6] unregister_editor_types (C:\Users\Hugo\Documents\Git\godotengine\godot\editor\register_editor_types.cpp:310)
[7] Main::cleanup (C:\Users\Hugo\Documents\Git\godotengine\godot\main\main.cpp:4522)
[8] widechar_main (C:\Users\Hugo\Documents\Git\godotengine\godot\platform\windows\godot_windows.cpp:184)
[9] _main (C:\Users\Hugo\Documents\Git\godotengine\godot\platform\windows\godot_windows.cpp:206)
[10] main (C:\Users\Hugo\Documents\Git\godotengine\godot\platform\windows\godot_windows.cpp:220)
[11] WinMain (C:\Users\Hugo\Documents\Git\godotengine\godot\platform\windows\godot_windows.cpp:234)
[12] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[13] <couldn't map PC to fn name>
-- END OF BACKTRACE --

• At the minimum bottom panel height, it's possible for the Objects panel to overlap the bottom of the editor:

• The Diff against dropdown should gray out the currently selected snapshot, as there's no point in comparing a snapshot against itself:

When selecting the snapshot that is currently being diffed against, the two should be swapped together (i.e. Diff against automatically changes).

• The JSON view should use the editor's code font, as this kind of data displays best with fixed-width fonts. See #89690 for an example of this, which also adds line numbers and syntax highlighting (there's a built-in JSON syntax highlighter available in the editor).

What does "RC" mean here? I'm guessing it means "reference count". I think we have the space to spell it out here:

• I wonder if tooltips can be added to each column in the Objects and RefCounted sections to explain some of the columns, especially In/Out in Objects and Total Refs/User Refs/User Cycles which are not obvious at a glance.

[Faless]

A couple of things to keep in mind:

• Editor code should go in a editor subfolder (inside the module folder) and only be included for editor builds (see modules/multiplayer/editor/ and modules/multiplayer/SCsub as example).
• My feeling is that we should not build the module at all in release builds. I feel like we should not bloat release builds with things that only matter in debug (same goes for things like Expression Evaluator (REPL support with inspector sequent results) #60134, but also the network profiler, etc).

[AleksLitynski]

I think this is ready for another review. Some notes on the changes I've made:

Fixed:

• Snapshots now include the version of the game client and the editor. Visible in the summary view.
• Removed underlines on summary page and added a little more whitespace.
• Timestamps are now in local time.
• Snapshots are now compressed. I compress them in the game itself so sending the snapshot from the game to the editor is faster.
• When adding compression, I removed the concept of a version number from the .odb_snapshot file. I was never sure how it should work, and compressing the file made it harder to imagine it being useful. My current thought is we can add a .odb_snapshot_2 file extension if we change the file contents.
• Log messages changed to print_verbose
• Fixed bottom panel overflowing on objects tab and refcounted tab. It was the content on the far right that forced the rest to overflow, so that content now scrolls when too small.
• Diff against works as described. Selected snapshot is greyed out and snapshots swap when you selected the currently diffed snapshot.
• The JSON view now uses the same visualizer as the native shader visualizer. Instead of copy/pasting all those properties, I refactored the native shader visualizer into a new control (editor_json_visualizer.h) that gets used in both panels.
• RefCounteds are weird, because the engine holds half the references, and the references in the ObjectDB can be exposed by multiple getters. I updated the RefCounted panel to make the data as clear as possible. I think this panel is only really going to make sense to people other than me after I write some documentation.
• Trees don't currently support tooltips on the title bar, so I added title bar tooltips to the tree control. It's working for my trees, but I'm shaky on if I handled all the measurements correctly when figuring out which column the mouse is hovering over.
• Used the new title tooltips to add tooltips to the object and refcounted views.
• @Faless - I think I have the build set up the way you described now. The module is only built for template_debug and editor targets. The editor side code should only be in editor builds, and the client code will be in both build types.
• I merged down from master, so no more conflicts for the moment.

Not Fixed/Need Guidance:

• I wasn't able to reproduce the WARNING: Property not found: ProjectSettings issue. I suspect this is specific to the project you're testing against.
• I also couldn't reproduce the crash. I suspect the snapshot had a weird character in one of the fields being filtered, but I'm not sure. It would help to have a copy of a snapshot that can cause this crash.
• I fixed only the TTR(...) strings you expressly tagged. I can go through and cleanup all the strings in the PR, but I need to know what changes to make.

Generally speaking, taking a snaphot means calling every single getter in a game. If any of them are buggy or do something surprising, the snapshot might fail or log errors like you saw in ProjectSettings. I'm not sure what to do about that. I can start asking people to send me copies of their projects and .odb_snapshot files, but that feels like an end problem, if people are even willing to send them to me. What's the right thing to do for bugs like that?

[AleksLitynski]

The latest build error I fixed had a similar stack trace to the sporadic crash @Calinou reported above. Not sure if it's the same root cause, but this commit may have fixed the filter box crash described above.

[Calinou]

Tested locally, it works as expected. Keep up the good work 🙂

I've noticed a few remaining UI issues:

• The line spacing in the bullet point list at the right of the RefCounted tab should be increased for more comfortable reading:

This issue isn't present in bullet point lists found elsewhere, such as the Summary tab.

• The JSON tab doesn't use a fixed-width font correctly, it uses the default project font instead:

Check where you are getting the editor's code font; it may be done too early and should be done in NOTIFICATION_THEME_CHANGED instead.

• Loading the JSON tab can take a long time (easily 10+ seconds on a high-end machine when comparing two snapshots on Truck Town), so I suggest displaying some kind of "Generating JSON, this may take a while..." text in the TextEdit(s) then actually putting the JSON text in place one or two frames later (call_deferred() can help here).

• In a future PR, we could add a keyboard shortcut that can be pressed in the running project to capture an ObjectDB snapshot (similar to how pressing F8 quits the game when the project window is focused). This would make it easier to capture snapshots during gameplay.

[mihe]

I think the last major thing to address is the orphan nodes being mixed with node tree. You could just put them into a parent item named "Orphan Nodes", and possibly give them some distinct icon (like MissingNode).

I've gone ahead and added this now, so there should be another tree item separate from the scene root called "Orphan Nodes" with the MissingNode icon, like so:

I wasn't sure whether to add it at the top or bottom. I figured keeping the scene root at the top made the most sense, but I'd be happy to change it around.

[mihe]

I went ahead and squashed all of my commits to a single one. We still need to squash the rest of them, but I figured that can wait until the merge, in case the original PR commit is of some use until then.

[Adrenesis]

I tested this PR on my game, and it allowed me to find every memory leaks in my game in less than 1 hour, so to me on a user point of view that's a big yes.

It's very efficient and well done. I also want to mention, that this is a very important feature for Godot to have. As far as I'm concerned stable release should implement it asap.

[AleksLitynski]

Hello! It's been a long time since I last looked at this PR, and I figured I would check in.

Is there anything I can do to help move it forwards? I don't think I was very helpful with the smaller cleanup work, but if there's any bigger units of work that have come up, I'd be happy to tackle them.

[KoBeWi]

The PR is already queued for 4.6, so you don't need to do anything.

[WhalesState]

Is there anything I can do to help move it forwards? I don't think I was very helpful with the smaller cleanup work, but if there's any bigger units of work that have come up, I'd be happy to tackle them.

Maybe we can keep this PR to just two commits: one for the Tree column title tooltip text and another for ObjectDB profiling.

[WhalesState]

Setting size flags are not needed for children of Control nodes. both can be removed.

[WhalesState]

Setting vertical size flags on HSplitContainer is not needed, the default fill is enough.

[WhalesState]

Setting horizontal size flags for VBoxContainer children is not needed.

[WhalesState]

Setting anchors preset is not needed for Container children.

[WhalesState]

This line can be removed.

[WhalesState]

This line can be removed.

[WhalesState]

This line can be removed.

[WhalesState]

Suggested change

	pc->add_theme_style_override("panel", sbf);
	pc->add_theme_style_override("panel", EditorNode::get_singleton()->get_editor_theme()->get_stylebox(SNAME("title_button_normal"), SNAME("Tree")));

sbf can be removed.

[WhalesState]

This line can be removed.

[WhalesState]

This line can be removed.

[WhalesState]

Note: All theme changes should be updated inside NOTIFICATION_THEME_CHANGED, and margins should be multiplied by ED_SCALE.

[Repiteo]

@WhalesState I'd encourage setting up a separate PR with those suggestions, as integrating this tool early in 4.6 (i.e. now) takes priority

[Repiteo]

Thanks! Congratulations on your first merged contribution! 🎉