-
Notifications
You must be signed in to change notification settings - Fork 0
MGOS System
"You cannot optimize what you cannot observe. And Unreal's garbage collector is, by default, invisible."
Unreal Engine's garbage collector is powerful, but it is a black box. It runs when it decides to run. It collects what it determines is unreachable. It takes as long as it takes. The developer's visibility into this process is limited to:
- A stat counter (
stat gc) that shows cycle time — but not what was collected, what grew, or what is suspicious. - Object count queries — but no trend analysis, no delta tracking, no historical context.
-
obj listconsole commands — but these are point-in-time snapshots with no comparison capability.
When performance problems arise, the question "is it the garbage collector?" is surprisingly hard to answer:
- Frame hitches: The GC ran during gameplay, took 8ms, blew the frame budget. But was that 8ms normal? Is it getting worse? Is it caused by object accumulation or just a large collection?
- Memory growth: Total memory is increasing over a play session. Is this a leak? Is it intentional caching? Is it streaming assets that have not been collected yet? Without per-class tracking and baseline comparison, you are guessing.
- PendingKill saturation: Objects flagged for destruction are not being collected. Is this because the GC has not run? Because something is holding a reference? Because the PendingKill queue is growing faster than collection can drain it?
Existing tools like Unreal Insights provide allocation tracking but not GC behavioral analysis. memreport -full provides a snapshot but not trends. Third-party profilers focus on allocation patterns, not collection patterns.
The fundamental gap is this: there is no tool that watches the garbage collector over time, builds a behavioral model, and classifies what it observes into actionable categories. Is this GC pattern stable? Is it accumulating? Does it look like a leak? Is it a burst clean-up that is healthy but expensive?
Without this observability, teams either ignore GC performance (and ship with hitches) or spend hours in manual investigation every time a performance regression appears.
MGOS (Memory and GC Observability System) is a pure observation layer over Unreal's garbage collector. It hooks into the engine's Pre-GC and Post-GC delegates, captures snapshots of the object population before and after each collection cycle, computes diffs, and classifies the resulting patterns.
MGOS is an observer. Never an actor. This rule is absolute:
- It NEVER forces a garbage collection.
- It NEVER modifies object flags (RF_NoDestroy, RF_RootSet, etc.).
- It NEVER alters object lifecycle in any way.
- It NEVER holds strong references to observed objects (which would prevent their collection).
- It NEVER creates significant load during GC callbacks.
Violating any of these would make MGOS a Heisenberg observer — changing the system it is trying to measure. The system is designed to be invisible to the garbage collector.
| Mode | Behavior | Default In |
|---|---|---|
| Off | Delegates registered but all callbacks early-return. Zero overhead. | -- |
| Passive | Capture core metrics only. No per-class iteration. Minimal overhead. | Shipping |
| Snapshot | Capture core + extended metrics. Top-K class counting. Moderate overhead. | Development |
| DeepTrack | Full analysis. Per-class heuristics, survivorship tracking, leak scoring. | Test |
The key design constraint: DeepTrack is never default in Shipping builds. It requires explicit opt-in. Passive mode is safe for production because it only queries aggregate object counts — it never iterates the object graph.
Mode changes are dynamic. You can switch from Passive to DeepTrack at runtime to investigate a suspected issue, then switch back. Downgrading mode does not clear accumulated data. Upgrading mode requires a warmup period before advanced classification engages.
Pre-GC Callback Post-GC Callback
+-------------+ +----------------------------+
| Snapshot A | | Snapshot B |
| (object | GC RUNS | (object counts post-GC) |
| counts | ==========> | |
| pre-GC) | | Diff = B - A |
| + StartTime | | Push to History |
+-------------+ | Classify Pattern |
| Emit Events |
+----------------------------+
All analysis happens in the Post-GC callback. The Pre-GC callback only captures a snapshot and a timestamp. This minimizes interference with the GC itself.
+------------------------------------------------------------------+
| MGOS System v1.0 |
| (Engine-Scope Subsystem) |
+------------------------------------------------------------------+
| |
| +-----------------+ +------------------+ +--------------+ |
| | Snapshot | | History Store | | Profile | |
| | Builder | | | | Engine | |
| | | | Circular buffer | | | |
| | Pre-GC: A | | N snapshots | | 6-state | |
| | Post-GC: B |---->| N-1 diffs |--->| classifier | |
| | Diff = B - A | | Moving averages | | Heuristic | |
| | | | Variance stats | | rules | |
| +-----------------+ +------------------+ +--------------+ |
| ^ | |
| | v |
| +-----------------+ +------------------+ +--------------+ |
| | Engine Hooks | | Baseline Mgmt | | Event Bus | |
| | | | | | | |
| | PreGC delegate | | Warmup -> Valid | | 8 delegates | |
| | PostGC delegate | | -> Stale cycle | | (4 native + | |
| | Travel hooks | | Per exec-mode | | 4 dynamic) | |
| +-----------------+ +------------------+ +--------------+ |
| |
| +-------------------------------------------------------------+ |
| | Config DA (auto-discovered) | |
| | | |
| | Thresholds | Window size | Tracked classes | Mode defaults | |
| +-------------------------------------------------------------+ |
+------------------------------------------------------------------+
The Profile Engine evaluates GC history and classifies the current behavioral pattern into one of six states:
+----------+
+-------->| Stable |<--------+
| +----------+ |
| | |
| v |
| +-----------------+ |
| | Accumulation |------+
| +-----------------+
| |
| v
| +-----------------+
+-----| Leak Suspected |
+-----------------+
+------------------+ +-------------------------+
| Burst Clean | | PendingKill Saturation |
+------------------+ +-------------------------+
+------------------+
| Root Expansion |
+------------------+
| State | What It Means |
|---|---|
| Stable | Object population is within expected bounds. GC duration variance is low. No growth trend. |
| Accumulation | Objects are being created faster than they are collected. Total count is trending upward. May be temporary (streaming) or concerning (growing leak). |
| Leak Suspected | Accumulation with no return to baseline. High survivorship score. This is a high-probability inference, NOT proof of a leak. |
| Burst Clean | An unusually large collection just occurred. Many objects destroyed in one cycle. Often healthy (level unload) but worth noting for frame budget. |
| PendingKill Saturation | Objects flagged for destruction are accumulating. The PendingKill queue is growing faster than the GC drains it. May indicate a retention problem. |
| Root Expansion | The estimated root set is growing. Objects that cannot be collected are increasing. Combined with no-return to baseline, suggests persistent references are accumulating. |
Important caveat: These classifications are behavioral inferences, not causal diagnoses. "Leak Suspected" means "the pattern is consistent with a leak." It cannot tell you which object is holding the reference. It cannot trace the reference chain. It detects the symptom, not the cause. Use it as a signal to investigate with targeted tools, not as a definitive answer.
Classification requires a baseline — "what does normal look like?" The baseline system manages this:
+----------------+ +--------+ +-------+ +-------+
| Uninitialized |---->| Warmup |---->| Valid |---->| Stale |
+----------------+ +--------+ +-------+ +-------+
^ |
+---------------------------+
(recapture)
- Uninitialized — No baseline captured yet. Advanced classification disabled.
- Warmup — Collecting initial cycles. Waiting for variance to stabilize.
- Valid — Baseline is representative. Classification is active.
- Stale — A significant event (level transition, travel) invalidated the baseline. Recapture needed.
Baselines are separated by execution mode (PIE vs. Standalone vs. Shipping) because editor objects contaminate PIE baselines.
In DeepTrack mode, the system tracks specific classes declared in the configuration asset and computes per-class health metrics:
| Metric | What It Measures |
|---|---|
| Growth slope | Rate of instance count increase over the observation window |
| Over-baseline ratio | Percentage of recent cycles where count exceeds baseline |
| Peak deviation | Maximum departure from baseline in the observation window |
| No-return flag | Whether the count has ever returned to baseline within the window |
Per-class health follows its own state machine: Stable, Accumulating, Persistent Over Baseline, Leak Suspected. If any tracked class reaches Leak Suspected, the global state escalates.
The system also captures process memory statistics from the OS on each GC cycle. This provides visibility into memory growth that is invisible to the GC — native allocations, third-party libraries, and custom allocators. The baseline includes a process memory component, and the profile engine can raise incidents when process memory diverges from the UObject-based model, suggesting non-GC-managed growth.
A single configuration asset controls all system behavior:
Mode Settings:
- Default operational mode per build configuration
- Whether DeepTrack is allowed in Shipping builds (off by default)
Classification Thresholds:
- Accumulation rate threshold
- Leak suspicion threshold
- Burst clean Z-score threshold
- PendingKill saturation threshold
- Stability variance threshold
- Confirmation cycle count (how many consecutive cycles before a state change is committed)
Observation Window:
- Window size (number of recent cycles to consider)
- Test window size (shorter window for automated testing)
- Warmup cycle count (cycles before baseline is valid)
Per-Class Tracking:
- List of tracked classes (for DeepTrack per-class heuristics)
- Excluded classes (always ignored, e.g., editor-only types)
- Top-K count (how many top classes to report in Snapshot mode)
Baseline:
- Baseline capture strategy (after warmup, at startup, manual, after event)
- Epsilon values (absolute and relative tolerance for noise filtering)
Anti-Noise:
- Minimum PendingKill count before saturation analysis activates
- Loading phase suppression flag
- Inter-cycle monitoring interval (periodic check between GC cycles)
A dedicated editor panel with four sections:
Overview — Current operational mode, classification state with color coding, baseline state, cycle count, and process memory. The classification state is prominently displayed: a green "Stable" means no concerns, a yellow "Accumulation" means watch closely, a red "Leak Suspected" means investigate.
Snapshot History — Scrollable timeline of recent GC cycles showing duration, objects destroyed, PendingKill delta, and classification at each point. Sparkline graphs visualize trends over the observation window.
Per-Class Report (DeepTrack only) — Table of tracked classes with their current health state, growth slope, over-baseline ratio, and peak deviation. Sorted by severity so the most concerning classes appear first.
Incident Log — Chronological list of raised incidents with severity (Info / Warning / Critical), category, description, and cycle ID. Incidents are only raised after the confirmation cycle count is met, so this log reflects genuine state changes, not noise.
8 commands for runtime diagnostics:
| Command | Purpose |
|---|---|
pgx.mgos.status |
Current mode, state, baseline, cycle count |
pgx.mgos.mode |
Get or change operational mode |
pgx.mgos.baseline |
Show baseline state or trigger recapture |
pgx.mgos.history |
Recent GC cycle history with diffs |
pgx.mgos.profile |
Current classification state with confidence |
pgx.mgos.incidents |
Recent incidents sorted by severity |
pgx.mgos.classes |
Per-class health report (DeepTrack only) |
pgx.mgos.suppress |
Toggle suppression (pauses classification without mode change) |
MGOS runs as an engine-scope subsystem. This means:
- It initializes before any game world exists.
- It survives level transitions, travel, and world teardown.
- It observes GC cycles regardless of which level is loaded.
- It remains active for the entire engine session.
This is essential because garbage collection is a global engine operation. A world-scoped observer would miss GC cycles during transitions and would lose its history every time a level changes.
The platform profile system can constrain MGOS behavior:
- Low-spec platforms might restrict to Passive mode even in Development builds.
- High-spec platforms might allow Snapshot mode in Shipping for telemetry.
- DeepTrack is always gated by explicit configuration, regardless of profile.
MGOS binds to engine travel delegates (PreLoadMap, PostLoadMapWithWorld, OnWorldCleanup) to mark baseline as Stale during transitions. This prevents false positive leak detection caused by the massive object delta during level changes.
Because MGOS runs at engine scope (before any game world), Blueprint access is intentionally limited. The key query and control methods are available through the standard framework Blueprint access patterns, organized into Query, Control, and State categories.
8 delegates (4 native, 4 dynamic) allow other systems to react to GC events:
- GC Started — A collection cycle is beginning. Useful for pausing time-sensitive operations.
- GC Completed — A cycle finished. Includes the snapshot diff (how many objects changed).
- Profile State Changed — The classification changed (e.g., Stable to Accumulation). This is the primary signal for automated responses.
- Incident Raised — A new incident was committed after confirmation. Useful for logging, telemetry, or automated alerts.
Garbage collection performance is a silent killer of frame rates. Unlike rendering performance (which is visible in GPU profilers) or physics performance (which shows up in CPU profilers), GC performance is ephemeral. A bad GC cycle happens, causes a hitch, and leaves no evidence unless you were already looking for it.
Most teams discover GC problems in one of two ways:
-
QA reports frame hitches — But which hitches are GC-related? Without instrumentation, you cannot tell. You might spend days optimizing rendering when the problem was a 12ms GC cycle triggered by object accumulation.
-
Memory profiling reveals unexpected growth — But is it a leak or intentional? Without a baseline and trend analysis, you cannot tell. You might spend days hunting a leak that is actually a streaming cache working as designed.
MGOS provides the missing observability layer. It watches the garbage collector continuously, builds a behavioral model, and raises specific, actionable signals. Not "something is wrong" but "object accumulation has exceeded the threshold for 15 consecutive cycles, concentrated in these 3 classes, with no return to baseline."
The system is production-safe because it defaults to Passive mode in shipping builds. The classification is honest about its limitations — inference, not causality. And the observer-only design means it cannot make things worse.
For teams that ship live games, the incident system provides the foundation for automated telemetry. A "Leak Suspected" incident in production telemetry is a specific, trackable signal that can trigger investigation before players report the symptom.
- Development Preview
- Getting Started
- Release branch catalog
- Public Plugin Matrix
- Early Preview Plugins
- Known Issues
- Architecture Overview
- Plugin Topology
- Module Reference
- Configuration and Registry
- Data-Driven Design
- Profiles and Budgets
- Gameplay Tag Architecture
- Initialization Pipeline
- Cross-Plugin Communication
- Message System
- Event Handlers
- Logging and Trace
- Runtime Flows
- Blueprint API Design
- Editor Integration
- Editor Visual System