-
Notifications
You must be signed in to change notification settings - Fork 0
Data Registry
O(1) lookup. 18 validation rules. CI-ready. The index your project is missing.
Every Unreal Engine project of meaningful scale accumulates hundreds of Data Assets. Item definitions, enemy configurations, weapon stats, quest data, loot tables. They live in scattered Content directories, organized by convention, discovered by hope.
When your game needs to answer the question "give me the item definition tagged Weapon.Sword.Iron," one of three things happens:
-
Full Asset Registry scan. The code queries the Asset Registry for all assets of a given type, iterates the results, loads each one, checks its tag, and returns the match. This is O(n) at best -- scanning hundreds of assets to find one. It happens every time. There is no cache unless someone builds one.
-
Hardcoded soft references. Someone puts a direct asset path in a config struct or Blueprint variable. This works until the asset moves, the path changes, or a team member reorganizes the Content directory. Then it silently fails at runtime because soft references do not validate at edit time.
-
Custom lookup tables. Each team builds their own tag-to-asset mapping. The inventory system has one. The ability system has another. The quest system has a third. None share infrastructure. None validate against each other. Duplicate tags across tables are discovered at 2 AM when QA reports that the wrong item spawns.
The secondary problem is category queries. "Give me all weapons" or "give me all items in the Consumable category" requires scanning the entire collection. There is no secondary index. Adding one means building custom infrastructure per-system.
And the validation problem: there is no automated way to verify that your data assets are well-formed. Missing tags, null references, duplicate entries, orphaned assets -- these are discovered at runtime or, worse, in the shipped build.
The Data Registry provides a central index with O(1) lookup by GameplayTag, category-based secondary indexes, and a validation pipeline that catches problems before they reach runtime.
v2.0 introduces DataTable-first authoring: instead of creating individual Data Assets for registry entries, you author data in DataTables (which the engine natively supports with row editing, CSV import/export, and diffing), then group them into logical databases via Registry Definitions. The subsystem ingests these tables at initialization and builds both index layers automatically.
Content Browser (visual authoring)
|
v
+--------------------+ +--------------------+
| DataTable A | | DataTable B |
| (Weapons category) | | (Armor category) |
| - Row per item | | - Row per item |
| - Tag, Name, Ref | | - Tag, Name, Ref |
+--------------------+ +--------------------+
| |
+----------+---------------+
|
v
+--------------------------------------+
| Registry Definition DA |
| - Database Tag: "Items" |
| - DataTables: [A, B, ...] |
| - Category Whitelist (optional) |
| - Conflict Policy: FailOnConflict |
+--------------------------------------+
|
v
+--------------------------------------+
| Data Registry Subsystem | <-- GameInstance scope
| |
| Primary Index (Composite Key) |
| +--------------------------------+ |
| | DB + Category + Item --> Entry | | O(1) lookup
| +--------------------------------+ |
| |
| Secondary Index (Category Buckets) |
| +--------------------------------+ |
| | DB + Category --> [Items] | | O(1) + linear in category
| +--------------------------------+ |
| |
| Legacy Database Layer |
| +--------------------------------+ |
| | DB Tag --> Database object | | v1.0 compatibility
| | (auto-discover, manual register)| |
| +--------------------------------+ |
+--------------------------------------+
Dual indexing makes both point queries and range queries fast:
-
Primary Index: A composite key (Database Tag + Category Tag + Item Tag) maps directly to a registry entry. Finding "the Iron Sword in the Items database, Weapons category" is a single hash lookup.
-
Secondary Index: A hash of (Database Tag + Category Tag) maps to a bucket containing all items in that category. Finding "all Weapons in the Items database" is one hash lookup plus a linear scan of the category.
Conflict policies prevent silent data corruption when two DataTables register the same item tag:
| Policy | Behavior |
|---|---|
| Fail On Conflict | Log an error and reject the duplicate. The first registration wins. Data integrity over convenience. |
| First Wins | Silently keep the first registration. Useful when override layers should not replace base data. |
| Last Wins | Silently replace with the latest registration. Useful when override layers should replace base data. |
DataTables are the engine's native tabular data format. They support:
- Row-by-row editing in the Details panel
- CSV import/export for bulk authoring
- Diff-friendly text format for version control
- Blueprint and C++ row access
PGX extends the standard DataTable with metadata: which database it belongs to, what category it represents, and a human-readable description. These extended DataTables are authored in the Content Browser like any other asset.
A Registry Definition groups multiple DataTables into a logical database. One Definition might combine a "Core Weapons" table, a "DLC Weapons" table, and an "Event Weapons" table into a single "Weapons" database. The subsystem ingests all tables from all definitions at initialization.
Given a tag, the registry returns the asset reference immediately. No scanning. No iteration. The composite key index is built once at initialization and queried in constant time thereafter.
For typed resolution, the registry validates that the returned asset is of the expected type. A request for a weapon definition that accidentally points to an armor definition returns null with a diagnostic message rather than a silent type mismatch.
Registry entries store soft references to assets. Nothing is loaded into memory until explicitly requested. This means a registry of 10,000 items consumes memory only for the index metadata, not for the assets themselves.
When an asset is needed:
- Async load: Request the asset with a callback. The system loads it in the background and fires the callback when ready. Preferred for gameplay code.
- Sync load: Load the asset immediately on the calling thread. Acceptable during initialization or editor code. Avoid in hot paths.
- Cache query: Check if the asset is already loaded without triggering a load.
Cache invalidation releases all loaded assets in a database without unregistering entries. The index remains intact; assets simply need to be loaded again on next access.
The validation pipeline catches data problems before they reach runtime:
| Category | Rules | What They Catch |
|---|---|---|
| Schema (3) | Missing required fields, invalid row structure, type mismatches | |
| Tag Integrity (3) | Empty tags, malformed tag hierarchies, tags outside allowed branches | |
| Duplicate/Conflict (3) | Duplicate item tags across tables, cross-database conflicts, policy violations | |
| Asset References (4) | Null references, broken soft references, redirector chains, missing assets | |
| Runtime Quality (3) | Excessive entry counts, oversized databases, performance-impacting configurations | |
| Policy/Documentation (2) | Missing descriptions, undocumented conflict policy choices |
Each rule produces an issue with a severity level (Info, Warning, Error) and a human-readable message explaining the problem and how to fix it.
For common issues, the validation pipeline can automatically apply fixes:
- Null reference removal: Entries pointing to deleted assets are removed from the DataTable.
- Redirector resolution: Entries pointing through asset redirectors are updated to point directly to the final asset.
Autofix operations use the engine's transaction system for full undo support. Nothing is modified without the developer's confirmation.
A commandlet runs the full validation pipeline from the command line:
UnrealEditor-Cmd.exe MyProject -run=RegistryValidation [-strict]
Exit code 0: All validations passed
Exit code 1: Errors found (or warnings in strict mode)
This integrates directly into CI/CD pipelines. A failing validation blocks the build before bad data reaches a packaged build.
Validation results export to JSON (for CI consumption) and CSV (for spreadsheet review). Reports are written to a consistent directory for easy collection by build systems.
| Field | Purpose |
|---|---|
| Database Tag | Identifies the logical database (e.g., "Items", "Quests", "Enemies") |
| DataTables | Array of DataTable references to ingest |
| Category Whitelist | Optional: only ingest rows matching these category tags |
| Conflict Policy | How to handle duplicate item tags across tables |
| Setting | Purpose |
|---|---|
| Scan Roots | Content directories to scan for Registry Definitions |
| Entry Budget | Maximum entries per database (performance guard) |
| Strict Mode | Treat warnings as errors during validation |
A two-pane inspector:
+------------------+------------------------------------------+
| Databases | Entries |
| | |
| > Items (847) | [Search: ________] [Filter: Weapons] |
| Quests (203) | |
| Enemies (156) | Tag Category Name |
| Loot (89) | Sword.Iron Weapons Iron Sword |
| | Sword.Steel Weapons Steel Sword |
| | Shield.Wood Shields Wood Shield |
| | ... |
+------------------+------------------------------------------+
| Status: 1295 entries across 4 databases | Memory: 2.3 MB |
+----------------------------------------------------------+
Left panel: all registered databases with entry counts. Right panel: entries for the selected database, with search and category filtering. Status bar shows aggregate statistics.
+------------------+----------------------------+-------------+
| Definitions | Issues | Details |
| | | |
| > Items | [E] RDT020: Duplicate | Rule: |
| Quests | tag "Sword.Iron" | RDT020 |
| > Enemies | [W] RDT030: Null ref in | Severity: |
| | row 47 | Error |
| | [I] RDT051: Missing desc | Fix: |
| | on definition | Remove |
| | | duplicate |
| | [Validate All] [Autofix] | row from |
| | [Export JSON] [Export CSV] | TableB |
+------------------+----------------------------+-------------+
Left: Registry Definitions to validate. Center: issues found, color-coded by severity. Right: details for the selected issue with remediation guidance.
An Asset-Registry-powered view of all Data Assets in the project, grouped by system. Shows which systems have configs, which are missing, and provides one-click navigation to edit any config DA.
pgx.registry.list List all databases with entry counts
pgx.registry.stats Detailed statistics per database
pgx.registry.export Export metadata for a single database (JSON)
pgx.registry.export.all Export metadata for all databases (JSON)
pgx.registry.ingest Re-ingest all registry definitions
pgx.registry.validate Run validation on registry definitions
pgx.registry.definitions List active registry definitions
Every PGX subsystem that uses Data Assets registers a database during its initialization:
System initializes
|
v
Create database with system's DA type
(auto-discover = true)
|
v
Asset Registry scan finds all DAs of that type
|
v
Each DA registered with its tag as the key
|
v
System can now resolve any of its DAs by tag in O(1)
This means the Audio system resolves sound definitions by tag, the Save system resolves save configs by tag, the Game Flow system resolves flow configs by tag -- all through the same registry infrastructure. No per-system lookup code.
| Event | When It Fires |
|---|---|
| Asset Registered | An entry is added to a database |
| Asset Unregistered | An entry is removed from a database |
| Database Created | A new database is created |
| Cache Invalidated | All loaded assets in a database are released |
| Tables Ingested | DataTables have been ingested into a database (v2.0) |
| Validation Complete | Validation pipeline finishes for a database (v2.0) |
Data-driven games live or die by their ability to manage, query, and validate large collections of authored data. The engine provides the Asset Registry for discovery and DataTables for authoring, but neither provides indexed lookup, category queries, conflict detection, or automated validation.
The Data Registry fills that gap. It turns "scan everything and hope" into "index once, query instantly." It turns "duplicate tags are discovered at runtime" into "duplicate tags are caught at edit time." It turns "data validation is manual" into "data validation runs in CI and blocks broken data."
For teams with hundreds or thousands of data entries, this is the difference between a data pipeline that scales and one that does not.
- Development Preview
- Getting Started
- Release branch catalog
- Public Plugin Matrix
- Early Preview Plugins
- Known Issues
- Architecture Overview
- Plugin Topology
- Module Reference
- Configuration and Registry
- Data-Driven Design
- Profiles and Budgets
- Gameplay Tag Architecture
- Initialization Pipeline
- Cross-Plugin Communication
- Message System
- Event Handlers
- Logging and Trace
- Runtime Flows
- Blueprint API Design
- Editor Integration
- Editor Visual System