v1.0.0

We are happy to celebrate the v1.0.0 release of SPEAR. We are choosing to tag this release as a major version update to indicate our confidence that SPEAR is ready to be deployed in a wide range of projects.

Highlights

Figure: The SPEAR Python library can call any C++ function, and can access any C++ variable, on any game entity, and any game subsystem, in any Unreal Engine application, provided the function or variable has been exposed to Unreal's visual scripting system, i.e., Blueprints. We demonstrate this unique capability by controlling 5 embodied agents with distinct action spaces across several Epic Games sample projects: a person and a car from CitySample (top); a flying robot from StackOBot (bottom left); a person with parkour skills from GameAnimationSample (bottom middle); and multiple agents in a resource collecting game called CropoutSample (bottom right). SPEAR can control these agents and perform customizable high-speed rendering without requiring any modifications to the SPEAR library code or the projects themselves, aside from adding a single-line declaration to each project definition file. No further modifications are required to enable comprehensive programmatic control of complex in-the-wild Unreal applications from Python via SPEAR.

Figure: SPEAR includes a customizable camera entity that can render a superset of the ground truth modalities available in the Hypersim dataset, and can render 1080p photorealistic beauty images (top row) directly into a user's NumPy array at 55 frames per second. The SPEAR camera can also render fine-grained 24-bit entity IDs that can be used for both material segmentation and object segmentation tasks (middle row), and a non-Lambertian intrinsic image decomposition consisting of diffuse reflectance, diffuse illumination, and a non-diffuse residual term (bottom row).

Figure: By calling existing Unreal Engine C++ functions that are available through SPEAR's general-purpose Python API, it is straightforward to control the lighting in any scene. Here, we programmatically control the lighting to simulate time-of-day changes in the ElectricDreams sample project that is freely available from Epic Games.

Figure: It is also straightforward to use SPEAR to programmatically interact with Unreal's Procedural Content Generation (PCG) system for dynamically generating scene geometry. Here, we horizontally translate a PCG entity (the big rock structure in the approximate center of each image) across the scene from left to right in the ElectricDreams sample project via a simple Python program. Notice how the main rock structure harmonizes with the rest of the scene in a convincing way (e.g., the water and ground adjusts around the main rock, logs appear and connect with nearby structures), even when it is being driven by simple programmatic control.

Figure: SPEAR includes a customizable multi-view camera entity that can render a scene from multiple views at exactly the same time in the Unreal simulation. Here, we render synchronized images from the Metahumans sample project that is freely available from Epic Games.

Feature summary

• SPEAR can call any C++ function, and can access any C++ variable, on any game entity, and any game subsystem, provided the function or variable has been exposed to Unreal's visual scripting system, referred to in the UE ecosystem as the Blueprint System or simply as Blueprints. There are over 13K functions and over 44K variables that are already exposed to Blueprints (and are therefore accessible in SPEAR) in the UE codebase, and it is trivial to expose new functions and variables by adding a single-line annotation next to a function or variable in a C++ header (see example below).
• SPEAR provides efficient zero-copy NumPy interoperability, e.g., SPEAR can copy 1080p photorealistic beauty images (see figure above) from the GPU directly into a user's NumPy array at 55 frames per second without requiring any intermediate data copying.
• SPEAR includes a camera entity that can render a superset of the ground truth modalities available in the Hypersim dataset (see figure above), including fine-grained 24-bit entity IDs that can be used for both material segmentation and object segmentation tasks, and a non-Lambertian intrinsic image decomposition consisting of diffuse reflectance, diffuse illumination, and a non-diffuse residual term.
• SPEAR can programmatically control standalone shipping games that are already running, live simulations running inside the Unreal Editor, Unreal's path tracer, and the Unreal Editor itself, all through a unified Pythonic interface.
• SPEAR gives users precise control over how their UE work is executed across UE frames, while also allowing users to execute complex work graphs (i.e., with arbitrary data dependencies among work items) deterministically within a single frame.
• SPEAR can be used in any Python environment, even on a remote machine, and does not need to be invoked from inside the Unreal Editor.