XVERSE 3D-GS UE Plugin

English | 中文

Introduction

XVERSE 3D Gaussian Splatting UE plugin（XVERSE 3D-GS UE Plugin）is an Unreal Engine 5 (UE5) based plugin developed by XVERSE Technology Inc. (Zhenshen, China), aiming to provide real-time visualization, management, editing, and scalable hybrid rendering of Gaussian Splatting model, which is a recently arised technique for reconstructing 3D scenes from multiple photos, more details can refer to here.

Currently, our plugin is implemented based on UE5 Niagara, fully exploiting the advantage of the features provided by UE5 to realize efficient rendering and managing for Guassian Splatting models.

Since we are at early access, current accessible features are summarized below:

• Niagara-Based High-quality real-time visualizing and rendering for 3D Gaussian Splatting
• Easily importing and converting from the original Gaussian Splatting scene (.ply file) to ours
• Hybrid rendering with other UE assets
• RTS and volume based crop
• Crop a pure niagara for VFX
• Support dynamic illumination of Gaussian Splatting scene
• Automatic lod generation, supporting over 200,000 point clouds in niagara
• Training tools in Windows system (XV3DTools v1.0)： train a given mp4 video to GaussianSplatting ply file

We will actively release new features in this repo, please stay tuned. Some future updates will contain:

• Dynamic Lod Rendering
• Automatic collision generation
• Support preview window
• Interactively Edit 3D-GS assets
• 3D-GS assets compression

Getting Started

System Requirements

• Windows 10 or 11
• Unreal Engine 5.1.x / 5.2.x / 5.3.x
• cuda11 or higher (if using the XV3DTools)

Plugin download

1. Open github Release page

2. Download the zip corresponding to the UE version you are using

3. Unzip and move folder XV3dGS to folder Plugins

Demo Data Download Links:

We have provided two sets of test data available for download at:

• Baidu Pan Download Link
• Google Drive Download Link

Included in these downloads are demo_fox_1080p.mp4 and demo_office_1080p.mp4, serving as two sample MP4 files for testing XV3DTools' local training capabilities.

demo_fox_gs.ply and demo_office_gs.ply are 3DGS models trained by XV3DTools, which can be imported into Unreal Engine (UE) using this plugin for real-time rendering.

Open demo project

This repo contains a demo project with an example scene and level

0. download using git clone

git clone https://github.com/xverse-engine/XV3DGS-UEPlugin.git

1. Open XV3DGS.uproject to start UE
2. Open demo level

Import your guassian splatting model

Click on the 'Load Assets' button on the navigation bar or navigate to Window/XV3dgs.

This action will bring up an integrated tool page by default set to load the PLY page. Click on the 'Load PLY' button.

This will trigger a resource selection window to appear.

Choose your Gaussian model file (in .ply format), then it will automaticly generate LOD and loaded into your Content Browser as a blueprint.

Drag the blueprint into Level Viewport and you can see your Gaussian scene.

Local Training on Windows Platform

We develop the XV3DTools(V1.1.1) to train MP4 videos into Gaussian Splatting ply.

Please Note: Installation of CUDA 11 or higher is required. CUDA Download Link

XV3DTools Usage Steps:

0. Download XV3DTools(V1.1.1) tool from the cloud drive to your local machine: Baidu Pan Download Link Google Drive Download Link

1. Click on the icon to open the interface and switch to the 'training' page.

2. Click button to set the parameter "XV3DTools Folder" to the local path where XV3DTools tool is located.

3. Click button to select the MP4 file for processing and set it as the parameter "Input MP4 Path" (requires video frame rate > 20).

4. Click button to choose the output folder and set it as the parameter "Output Folder".

5. click Start Training button.

6. Before training, you can open "Advanced Options" and click "Refresh Estimated Time" button to estimate the training time. The parameter "Training iteration per image" can be adjusted to control the number of iterations per image.

7. Training takes time. Once completed, locate the trained ply file in the output folder.

Training Speed Reference:

• Test Machine Configuration: Windows 10, CUDA 11.8, GeForce RTX 2080 (8GB VRAM), i9-9900 processor.
• demo_fox.mp4: Video duration of 10 seconds, training time of 10 minutes.
• demo_office.mp4: Video duration of 430 seconds, training time of 360 minutes.

Packaging UE level for Windows Using Our Plugins

You can refer to the following tutorial to package your own UE levels into exe files that can be run on windows System.

packing tutorial

We provide a packaged Windows executable demo, you can download pack_win_exe.zip from release, simply double-click the exe file to start it.

Feature Introduction

RTS (Rotation, Translation, Scaling) Transform

Select the 3D-GS asset and drag it into the scene. then you can Transform it.

You can also select a 3D-GS actor in the level viewport and transform it using GIZMO.

Model Clipping by Gaussian model

1. Click to select your Gaussian model blueprint, then click to select the "Clip Volume Component", and open the "Visible" box in the "Clip Volume Component".

2. With the "Clip Volume Component" selected, you can see a crop volume in the window. You can modify the rotation, translation, and scale of the volume by adjusting the Transform attribute. Alternatively, use the keyboard shortcuts WER to visually adjust the GIZMO for rotation, translation, and scale.

3. Keep the "Clip Volume Component" selected, and choose different cropping options in the Crop Type. The default state is no cropping, but you can change it to crop the inside or outside of the volume. The example below shows cropping the outside of the volume.

4. Click the "Crop GSPoints" button to clip the specified portion and obtain a clean Niagara.

5. This clean Niagara can be used for VFX. For more details, please refer to our VFX tutorial (Coming soon).

Dynamic illumination

Dynamic lighting is not default opened.

you can enable it by turning on the withRelighting button.

Supports all the UE light types

The light source can be moved at will

Change color

Modify the AlbedoTint parameter to change the overall color.

LOD

For imported point clouds, LOD is automatically calculated, with the default being to explicitly display the maximum LOD within the limit of 2 million points for Niagara. The parameter "VisWhichLOD" can be used to force the use of a specific LOD.

Dynamic switching of LOD based on the view camera will be supported in future updates.

Release Note

v1.1.1

• New Model Clipping allowing the creation of a clean Niagara for VFX.
• XV3DTools v1.1.1: Users can now adjust the training iteration.

v1.1.0

• Training tools in Windows system (XV3DTools v1.0) to train a given mp4 video to GaussianSplatting ply file
• Automatic lod generation, supporting over 200,000 point clouds in niagara
• fix bug in transform

v1.0.1

• Fix bug in v1.0.0：Error while packaging for Windows in UE5.2 and UE5.3

v1.0.0

• Niagara-Based High-quality real-time visualizing and rendering for 3D Gaussian Splatting
• Easily importing and converting from the original Gaussian Splatting scene (.ply file) to ours
• Hybrid rendering with other UE assets
• RTS and Geometry Editing
• Apply VFX effect to Gaussian Splatting scene
• Support dynamic illumination of Gaussian Splatting scene
• Automatic enhancement of Gaussian Splatting scene

Contributors