StreamSplat: Hybrid Neural-Mesh Rendering on the Web

Real-time 3D Gaussian Splatting and Mesh Fusion using a Client-Server Web Architecture

This project enables real-time photorealistic rendering in the browser by combining server-side neural rendering (3DGS/4DGS) with lightweight client-side mesh rendering and depth-based screen-space fusion. Built for platform-independent delivery using WebGL/WebGPU.

▶ View Live Demo
Paper (Web3D 2025)

---

Overview

Recent advances in neural rendering (e.g., 3D Gaussian Splatting) offer photorealistic scene reconstruction, but their computational cost makes direct deployment on web browsers infeasible.
To address this, StreamSplat proposes a hybrid rendering pipeline:

• Server: Performs neural rendering (3DGS/4DGS) and image encoding (NVENC)
• Client (Web): Receives compressed video & depth, performs screen-space depth-aware fusion with locally rendered 3D mesh assets

This decouples neural asset logic from the client, enabling scalable and interactive delivery on diverse web devices (desktop, tablet, mobile).

---

System Architecture

  +----------------+      H.264 Video + Depth     +----------------------+
  |  Server-side   |  ------------------------->  |     Web Client       |
  | Gaussian Render|                              |  WebGL/WebGPU Viewer |
  | (3DGS / 4DGS)  |  <-------------------------  |   + Mesh Fusion      |
  +----------------+       User Input / Pose      +----------------------+

• Neural renderer: 3D Gaussian Splatting w/ CUDA-accelerated encoder
• Transport: WebSocket
• Client viewer: Three.js + WebGL / WebGPU (DepthTexture fusion, user interaction, async decoding)

---

Features

• Hybrid Client-Server architecture for web deployment
• 3DGS/4DGS neural rendering offloaded to GPU-enabled server
• Client-side depth-aware fusion with local mesh rendering
• WebGL/WebGPU-based viewer (Three.js)
• Real-time streaming with H264 (NVENC)
• Hardware-accelerated decoding (WebCodecs)

---

Getting Started

1. Server Setup (Ubuntu + CUDA)

# Clone and install dependencies
git clone https://github.com/elan1537/HybridPipeline.git
cd StreamSplat/backend
pip install -r requirements.txt

# Run neural renderer
python src/server.py --scene_path=...

2. Client Viewer

Serve the frontend with any HTTP server:

cd frontend
npm install
npm run dev

---

Citation

If you use this project or codebase, please cite the following:

@inproceedings{park2025streamsplat,
  author    = {Sehyeon Park and Yechan Yang and Myeongseong Kim and Byounghyun Yoo},
  title     = {StreamSplat: A Hybrid Client-Server Architecture for Neural Graphics using Depth-based Fusion on the Web},
  booktitle = {Proc. Web3D},
  year      = {2025},
  pages     = {1--10},
  doi       = {10.1145/3746237.3746316}
}