- Overview
- Features
- System Requirements
- Installation
- Quick Start
- Detailed Usage
- Pipeline Architecture
- Configuration Options
- Hardware Optimization
- 360 Video Processing
- Backends Comparison
- Export Formats
- Web Viewer
- Performance Tuning
- Troubleshooting
- API Reference
- Examples
- FAQ
The 360 Video to Gaussian Splatting Pipeline is a professional-grade, production-ready tool for converting 360-degree equirectangular video into 3D Gaussian Splatting representations. This pipeline is specifically optimized for Apple Silicon M4 Max with 36GB unified memory but supports all major platforms.
- Automatic hardware detection with platform-specific optimizations
- Multiple backend support: gsplat (with 3DGUT), OpenSplat, 360-gaussian-splatting
- 360-specific processing with equirectangular and fisheye camera models
- Memory-efficient processing with monitoring and adaptive batch sizing
- Resume capability for interrupted processing
- Web-based viewer with 360° navigation
- Multiple export formats including compressed SPZ (90% size reduction)
- Current Version: 2.0.0
- Python Required: 3.8+
- License: MIT
✅ 360 Video Support
- Native equirectangular projection handling
- Fisheye camera model support
- Spherical metadata preservation
- Automatic aspect ratio validation
✅ Advanced Preprocessing
- Hardware-accelerated frame extraction (VideoToolbox on macOS)
- Optical flow stabilization for 360 content
- Camera operator removal with masking
- Color correction and denoising
✅ Multiple SfM Backends
- COLMAP with fisheye camera models
- OpenSfM with spherical projection
- Automatic camera calibration
✅ State-of-the-Art Training
- gsplat with 3DGUT for distorted cameras
- OpenSplat with Metal acceleration
- 360-gaussian-splatting for panoramic scenes
- Adaptive memory management
✅ Professional Export
- PLY format with compression
- Web-optimized .splat format
- SPZ format (90% compression)
- Interactive web viewer
- Metal acceleration for OpenSplat
- VideoToolbox hardware encoding
- Unified memory optimization
- Neural Engine support ready
- CUDA acceleration
- 3DGUT distortion handling
- Multi-GPU support
- Optimized for RTX 3060/2070
- CPU fallback modes
- Docker support ready
- Cloud deployment compatible
- CPU: 8-core processor (Intel/AMD/Apple Silicon)
- RAM: 16GB (32GB recommended)
- GPU: 8GB VRAM (NVIDIA) or Apple Silicon with 16GB unified memory
- Storage: 100GB free space for processing
- OS: Ubuntu 20.04+, Windows 10+, macOS 12+
- CPU: Apple M4 Max or Intel i9/AMD Ryzen 9
- RAM: 32GB+ (unified or dedicated)
- GPU: RTX 3060 12GB+ or Apple Silicon M4
- Storage: 500GB NVMe SSD
- Network: For downloading dependencies
- Python 3.8+
- FFmpeg 4.4+ (with hardware acceleration support)
- COLMAP 3.8+ or OpenSfM
- OpenCV 4.5+
- PyTorch 2.0+ (with CUDA/Metal support)
- CUDA Toolkit 11.8+ (NVIDIA GPUs)
- Xcode Command Line Tools (macOS)
- Visual Studio 2019+ (Windows)
# Clone the repository
git clone https://github.com/yourusername/360-to-gaussian-splatting.git
cd 360-to-gaussian-splatting
# Create conda environment
conda create -n gaussian360 python=3.10 -y
conda activate gaussian360
# Install dependencies
pip install -r requirements.txt
# Run setup script
python setup.py install# Install Homebrew dependencies
brew install ffmpeg colmap cmake opencv python@3.10
# Install Python packages with Metal support
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install opencv-python numpy tqdm pyyaml psutil plyfile
# Build OpenSplat with Metal support
git clone https://github.com/pierotofy/OpenSplat.git
cd OpenSplat
mkdir build && cd build
cmake -DCMAKE_PREFIX_PATH=$(python -c 'import torch;print(torch.utils.cmake_prefix_path)') \
-DGPU_RUNTIME=MPS ..
make -j$(sysctl -n hw.logicalcpu)
sudo make install# Install system dependencies
sudo apt-get update
sudo apt-get install -y \
ffmpeg \
colmap \
libopencv-dev \
python3-pip \
build-essential \
cmake
# Install CUDA (if not already installed)
# Download from: https://developer.nvidia.com/cuda-downloads
# Install Python packages with CUDA support
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install gsplat opencv-python numpy tqdm pyyaml psutil plyfile
# Install gsplat from source for latest features
git clone https://github.com/nerfstudio-project/gsplat.git
cd gsplat
pip install -e .# Install dependencies via conda
conda install -c conda-forge ffmpeg opencv
conda install pytorch torchvision cudatoolkit=11.8 -c pytorch
# Install Python packages
pip install gsplat opencv-python numpy tqdm pyyaml psutil plyfile
# Download and install COLMAP
# https://github.com/colmap/colmap/releases
# Add COLMAP to PATH
set PATH=%PATH%;C:\Program Files\COLMAP\bin# Clone the specialized fork
git clone https://github.com/inuex35/ind-bermuda-opensfm
git clone --recursive https://github.com/inuex35/360-gaussian-splatting
cd 360-gaussian-splatting
# Install submodules
pip install submodules/diff-gaussian-rasterization
pip install submodules/simple-knn
pip install plyfile pyproj# Verify installation
python 360_to_gaussian_splatting_pro.py --help
# Run system check
python -c "from hardware_check import detect_hardware; print(detect_hardware())"# Simplest usage - automatic settings
python 360_to_gaussian_splatting_pro.py input_video.mp4 output_dir/
# With camera operator removal
python 360_to_gaussian_splatting_pro.py input_video.mp4 output_dir/ \
--remove-operator
# Specific time range
python 360_to_gaussian_splatting_pro.py input_video.mp4 output_dir/ \
--start-time 10 --end-time 60python 360_to_gaussian_splatting_pro.py insta360.mp4 output/ \
--backend opensplat \
--iterations 40000 \
--memory-limit 30python 360_to_gaussian_splatting_pro.py insta360.mp4 output/ \
--backend gsplat \
--iterations 30000 \
--gpu-id 0python 360_to_gaussian_splatting_pro.py insta360.mp4 output/ \
--no-gpu \
--iterations 10000 \
--num-workers 8python 360_to_gaussian_splatting_pro.py [OPTIONS] INPUT_VIDEO OUTPUT_DIRINPUT_VIDEO: Path to input 360 video fileOUTPUT_DIR: Directory for output files
--fps FLOAT Frame extraction rate (default: 2.0)
--start-time FLOAT Start time in seconds
--end-time FLOAT End time in seconds
--max-frames INT Maximum frames to extract
--quality INT JPEG quality 1-100 (default: 95)--not-360 Input is not a 360 video
--projection {equirectangular,cubemap}
Projection type (default: equirectangular)--no-stabilize Skip frame stabilization
--denoise Apply denoising filter
--sharpen Apply sharpening filter
--remove-operator Remove camera operator
--operator-mask PATH Mask image for operator
--clean-plate PATH Clean plate image--sfm-backend {colmap,opensfm}
SfM backend (default: colmap)
--camera-model MODEL Camera model (default: OPENCV_FISHEYE)
--max-features INT Max features per image (default: 8192)--backend {auto,gsplat,opensplat,360gs}
Training backend (default: auto)
--iterations INT Training iterations (default: 30000)
--sh-degree INT Spherical harmonics degree (default: 3)
--checkpoint-interval INT Save interval (default: 5000)--no-gpu Disable GPU acceleration
--gpu-id INT GPU device ID (default: 0)
--num-workers INT CPU workers (0=auto)
--memory-limit FLOAT Memory limit in GB--no-resume Start fresh, ignore saved state
--validate-only Only validate input
--dry-run Show what would be done
-v, --verbose Enable verbose output
--debug Enable debug mode--export-formats {ply,splat,spz} [...]
Export formats (default: ply splat)
--no-compress Disable compression
--no-viewer Don't create web viewerfrom pathlib import Path
from pipeline import Pipeline, PipelineConfig
# Create configuration
config = PipelineConfig(
input_video=Path("my_360_video.mp4"),
output_dir=Path("output/"),
fps=2.0,
is_360_video=True,
gs_backend="opensplat",
iterations=30000,
remove_operator=True,
create_web_viewer=True
)
# Run pipeline
pipeline = Pipeline(config)
success = pipeline.run()Input Video → Frame Extraction → Preprocessing → SfM → Gaussian Splatting → Export
↓ ↓ ↓ ↓ ↓ ↓
Validation HW Accelerated Stabilization Camera Training Web Viewer
Memory Managed Operator Removal Models Monitoring Compression
Pipeline
├── VideoProcessor
│ ├── Frame Extraction (FFmpeg/OpenCV)
│ ├── Stabilization (Optical Flow)
│ └── Operator Removal (Masking/Inpainting)
├── EquirectangularProcessor
│ ├── Projection Validation
│ └── Cubemap Conversion
├── StructureFromMotion
│ ├── COLMAP Backend
│ └── OpenSfM Backend
├── GaussianSplattingTrainer
│ ├── gsplat (3DGUT)
│ ├── OpenSplat (Metal)
│ └── 360-gaussian-splatting
├── ExportManager
│ ├── PLY Export
│ ├── SPLAT Export
│ └── SPZ Compression
└── WebViewerCreator
└── Interactive 3D ViewerThe pipeline maintains state for resume capability:
{
"metadata": {
"width": 5760,
"height": 2880,
"fps": 30.0,
"duration": 120.5,
"is_spherical": true
},
"frames_extracted": ["frame_000001.jpg", ...],
"frames_stabilized": ["stabilized/frame_000001.jpg", ...],
"operator_removed": ["processed/frame_000001.jpg", ...],
"sfm_complete": true,
"sfm_path": "output/sfm",
"training_complete": true,
"model_path": "output/gaussian_splatting/model.ply"
}Create a config.yaml for complex projects:
# Project settings
project:
name: "beach_360_scan"
description: "Insta360 X4 beach environment"
# Input settings
input:
video: "insta360_beach.mp4"
is_360: true
projection: "equirectangular"
# Processing settings
processing:
fps: 2.0
start_time: 10.0
end_time: 120.0
max_frames: 200
preprocessing:
stabilize: true
denoise: false
sharpen: true
remove_operator: true
operator_mask: "masks/operator.png"
# SfM settings
sfm:
backend: "colmap"
camera_model: "OPENCV_FISHEYE"
max_features: 8192
matcher: "exhaustive"
# Training settings
training:
backend: "opensplat" # Best for M4 Max
iterations: 40000
sh_degree: 3
checkpoint_interval: 5000
# Memory optimization for 36GB
memory_limit_gb: 30
batch_size: 2
# Export settings
export:
formats: ["ply", "splat", "spz"]
compress: true
create_viewer: true
# Hardware settings
hardware:
use_gpu: true
gpu_id: 0
num_workers: 8# From command line
python 360_to_gaussian_splatting_pro.py --config config.yaml
# From Python
config = PipelineConfig.from_yaml("config.yaml")hardware:
backend: "opensplat" # Best Metal support
memory_limit_gb: 30 # Leave 6GB for system
use_unified_memory: true
processing:
ffmpeg_hwaccel: "videotoolbox"
batch_size: 2
tile_based_rendering: true
training:
iterations: 40000
densify_grad_threshold: 0.00002- Use VideoToolbox for 4x faster frame extraction
- Enable Metal acceleration in OpenSplat
- Monitor thermal throttling with
asitop - Use unified memory advantages - no CPU/GPU copying
- Batch process frames to maximize throughput
hardware:
backend: "gsplat" # Best CUDA support
memory_limit_gb: 10 # Conservative for stability
training:
enable_3dgut: true
iterations: 30000
batch_size: 1
mixed_precision: truehardware:
backend: "opensplat" # More memory efficient
memory_limit_gb: 6
training:
iterations: 20000
densify_grad_threshold: 0.00004 # Less dense
checkpoint_interval: 2000 # Frequent saveshardware:
use_gpu: false
num_workers: 16 # Use all cores
training:
backend: "opensplat" # Has CPU support
iterations: 10000
batch_size: 1The pipeline handles equirectangular (2:1 aspect ratio) video natively:
# Automatic detection
if video_width / video_height == 2.0:
is_360_video = TrueOPENCV_FISHEYE- Recommended for 360SIMPLE_RADIAL_FISHEYE- Simpler modelFOV- Field of view model
spherical- Native 360 supportequirectangular- Direct projection
For Insta360 X4:
camera:
model: "OPENCV_FISHEYE"
k1: -0.0851
k2: 0.0166
k3: -0.0022
k4: 0.0003Special optical flow stabilization that preserves spherical continuity:
# Uses BORDER_WRAP for seamless edges
cv2.warpPerspective(frame, transform, (w, h),
borderMode=cv2.BORDER_WRAP)Pros:
- State-of-the-art quality
- 3DGUT handles distorted cameras
- Fast training (CUDA)
- Active development
Cons:
- Requires NVIDIA GPU
- Higher memory usage
- Complex installation
Best for: NVIDIA GPUs with 12GB+ VRAM
Pros:
- Cross-platform (Metal/CUDA/CPU)
- Memory efficient
- Easy installation
- Good Apple Silicon support
Cons:
- Slightly lower quality
- Less features
- Newer project
Best for: Apple Silicon, memory-constrained systems
Pros:
- Designed for 360 content
- OpenSfM integration
- Panorama-specific optimizations
Cons:
- Requires specific setup
- Less maintained
- Limited documentation
Best for: Dedicated 360 video projects
Standard format for point clouds:
- Uncompressed: ~236 bytes per Gaussian
- Compressed (.ply.gz): ~60% size reduction
- Compatible with: MeshLab, CloudCompare, Blender
Optimized for web viewers:
- Binary format
- ~100 bytes per Gaussian
- Includes color and opacity
- Web-compatible
Highly compressed format:
- 90% size reduction
- Lossy compression
- Optimized for streaming
- Limited tool support
- Interactive 3D navigation with mouse/touch
- Auto-rotation mode
- Screenshot capability
- Performance stats (FPS, point count)
- Mobile responsive
- 360° optimized controls
# Local testing
cd output/exports/web_viewer
python -m http.server 8000
# Open http://localhost:8000
# Production deployment
# Upload to any static hosting (GitHub Pages, Netlify, etc.)Edit the generated index.html:
// Change background color
scene.background = new THREE.Color(0x1a1a1a);
// Adjust point size
material.size = 0.005;
// Change camera FOV
camera.fov = 75;# Monitor memory usage
memory_monitor = MemoryMonitor(config, logger)
if not memory_monitor.check_memory("operation"):
# Reduce batch size or pause
time.sleep(2)# Automatic batch size based on available memory
if available_memory_gb < 8:
batch_size = 1
elif available_memory_gb < 16:
batch_size = 2
else:
batch_size = 4Start with lower quality for testing:
# Quick test run
python 360_to_gaussian_splatting_pro.py video.mp4 test/ \
--iterations 5000 \
--max-frames 50
# Production quality
python 360_to_gaussian_splatting_pro.py video.mp4 final/ \
--iterations 50000 \
--quality 95# Frequent checkpoints for long training
--checkpoint-interval 2000 # Every 2000 iterations
# Resume from checkpoint
--resume # Automatically finds last checkpointSymptoms:
RuntimeError: CUDA out of memory
Metal GPU Frame Capture: Ran out of memory
Solutions:
# Reduce memory usage
--densify-grad-threshold 0.0004 # Higher = fewer points
--max-frames 100 # Limit input frames
--no-gpu # Use CPU fallbackSymptoms:
No valid reconstruction found
Too few feature matches
Solutions:
# Increase features
--max-features 16384
# Use different matcher
--sfm-backend opensfm
# Try different camera model
--camera-model SIMPLE_RADIAL_FISHEYESymptoms:
Video not detected as 360
Wrong aspect ratio
Solutions:
# Force 360 processing
--projection equirectangular
# Check metadata
ffprobe -show_streams video.mp4 | grep sphericalSymptoms:
Loss not decreasing
Artifacts in output
Solutions:
# Adjust learning rates
position_lr_init = 0.00008 # Halve the learning rate
densify_until_iter = 25000 # Extend densificationEnable comprehensive debugging:
python 360_to_gaussian_splatting_pro.py video.mp4 debug/ \
--debug \
--verbose \
--dry-run # Show commands without executing# View main log
tail -f output/logs/pipeline_*.log
# Check errors only
grep ERROR output/logs/errors.log
# Memory usage over time
grep "Memory usage" output/logs/pipeline_*.log@dataclass
class PipelineConfig:
"""Pipeline configuration"""
input_video: Path
output_dir: Path
fps: float = 2.0
is_360_video: bool = True
gs_backend: str = "auto"
# ... many more options
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary"""
@classmethod
def from_yaml(cls, path: Path) -> 'PipelineConfig':
"""Load from YAML file"""class Pipeline:
"""Main pipeline orchestrator"""
def __init__(self, config: PipelineConfig):
"""Initialize pipeline with configuration"""
def run(self) -> bool:
"""Execute complete pipeline"""
def validate_input(self) -> Dict[str, Any]:
"""Validate input video"""class VideoProcessor:
"""Video processing operations"""
def extract_frames(self, metadata: Dict) -> List[Path]:
"""Extract frames from video"""
def stabilize_frames(self, frames: List[Path]) -> List[Path]:
"""Apply stabilization"""
def remove_camera_operator(self, frames: List[Path]) -> List[Path]:
"""Remove operator from frames"""def detect_hardware() -> HardwareInfo:
"""Detect system hardware capabilities"""
def setup_logging(output_dir: Path, verbose: bool) -> Logger:
"""Setup logging system"""
def monitor_memory() -> MemoryStats:
"""Get current memory statistics"""#!/bin/bash
# process_insta360.sh
INPUT="insta360_x4_beach.mp4"
OUTPUT="beach_gaussian/"
python 360_to_gaussian_splatting_pro.py "$INPUT" "$OUTPUT" \
--fps 2 \
--remove-operator \
--backend opensplat \
--iterations 30000 \
--export-formats ply splat spz \
--create-viewerpython 360_to_gaussian_splatting_pro.py timelapse.mp4 output/ \
--start-time 300 \
--end-time 600 \
--fps 0.5 \
--iterations 50000 \
--sh-degree 3# production_pipeline.py
from pathlib import Path
from pipeline import Pipeline, PipelineConfig
config = PipelineConfig(
input_video=Path("professional_360.mp4"),
output_dir=Path("production_output/"),
# High quality settings
fps=4.0,
frame_quality=100,
max_features=16384,
# Preprocessing
stabilize=True,
denoise=True,
sharpen=True,
remove_operator=True,
# Training
gs_backend="gsplat",
iterations=50000,
sh_degree=3,
# Export all formats
export_formats=["ply", "splat", "spz"],
compress_output=True,
create_web_viewer=True
)
pipeline = Pipeline(config)
pipeline.run()# batch_process.py
import glob
from pathlib import Path
from pipeline import Pipeline, PipelineConfig
videos = glob.glob("360_videos/*.mp4")
for video_path in videos:
video = Path(video_path)
output = Path("outputs") / video.stem
config = PipelineConfig(
input_video=video,
output_dir=output,
gs_backend="opensplat",
iterations=20000
)
pipeline = Pipeline(config)
try:
pipeline.run()
print(f"✓ Processed: {video.name}")
except Exception as e:
print(f"✗ Failed: {video.name} - {e}")A: OpenSplat provides the best performance on Apple Silicon with Metal acceleration. It efficiently uses the unified memory architecture and provides good quality results.
A: For a typical 360 video:
- Quick preview: 50-100 frames (fps=0.5-1.0)
- Standard quality: 200-500 frames (fps=2.0)
- High quality: 500-1000 frames (fps=4.0)
A: Typical processing times:
- Frame extraction: 5-10 minutes
- SfM (COLMAP): 30-60 minutes
- Training: 2-6 hours
- Export: 5-10 minutes
Total: 3-8 hours depending on settings and hardware.
A: Yes! Use the --not-360 flag. The pipeline will use standard pinhole camera models instead of fisheye.
A: Typical sizes:
- PLY: 200-500 MB (2-5M Gaussians)
- PLY.gz: 80-200 MB (compressed)
- SPLAT: 100-250 MB
- SPZ: 20-50 MB (90% compression)
A: Several strategies:
- Reduce frame count (
--max-frames 100) - Increase densification threshold (
--densify-grad-threshold 0.0004) - Lower iterations (
--iterations 15000) - Use CPU mode (
--no-gpu) - Enable checkpointing for recovery
A: Currently, the pipeline uses single GPU. For multi-GPU, you can:
- Process different videos on different GPUs
- Split frame ranges and process in parallel
- Use distributed training (experimental)
A: Create a black and white image where:
- White pixels: Areas to remove (operator)
- Black pixels: Areas to keep
- Save as PNG with same resolution as video
- Use soft edges (blur) for better results
A:
- COLMAP: More robust, better for standard scenes, GPU accelerated
- OpenSfM: Better 360 support, handles spherical projection natively
A: The web viewer is static HTML/JS:
- Upload the
web_viewerfolder to any web host - No server-side processing needed
- Works on GitHub Pages, Netlify, Vercel, etc.
- Mobile responsive out of the box
- Check the FAQ section
- Review Troubleshooting
- Search existing issues on GitHub
- Join our Discord community
- Open a new issue with:
- System specifications
- Full command used
- Error messages
- Log files
We welcome contributions! Please:
- Fork the repository
- Create a feature branch
- Add tests for new features
- Submit a pull request
MIT License - see LICENSE file for details
- NVIDIA for 3DGUT and gsplat
- OpenSplat contributors
- COLMAP and OpenSfM teams
- Insta360 for camera specifications
- The Gaussian Splatting research community
Version: 2.0.0 | Last Updated: June 2025 | Optimized for Apple M4 Max (36GB)