Feed-forward multi-view inverse rendering: joint neural-SDF geometry, Cook-Torrance GGX materials, and a point-light + ambient illumination model recovered from sparse RGB views in a single forward pass.
Unlike radiance-field-only pipelines, Prism keeps geometry and appearance physically separable by coupling an implicit surface to differentiable NeuS-style rendering, then training the surface, the BRDF, and the light against the input photographs jointly.
- Image encoding (
prism/encoder.py) — a ResNet-34 extracts shallow per-view feature maps and a global latent code summarizing the scene. - SDF (
prism/sdf_mlp.py) — for each 3D query point, projected per-view features and the global latent condition an MLP that outputs the signed distance. The lower MLP supplies geometry; higher levels supply local appearance context. - Material + lighting heads — small MLPs decode Cook-Torrance GGX parameters (albedo, roughness, metalness) at the predicted surface, and the scene's point-light position, intensity, and ambient term.
- Differentiable rendering (
prism/renderer.py) — NeuS importance-sampled ray marching produces expected hit points;brdf.pyshades them under the predicted light using the GGX BRDF. - Multi-objective loss (
prism/losses.py) — photometric (L1/SSIM), depth, normal, silhouette / free-space, SDF sign / band, eikonal, and visual-hull terms. Loss weights and ablation switches live inconfig.py.
On 19 held-out OmniObject3D objects across 6 categories:
- Chamfer distance: 0.209 ± 0.043 (lower is better)
- Foreground PSNR: 14.55 ± 2.52 dB (higher is better)
Loss ablations show the largest PSNR impact from removing photometric supervision (−4.48 dB), then depth (−1.14 dB), with smaller contributions from normal (−0.23 dB) and eikonal (−0.08 dB).
- Python — PyTorch, NumPy
- Rendering — custom NeuS-style importance sampler + GGX shading
- Data — OmniObject3D (Blender renders + raw scans)
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# train from scratch
python train.py --data_root /path/to/omniobject3d/extracted
# resume from default checkpoint (model.pt)
python train.py --resume
# overfit one object for debugging
python train.py --overfit --overfit_object bottle_001
# quantitative evaluation
python evaluate.py --checkpoint model.pt --out_dir eval_results/metricsAggregate and per-object metrics are written to eval_results/metrics/metrics.json.
Final_Project_Report.pdf— full write-upPRISM_architecture.md— architecture sketch
- Spatially uniform BRDF per object (no SVBRDF maps yet).
- Point-light approximation cannot capture HDRI environment lighting.
- Thin structures are limited by ray-sampling budget; deeper hierarchies or longer training help.
MIT.