Final project for EN.520.681 Perception for Embodied AI. This repository studies a rigid-body pendulum represented by 3D Gaussians whose visual parameters support differentiable rendering and whose density parameters define physical properties for dynamics prediction.
The core idea is to augment each Gaussian with a learnable physical density. From the Gaussian means, covariances, and densities, the model analytically computes mass, center of mass, and inertia, then uses those quantities in an Euler-Lagrange model for a 3-DOF pendulum.
Thank Codex for the great help!!!
Each Gaussian stores visual and physical parameters in the body frame:
- color
- opacity
- mean
- covariance
- density
For each Gaussian, density determines mass through the Gaussian volume integral. The full rigid body mass, center of mass, and inertia are computed by summing per-Gaussian contributions and applying the parallel-axis theorem. The learned physical properties are then used by the pendulum dynamics model.
The training objective combines:
- visual reconstruction loss from Gaussian splatting
- dynamics residual loss from the Euler-Lagrange equation
See report.pdf for the full method, derivations, experiments, and discussion. Note that the densify/prune functionality is not used. We initialize gaussians with prior knowledge of the shape of pendulum.
.
+-- data/ # Recorded MuJoCo dataset metadata and images
+-- eg_pendulum/ # Example rollout outputs and inertia logs
+-- literature/ # Literature review notes
+-- xmls/ # MuJoCo XML models
+-- data_recorder.py # MuJoCo dataset recording utilities
+-- dataset.py # PyTorch datasets for vision and dynamics records
+-- gsdynamics.py # Forward simulation using learned Gaussian dynamics
+-- pendulum.py # Pendulum-related utilities
+-- render.py # Differentiable 3D Gaussian rendering
+-- splatting.py # Gaussian model, physics, and Euler-Lagrange logic
+-- trainer.py # Joint vision and dynamics training loop
+-- tests.py # Rendering and dataset smoke tests
+-- train.ipynb # Notebook workflow for training and experiments
+-- utils.py # Rotation, quaternion, and math utilities
+-- report.pdf # CVPR-style final report
+-- slide.pdf # Final presentation slides
The learned model recovers the pendulum's effective inertia around the pivot and produces torque-free rollouts that closely follow the MuJoCo reference trajectory.
The animations below compare the learned Gaussian Splatting dynamics rollout with the MuJoCo reference rollout.
Left: MuJoCo reference rollout. Right: learned Gaussian Splatting dynamics rollout.
The report also highlights an important limitation: matching observed dynamics does not uniquely identify the true mass distribution.