Physics against the real world

Physics against the real world

There is a moment in AR that lands more reliably than any amount of explanation: a virtual ball bounces off a real table.

Not "bounces off an invisible plane the app assumed was there" — off the actual shape of the actual room, reconstructed while you look at it.

From depth samples to something you can collide with

WebXR's depth feature gives you a per-frame depth buffer: for each pixel, roughly how far away that bit of the world is. On its own it is a picture of distances from one viewpoint, which is not something a physics engine can use.

Turning it into geometry is three steps:

1. Unproject. Each depth sample plus the camera intrinsics and pose gives a 3D point in world space. unprojectDepthPoint and the depth-grid helpers do this.
2. Accumulate into voxels. Points from many frames and many viewpoints go into an OccupancyGrid — a sparse voxel set marking which cells contain surface. Accumulation is what turns single-viewpoint snapshots into a model: walk around, and the parts you saw from different angles fill in.
3. Mesh it. meshOccupiedCells converts the sparse voxel set into a face-culled surface plus a list of axis-aligned boxes. Either can back a collider; the demo runs the triangle-mesh path, so occlusion and physics share exactly one piece of geometry and can never disagree about where the wall is.

The voxel grid is doing quiet work here: it fuses noisy per-frame depth into a stable model, and a cell repeatedly observed as occupied stays occupied even when one frame's depth is nonsense.

Why voxels rather than a mesh straight from depth

Meshing each frame's depth directly is possible and produces a shimmering surface that changes every frame, because depth noise changes every frame. A ball colliding with it jitters.

Voxels quantise. A cell is occupied or it is not, and small per-frame variations do not cross the threshold. The result is a stable world to collide against, at the cost of blockiness — a trade that is obviously right for physics and obviously wrong for photorealistic rendering, which is why the same grid is not used for both.

The desktop half

The physics demo runs in two modes with one code path:

• On device, in a live WebXR session, reconstructing the room around you as you look, with balls launched along your aim.
• On a desktop, replaying a recorded walk — the same occupancy mesh reconstructing from the replayed depth stream, in a deterministic world, with no phone involved.

The second mode is what makes the first developable. Collision tuning is a loop of small changes and immediate judgement, and doing that loop while standing in a room holding a phone is miserable. Doing it at a desk against a recording of that room, where the geometry is identical every run, is ordinary work.

It also means the behaviour is testable. A deterministic world plus a deterministic launch produces a deterministic trajectory, which is an assertion rather than an impression.

What is hard about it

• Unobserved space is not empty space. The grid only knows about surfaces the camera has seen. Everything else is unknown, and treating unknown as empty means balls fly through the wall you have not looked at yet.
• Latency. Reconstruction lags observation. A surface that just came into view is not collidable for a moment.
• Depth quality outdoors. Depending on the sensing technology, outdoor light can degrade depth substantially — and range matters more still, since depth is a near-field measurement and an outdoor scene mostly is not. This is a demo that works best indoors, and saying so is more useful than pretending otherwise.
• Voxel size. Too coarse and thin objects vanish; too fine and memory and meshing costs climb quickly.

Why it is in the repository

Partly because it is a good demo. Mostly because the occupancy grid is infrastructure that pays for itself elsewhere: it is also what lets a solved marker pose be checked against the physical world, rejecting poses that claim a code is floating in mid-air. The physics demo is the visible, enjoyable consequence of a component that earns its place for less photogenic reasons.