Depth on the web

Depth on the web

The WebXR depth-sensing API hands your page, per frame, an estimate of how far away the world is at each point. This is more capability than the web usually gets, and it comes with a shape that is easy to misjudge in both directions.

What you get

A depth buffer aligned with the camera view, and the information needed to turn a sample into a 3D point: intrinsics and the camera pose. From there:

• Unproject one sample to a point in world space.
• Sample on a grid rather than per pixel, since you rarely need full resolution and full resolution is expensive.
• Accumulate across frames into a persistent model — the occupancy grid — because a single frame only describes what is visible from one place.

The framework wraps these as DepthSampler, unprojectDepthPoint, createDepthUnprojector, createDepthGridLookup and OccupancyGrid, which mostly saves you from re-deriving the same projection maths and from re-learning that per-pixel work is a bad idea on a phone.

What it is genuinely good for

Measuring physical objects. A printed marker's real edge length can be measured from the depth at its corners instead of configured by a human — which removes an entire family of silent, hard-to-diagnose errors.

Occlusion. Knowing that a real object is in front of a virtual one is what makes the composite look like a scene rather than a sticker.

Collision geometry. Accumulated depth becomes a voxel model and then a mesh, which is what lets virtual objects interact with real surfaces.

Plausibility checks. Depth is an independent sensing path from the colour image, so it can contradict a conclusion drawn from the image — a solved marker pose floating a metre off the wall, for instance. Independent contradiction is the most valuable thing a second sensor provides, and it is the use most often overlooked.

Where it disappoints

Resolution is low. Far lower than the colour camera, and often interpolated. Fine structure — railings, wires, thin table legs — is not reliably there.

Edges are the worst case. At a depth discontinuity a sample may belong to either side. This is exactly where you most want accuracy, and where you get least. Sampling slightly inside a boundary rather than on it is a habit worth forming.

Range is limited, and error grows with distance. Depth is useful in room-scale and near-field terms. Across a plaza it is not.

Sunlight hurts. Depending on the sensing technology, bright outdoor light degrades depth substantially. For a project mostly concerned with outdoor AR, this is the significant limitation, and it is why depth-dependent features here are treated as enhancements rather than foundations.

Reflective and transparent surfaces are nonsense. Glass, water, polished floors. Not noisy — actively wrong, reporting whatever is reflected.

Support varies. Not every device or session exposes depth, so every feature built on it needs a defined behaviour when it is absent.

The design consequence

Depth belongs in the "makes it better" tier, not the "makes it work" tier.

Concretely, in this project: marker size can be measured from depth, and falls back to a configured value when depth is unavailable. Marker poses can be checked against reconstructed geometry, and are reported as unverified rather than rejected when there is nothing to check against. The physics demo, which genuinely requires depth, is a demo — and it is honest about working better indoors.

That tiering is the whole discipline. A feature that hard-requires depth excludes devices without it, breaks in sunlight and fails on glass. A feature that opportunistically uses depth is better where depth is good and merely ordinary where it is not.

The larger point about web capabilities

There is a temptation, when a platform finally exposes something powerful, to build on it as though it were load-bearing. Depth on the web is powerful and it is not uniform — across devices, across lighting, across surfaces.

The durable pattern is to treat a new capability as an enhancement with a defined fallback, and to know precisely which of your features would vanish if it were withdrawn tomorrow.