Planar PnP without OpenCV

Planar PnP without OpenCV

To place a 3D overlay on a printed marker you need its pose: where it is and how it is turned, relative to the camera. That is the Perspective-n-Point problem, and the reflex on the web is to reach for an OpenCV build compiled to WebAssembly.

For the planar case specifically, that reflex is wrong, and this project ended up removing the dependency rather than adding it.

What the problem actually is

You know four things about the marker in its own coordinates — the corners of a square lying flat on z = 0. You have detected where those four corners landed in the image. You know the camera's intrinsics, which in WebXR you can derive from the projection matrix the session hands you.

Find the rotation and translation that map the first onto the second.

The general PnP problem — arbitrary 3D points — is genuinely involved. The planar case is not. All the object points share a plane, and that degeneracy is a gift: the mapping between two planes under perspective is a homography, a 3×3 matrix you can estimate directly by solving a small linear system. From a homography plus known intrinsics you can decompose the rotation and translation analytically.

The IPPE approach (Infinitesimal Plane-based Pose Estimation) does this properly, and it also handles the part everyone forgets: a planar target viewed through a perspective camera admits two mathematically valid poses. They correspond to the plane tilting toward you or away. Both reproject the corners almost equally well, which is why a naive marker overlay flickers between two orientations — the solver is switching between two answers that are both technically correct.

Handling that means generating both candidates and choosing between them. This implementation picks by reprojection error alone — the lower-residual candidate wins, with no memory of the previous frame. That is simple and stateless, and it is also where a flickering overlay comes from when the two residuals are nearly equal: a temporal tiebreak is the obvious next refinement, and it is not in there today.

The pieces this needs

Written out, the whole thing is a handful of small, individually testable functions:

• homographyFromCorrespondences — the linear system from four point pairs.
• solveLinear — a small dense solver.
• ippePoseCandidates — homography plus intrinsics to the two candidate poses.
• nearestRotation3x3 — project a nearly-orthonormal matrix onto the closest true rotation, because floating point does not respect group membership.
• reprojectionErrorPx — the residual, which is both the disambiguator and the quality signal.
• validateQuad and signedQuadArea — sanity checks on the detected quad before any of the above runs.

Each is pure: numbers in, numbers out, no state, no I/O. Which means each is straightforwardly unit-testable and property-testable — you can generate a random pose, project the corners through it, and assert the solver recovers what you started with, to a tolerance. That test is worth more than any amount of visual inspection, and it is only possible because the code is yours.

Why not just ship OpenCV

• Size. An OpenCV WASM build is megabytes. On a page whose entire pitch is "open a link outdoors on mobile data and be in AR in seconds", that is not a rounding error.
• Startup cost. Fetching, compiling and instantiating a large WASM module competes with getting the AR session running.
• Surface area. The whole library ships to use one function.
• Debuggability. When a pose is wrong, stepping through your own hundred lines beats bisecting a foreign build.
• Determinism in tests. Pure JavaScript runs in the same test runner as everything else, with no binary to load.

The counter-argument is real and deserves stating: OpenCV's implementations are battle-tested against decades of edge cases, and rewriting numerical code is how subtle bugs are born. That is a strong argument when you need the general case. For a square target on a plane, the algorithm is small enough to understand completely and test exhaustively, which turns the argument around.

The transferable lesson

Before importing a large library for one function, work out whether your case is the general one or a degenerate one. Planar PnP is not "PnP with a constraint" — it is a different, much smaller problem that happens to share a name.

The same is often true elsewhere: the special case is not a subset of the general algorithm, it is a substitute for it.