Projection on affine space

[alphaville]

The method

Let $A\in\mathbb{R}^{m\times n}$ with ${\rm rank} A = m$ and $m \leq n$ (typically $m < n$). Let $b\in\mathbb{R}^m$. The projection of a vector $x\in\mathbb{R}^n$ on the set $X = \{x: Ax = b\}$ is given by
$$\Pi_X(x) = \mathrm{argmin}_{z\in X} \frac{1}{2}\|x - z\|^2.$$

This is a convex QP with Lagrangian
$$L(z, \lambda) = \frac{1}{2}\|x-z\|^2 + \lambda^\intercal (Az-b),$$
and its gradients with respect to $z$ and $\lambda$ are
$$\nabla_z L(z, \lambda) = x - z + A^\intercal \lambda$$
and
$$\nabla_\lambda L(z, \lambda) = Az - b.$$
We are looking for a pair $(z^\star, \lambda^\star)$ such that $\nabla_z L(z^\star, \lambda^\star) = 0$ and $\nabla_\lambda L(z^\star, \lambda^\star) = 0$, that is,

$$\begin{aligned}x - z^\star + A^\intercal \lambda^\star = 0, \\ Az^\star - b = 0.\end{aligned}$$

Multiplying the first equation by $A$ from the left and substituting $Az^\star = b$ we have

$$Ax - b + AA^\intercal \lambda^\star = 0 \Rightarrow AA^\intercal \lambda^\star = Ax - b,$$

We can solve this linear equaltion by computing the Cholesky factorisation of $AA^\intercal$, that is, $PAA^\intercal P^\intercal = LL^\intercal,$ where $L$ is a lower triangular matrix and $P$ is a permutation matrix. Having solved this for $\lambda^\star$, we can compute

$$z^\star = x - A^\intercal \lambda^\star.$$

We can also write

$$z^\star = x - A^\intercal (AA^\intercal)^{-1}(Ax-b).$$

Main Changes

• Implementation of projection on affine spaces of the form $\{x: Ax = b\}$
• Unit tests in Rust

Associated Issues

• None

TODOs

• API Docs
• All tests must pass
• Update CHANGELOG(s)
• Update webpage documentation
• Bump versions (in CHANGELOG, Cargo.toml and VERSION)

[ruairimoran]

we need to agree on the equations

[ruairimoran]

Might want to change that gradient in the PR method. Otherwise, lgtm

[alphaville]

I'm merging this now. This has been implemented in Rust, but we need to make this available via opengen too.