From a650fe92d7b4746722047b2eb6e721d2ba7efd02 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Beno=C3=AEt=20Legat?= Date: Thu, 13 Jan 2022 16:38:27 -0500 Subject: [PATCH 1/3] Improve quadratic duality in doc --- docs/src/background/duality.md | 39 ++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/docs/src/background/duality.md b/docs/src/background/duality.md index 4d3bce587e..4b23d0b423 100644 --- a/docs/src/background/duality.md +++ b/docs/src/background/duality.md @@ -160,24 +160,41 @@ If ``\mathcal{C}_i`` is a vector set, the discussion remains valid with between ``y_i`` and the vector of scalar-valued quadratic functions. !!! note - For quadratic programs with only affine constraints, the optimality condition - ``\nabla_x L(x, y^\star) = 0`` can be simplified as follows: + For quadratic programs with only affine conic constraints, ```math - 0 = \nabla_x L(x, y^\star) = Q_0x + a_0 - \sum_{i = 1}^m y_i^\star a_i + \begin{align*} + & \min_{x \in \mathbb{R}^n} & \frac{1}{2}x^TQ_0x + a_0^T x + b_0 + \\ + & \;\;\text{s.t.} & A_i x + b_i & \in \mathcal{C}_i & i = 1 \ldots m + \end{align*} ``` - which gives + with cones ``\mathcal{C}_i \subseteq \mathbb{R}^{m_i}`` for ``i = 1 \ldots m``, consider the Lagrangian function ```math - Q_0x = \sum_{i = 1}^m y_i^\star a_i - a_0 . + L(x, y) = \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \sum_{i = 1}^m y_i^T (A_i x + b_i) ``` - The Lagrangian function + Let ``z(y)`` denote ``\sum_{i = 1}^m A_i^T y_i - a_0``, the Lagrangian can be rewritten as ```math - L(x, y) = \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \sum_{i = 1}^m y_i (a_i^T x + b_i) + L(x, y) = \frac{1}{2}{x}^TQ_0x - z(y)^T x + b_0 - \sum_{i = 1}^m y_i b_i ``` - can be rewritten as + + The condition ``\nabla_x L(x, y) = 0`` can be simplified as follows: + ```math + 0 = \nabla_x L(x, y) = Q_0x + a_0 - \sum_{i = 1}^m y_i a_i + ``` + which gives ``Q_0x = z(y)``. + This allows to obtain that + ```math + \min_{x \in \R^n} L(x, y) = -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i b_i + ``` + so the dual problem is + ```math + \max_{y \in \mathcal{C}_i^*} \min_{x \in \R^n} -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i b_i. + ``` + If ``Q_0`` is invertible, we can eliminate ``x`` and obtain that ```math - L(x, y) = \frac{1}{2}x^TQ_0x - (\sum_{i = 1}^m y_i a_i^T - a_0^T) x + b_0 - \sum_{i = 1}^m y_i (a_i^T x + b_i) + \min_{x \in \R^n} L(x, y) = -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i b_i ``` - which, using the optimality condition ``\nabla_x L(x, y^\star) = 0``, can be simplified as + so the dual problem is ```math - L(x, y) = -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i (a_i^T x + b_i) + \max_{y \in \mathcal{C}_i^*} -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i b_i ``` From d5d9ce67e2977cd1d2843ad8e483806ff7dc42de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Beno=C3=AEt=20Legat?= Date: Thu, 13 Jan 2022 17:16:11 -0500 Subject: [PATCH 2/3] Fixes --- docs/src/background/duality.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/src/background/duality.md b/docs/src/background/duality.md index 4b23d0b423..d60e933962 100644 --- a/docs/src/background/duality.md +++ b/docs/src/background/duality.md @@ -129,7 +129,7 @@ Given a problem with quadratic functions: \begin{align*} & \min_{x \in \mathbb{R}^n} & \frac{1}{2}x^TQ_0x + a_0^T x + b_0 \\ -& \;\;\text{s.t.} & \frac{1}{2}x^TQ_ix + a_i^T x + b_i & \in \mathcal{C}_i & i = 1 \ldots m +& \;\;\text{s.t.} & \frac{1}{2}x^TQ_ix + a_i^T x + b_i & \in \mathcal{C}_i & i = 1 \ldots m. \end{align*} ``` with cones ``\mathcal{C}_i \subseteq \mathbb{R}`` for ``i = 1 \ldots m``, consider the Lagrangian function @@ -165,36 +165,36 @@ between ``y_i`` and the vector of scalar-valued quadratic functions. \begin{align*} & \min_{x \in \mathbb{R}^n} & \frac{1}{2}x^TQ_0x + a_0^T x + b_0 \\ - & \;\;\text{s.t.} & A_i x + b_i & \in \mathcal{C}_i & i = 1 \ldots m + & \;\;\text{s.t.} & A_i x + b_i & \in \mathcal{C}_i & i = 1 \ldots m. \end{align*} ``` - with cones ``\mathcal{C}_i \subseteq \mathbb{R}^{m_i}`` for ``i = 1 \ldots m``, consider the Lagrangian function + with cones ``\mathcal{C}_i \subseteq \mathbb{R}^{m_i}`` for ``i = 1, \ldots, m``, consider the Lagrangian function ```math - L(x, y) = \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \sum_{i = 1}^m y_i^T (A_i x + b_i) + L(x, y) = \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \sum_{i = 1}^m y_i^T (A_i x + b_i). ``` Let ``z(y)`` denote ``\sum_{i = 1}^m A_i^T y_i - a_0``, the Lagrangian can be rewritten as ```math - L(x, y) = \frac{1}{2}{x}^TQ_0x - z(y)^T x + b_0 - \sum_{i = 1}^m y_i b_i + L(x, y) = \frac{1}{2}{x}^TQ_0x - z(y)^T x + b_0 - \sum_{i = 1}^m y_i^T b_i. ``` - The condition ``\nabla_x L(x, y) = 0`` can be simplified as follows: + The condition ``\nabla_x L(x, y) = 0`` gives ```math - 0 = \nabla_x L(x, y) = Q_0x + a_0 - \sum_{i = 1}^m y_i a_i + 0 = \nabla_x L(x, y) = Q_0x + a_0 - \sum_{i = 1}^m y_i^T b_i ``` which gives ``Q_0x = z(y)``. This allows to obtain that ```math - \min_{x \in \R^n} L(x, y) = -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i b_i + \min_{x \in \mathbb{R}^n} L(x, y) = -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i^T b_i ``` so the dual problem is ```math - \max_{y \in \mathcal{C}_i^*} \min_{x \in \R^n} -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i b_i. + \max_{y_i \in \mathcal{C}_i^*} \min_{x \in \mathbb{R}^n} -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i^T b_i. ``` - If ``Q_0`` is invertible, we can eliminate ``x`` and obtain that + If ``Q_0`` is invertible, we have ``x = Q_0^{-1}z(y)`` hence ```math - \min_{x \in \R^n} L(x, y) = -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i b_i + \min_{x \in \mathbb{R}^n} L(x, y) = -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i^T b_i ``` so the dual problem is ```math - \max_{y \in \mathcal{C}_i^*} -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i b_i + \max_{y_i \in \mathcal{C}_i^*} -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i^T b_i. ``` From fe975c4e5591015d7e105aaeaf02512890d501ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Beno=C3=AEt=20Legat?= Date: Wed, 19 Jan 2022 13:04:02 -0500 Subject: [PATCH 3/3] Address comments --- docs/src/background/duality.md | 97 ++++++++++++++++++---------------- 1 file changed, 51 insertions(+), 46 deletions(-) diff --git a/docs/src/background/duality.md b/docs/src/background/duality.md index d60e933962..76c063b727 100644 --- a/docs/src/background/duality.md +++ b/docs/src/background/duality.md @@ -124,7 +124,50 @@ details. ## Dual for problems with quadratic functions -Given a problem with quadratic functions: +### Quadratic Programs (QPs) + +For quadratic programs with only affine conic constraints, +```math +\begin{align*} +& \min_{x \in \mathbb{R}^n} & \frac{1}{2}x^TQ_0x + a_0^T x + b_0 +\\ +& \;\;\text{s.t.} & A_i x + b_i & \in \mathcal{C}_i & i = 1 \ldots m. +\end{align*} +``` +with cones ``\mathcal{C}_i \subseteq \mathbb{R}^{m_i}`` for ``i = 1, \ldots, m``, consider the Lagrangian function +```math +L(x, y) = \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \sum_{i = 1}^m y_i^T (A_i x + b_i). +``` +Let ``z(y)`` denote ``\sum_{i = 1}^m A_i^T y_i - a_0``, the Lagrangian can be rewritten as +```math +L(x, y) = \frac{1}{2}{x}^TQ_0x - z(y)^T x + b_0 - \sum_{i = 1}^m y_i^T b_i. +``` + +The condition ``\nabla_x L(x, y) = 0`` gives +```math +0 = \nabla_x L(x, y) = Q_0x + a_0 - \sum_{i = 1}^m y_i^T b_i +``` +which gives ``Q_0x = z(y)``. +This allows to obtain that +```math +\min_{x \in \mathbb{R}^n} L(x, y) = -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i^T b_i +``` +so the dual problem is +```math +\max_{y_i \in \mathcal{C}_i^*} \min_{x \in \mathbb{R}^n} -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i^T b_i. +``` +If ``Q_0`` is invertible, we have ``x = Q_0^{-1}z(y)`` hence +```math +\min_{x \in \mathbb{R}^n} L(x, y) = -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i^T b_i +``` +so the dual problem is +```math +\max_{y_i \in \mathcal{C}_i^*} -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i^T b_i. +``` + +### Quadratically Constrained Quadratic Programs (QCQPs) + +Given a problem with both quadratic function and quadratic objectives: ```math \begin{align*} & \min_{x \in \mathbb{R}^n} & \frac{1}{2}x^TQ_0x + a_0^T x + b_0 @@ -149,52 +192,14 @@ A pair of primal-dual variables $(x^\star, y^\star)$ is optimal if ```math \max_{y_i \in \mathcal{C}_i^*} L(x^\star, y). ``` - That is, for all ``i = 1, \ldots, m``, ``\frac{1}{2}x^TQ_ix + a_i^T x + b_i`` is - either zero or in the normal cone of ``\mathcal{C}_i^*`` at ``y^\star``. - For instance, if ``\mathcal{C}_i`` is ``\{ x \in \mathbb{R} : x \le 0 \}``, this means that - if ``\frac{1}{2}x^TQ_ix + a_i^T x + b_i`` is nonzero at ``x^\star`` then ``y_i^\star = 0``. - This is the classical complementary slackness condition. + That is, for all ``i = 1, \ldots, m``, ``\frac{1}{2}x^TQ_ix + a_i^T x + b_i`` + is either zero or in the + [normal cone](https://en.wikipedia.org/wiki/Normal_cone) of + ``\mathcal{C}_i^*`` at ``y^\star``. For instance, if ``\mathcal{C}_i`` is + ``\{ z \in \mathbb{R} : z \le 0 \}``, this means that if + ``\frac{1}{2}x^TQ_ix + a_i^T x + b_i`` is nonzero at ``x^\star`` then + ``y_i^\star = 0``. This is the classical complementary slackness condition. If ``\mathcal{C}_i`` is a vector set, the discussion remains valid with ``y_i(\frac{1}{2}x^TQ_ix + a_i^T x + b_i)`` replaced with the scalar product between ``y_i`` and the vector of scalar-valued quadratic functions. - -!!! note - For quadratic programs with only affine conic constraints, - ```math - \begin{align*} - & \min_{x \in \mathbb{R}^n} & \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \\ - & \;\;\text{s.t.} & A_i x + b_i & \in \mathcal{C}_i & i = 1 \ldots m. - \end{align*} - ``` - with cones ``\mathcal{C}_i \subseteq \mathbb{R}^{m_i}`` for ``i = 1, \ldots, m``, consider the Lagrangian function - ```math - L(x, y) = \frac{1}{2}x^TQ_0x + a_0^T x + b_0 - \sum_{i = 1}^m y_i^T (A_i x + b_i). - ``` - Let ``z(y)`` denote ``\sum_{i = 1}^m A_i^T y_i - a_0``, the Lagrangian can be rewritten as - ```math - L(x, y) = \frac{1}{2}{x}^TQ_0x - z(y)^T x + b_0 - \sum_{i = 1}^m y_i^T b_i. - ``` - - The condition ``\nabla_x L(x, y) = 0`` gives - ```math - 0 = \nabla_x L(x, y) = Q_0x + a_0 - \sum_{i = 1}^m y_i^T b_i - ``` - which gives ``Q_0x = z(y)``. - This allows to obtain that - ```math - \min_{x \in \mathbb{R}^n} L(x, y) = -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i^T b_i - ``` - so the dual problem is - ```math - \max_{y_i \in \mathcal{C}_i^*} \min_{x \in \mathbb{R}^n} -\frac{1}{2}x^TQ_0x + b_0 - \sum_{i = 1}^m y_i^T b_i. - ``` - If ``Q_0`` is invertible, we have ``x = Q_0^{-1}z(y)`` hence - ```math - \min_{x \in \mathbb{R}^n} L(x, y) = -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i^T b_i - ``` - so the dual problem is - ```math - \max_{y_i \in \mathcal{C}_i^*} -\frac{1}{2}z(y)^TQ_0^{-1}z(y) + b_0 - \sum_{i = 1}^m y_i^T b_i. - ```