-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #39 from AKS1996/update_documentation
Update documentation
- Loading branch information
Showing
5 changed files
with
327 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| # Manual | ||
|
|
||
| !!! note | ||
| As of now, this package only works for optimization models that can be written either in convex conic form or convex quadratic form. | ||
|
|
||
|
|
||
| ## Supported objectives & constraints - scheme 1 | ||
|
|
||
| For `QPTH`/`OPTNET` backend (using `backward!` method), the package supports following `Function-in-Set` constraints: | ||
|
|
||
| | MOI Function | MOI Set | | ||
| |:-------|:---------------| | ||
| | `SingleVariable` | `GreaterThan` | | ||
| | `SingleVariable` | `LessThan` | | ||
| | `SingleVariable` | `EqualTo` | | ||
| | `ScalarAffineFunction` | `GreaterThan` | | ||
| | `ScalarAffineFunction` | `LessThan` | | ||
| | `ScalarAffineFunction` | `EqualTo` | | ||
|
|
||
| and the following objective types: | ||
|
|
||
| | MOI Function | | ||
| |:-------:| | ||
| | `SingleVariable` | | ||
| | `ScalarAffineFunction` | | ||
| | `ScalarQuadraticFunction` | | ||
|
|
||
|
|
||
| ## Supported objectives & constraints - scheme 2 | ||
|
|
||
| For `DiffCP`/`CVXPY` backend (using `backward_conic!` method), the package supports following `Function-in-Set` constraints: | ||
|
|
||
| | MOI Function | MOI Set | | ||
| |:-------|:---------------| | ||
| | `VectorOfVariables` | `Nonnegatives` | | ||
| | `VectorOfVariables` | `Nonpositives` | | ||
| | `VectorOfVariables` | `Zeros` | | ||
| | `VectorOfVariables` | `SecondOrderCone` | | ||
| | `VectorOfVariables` | `PositiveSemidefiniteConeTriangle` | | ||
| | `VectorAffineFunction` | `Nonnegatives` | | ||
| | `VectorAffineFunction` | `Nonpositives` | | ||
| | `VectorAffineFunction` | `Zeros` | | ||
| | `VectorAffineFunction` | `SecondOrderCone` | | ||
| | `VectorAffineFunction` | `PositiveSemidefiniteConeTriangle` | | ||
|
|
||
| and the following objective types: | ||
|
|
||
| | MOI Function | | ||
| |:-------:| | ||
| | `SingleVariable` | | ||
| | `ScalarAffineFunction` | | ||
|
|
||
|
|
||
| ## Creating a differentiable optimizer | ||
|
|
||
| You can create a differentiable optimizer over an existing MOI solver by using the `diff_optimizer` utility. | ||
| ```@docs | ||
| diff_optimizer | ||
| ``` | ||
|
|
||
| ## Adding new sets and constraints | ||
|
|
||
| Usage interface DiffOpt models is same as other MOI Optimizers. So the same `add_variable`, `add_constraint` utilities can be used. | ||
|
|
||
|
|
||
| ## Projections on cone sets | ||
|
|
||
| DiffOpt requires taking projections and finding projection gradients of vectors while computing the jacobians. For this purpose, we use [MathOptSetDistances.jl](https://github.com/matbesancon/MathOptSetDistances.jl), which is a dedicated package for computing set distances, projections and projection gradients. | ||
|
|
||
|
|
||
| ## Conic problem formulation | ||
|
|
||
| !!! note | ||
| As of now, the package is using `SCS` geometric form for affine expressions in cones. | ||
|
|
||
| Consider a convex conic optimization problem in its primal (P) and dual (D) forms: | ||
| ```math | ||
| \begin{split} | ||
| \begin{array} {llcc} | ||
| \textbf{Primal Problem} & & \textbf{Dual Problem} & \\ | ||
| \mbox{minimize} & c^T x \quad \quad & \mbox{minimize} & b^T y \\ | ||
| \mbox{subject to} & A x + s = b \quad \quad & \mbox{subject to} & A^T y + c = 0 \\ | ||
| & s \in \mathcal{K} & & y \in \mathcal{K}^* | ||
| \end{array} | ||
| \end{split} | ||
| ``` | ||
|
|
||
| where | ||
| - ``x \in R^n`` is the primal variable, ``y \in R^m`` is the dual variable, and ``s \in R^m`` is the primal slack | ||
| variable | ||
| - ``\mathcal{K} \subseteq R^m`` is a closed convex cone and ``\mathcal{K}^* \subseteq R^m`` is the corresponding dual cone | ||
| variable | ||
| - ``A \in R^{m \times n}``, ``b \in R^m``, ``c \in R^n`` are problem data | ||
|
|
||
| In the light of above, DiffOpt differentiates program variables ``x``, ``s``, ``y`` w.r.t pertubations/sensivities in problem data i.e. ``dA``, ``db``, ``dc``. This is achieved via *implicit differentiation* and *matrix differential calculus*. | ||
|
|
||
| > Note that the primal (P) and dual (D) are self-duals of each other. Similarly for the constraints we support, ``\mathcal{K}`` is same in format as ``\mathcal{K}^*``. | ||
|
|
||
| ### Reference articles | ||
|
|
||
| - [_Differentiating Through a Cone Program_](https://arxiv.org/abs/1904.09043) - Akshay Agrawal, Shane Barratt, Stephen Boyd, Enzo Busseti, Walaa M. Moursi, 2019 | ||
| - A fast and differentiable QP solver for PyTorch. Crafted by Brandon Amos and J. Zico Kolter. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| # Solving conic with PSD and SOC constraints | ||
|
|
||
| Consider an example program | ||
|
|
||
| ```math | ||
| \begin{split} | ||
| \begin{array} {llcc} | ||
| \mbox{minimize} & | ||
| \left\langle | ||
| \left[ | ||
| \begin{array} {ccc} | ||
| 2 & 1 & 0 \\ | ||
| 1 & 2 & 1 \\ | ||
| 0 & 1 & 2 | ||
| \end{array} | ||
| \right], | ||
| X \right\rangle | ||
| + x_0 & & \\ | ||
| \mbox{subject to} & | ||
| \left\langle | ||
| \left[ | ||
| \begin{array} {ccc} | ||
| 1 & 0 & 0 \\ | ||
| 0 & 1 & 0 \\ | ||
| 0 & 0 & 1 | ||
| \end{array} | ||
| \right], | ||
| X \right\rangle | ||
| + x_0 & = & 1, \\ | ||
| & | ||
| \left\langle | ||
| \left[ | ||
| \begin{array}{ccc} | ||
| 1 & 1 & 1 \\ | ||
| 1 & 1 & 1 \\ | ||
| 1 & 1 & 1 | ||
| \end{array} | ||
| \right], | ||
| X \right\rangle + x_1 + x_2 | ||
| & = & 1/2, \\ | ||
| & (x_0, x_1, x_2) \in \mathbb{Q}^3 \text{ or } x_0 \geq \sqrt{{x_1}^2 + {x_2}^2} \\ | ||
| & X \succeq 0, X \in \mathbb{S}^3_{+} | ||
| \end{array} | ||
| \end{split} | ||
| ``` | ||
| where | ||
| ```math | ||
| \mathbb{S}^n_{+} = | ||
| \left\lbrace | ||
| X \in \mathbb{S}^n: z^T X z \geq 0, \quad \forall z \in \mathbb{R}^n | ||
| \right\rbrace, | ||
| ``` | ||
|
|
||
| > Refered from Mosek examples: https://docs.mosek.com/9.2/toolbox/tutorial-sdo-shared.html#example-sdo1 | ||
|
|
||
| ## Equivalent DiffCP program to differentiate | ||
| ```python | ||
| import numpy as np | ||
| import cvxpy as cp | ||
| from scipy import sparse | ||
| import diffcp | ||
|
|
||
| A = sparse.csc_matrix((11+1,7+1), dtype=np.float64) | ||
| A[2 , 1] = 1.0 | ||
| A[3 , 1] = -1.0 | ||
| A[9 , 1] = -0.45 | ||
| A[10, 1] = 0.45 | ||
| A[11, 1] = -0.45 | ||
| A[2 , 2] = 1.0 | ||
| A[4 , 2] = -1.0 | ||
| A[9 , 2] = -0.8 | ||
| A[10, 2] = 0.318198 | ||
| A[11, 2] = -0.1 | ||
| A[2 , 3] = 1.0 | ||
| A[5 , 3] = -1.0 | ||
| A[9 , 3] = -0.9 | ||
| A[2 , 4] = 1.0 | ||
| A[6 , 4] = -1.0 | ||
| A[9 , 4] = -0.225 | ||
| A[2 , 5] = 1.0 | ||
| A[7 , 5] = -1.0 | ||
| A[9 , 5] = -0.1125 | ||
| A[10, 5] = 0.1125 | ||
| A[11, 5] = -0.1125 | ||
| A[2 , 6] = 1.0 | ||
| A[8 , 6] = -1.0 | ||
| A[11, 6] = -0.225 | ||
| A[9 , 7] = 1.0 | ||
| A[11, 7] = 1.0 | ||
|
|
||
| A = A[1:, 1:] | ||
|
|
||
| # equivalent to: https://github.com/jump-dev/MathOptInterface.jl/blob/master/src/Test/contconic.jl#L2575 | ||
|
|
||
| cone_dict = { | ||
| diffcp.POS: 7, | ||
| diffcp.PSD: [2], | ||
| diffcp.ZERO: 1 | ||
| } | ||
|
|
||
| b = np.array([0.0, 10.0, -0.0, -0.0, -0.0, -0.0, -0.0, -0.0, 0.0, 0.0, 0.0]) | ||
| c = np.array([-0.0, -0.0, -0.0, -0.0, -0.0, -0.0, -1.0]) | ||
|
|
||
| x, y, s, D, DT = diffcp.solve_and_derivative(A, b, c, cone_dict) | ||
| print(x) # MOI.VariablePrimal | ||
| print(s) # MOI.ConstraintPrimal | ||
| print(y) # MOI.ConstraintDual | ||
|
|
||
|
|
||
| dx, dy, ds = D(sparse.csc_matrix(np.ones((11,7))), np.ones(11), np.ones(7)) | ||
| print(dx) | ||
| print(ds) | ||
| print(dy) | ||
| ``` | ||
|
|
||
| ## Equivalent DiffOpt program | ||
| ```julia | ||
| using SCS | ||
| using DiffOpt | ||
| using MathOptInterface | ||
|
|
||
| const MOI = MathOptInterface; | ||
|
|
||
|
|
||
| model = diff_optimizer(SCS.Optimizer) | ||
| MOI.set(model, MathOptInterface.Silent(), true) | ||
|
|
||
| δ = √(1 + (3*√2+2)*√(-116*√2+166) / 14) / 2 | ||
| ε = √((1 - 2*(√2-1)*δ^2) / (2-√2)) | ||
| y2 = 1 - ε*δ | ||
| y1 = 1 - √2*y2 | ||
| obj = y1 + y2/2 | ||
| k = -2*δ/ε | ||
| x2 = ((3-2obj)*(2+k^2)-4) / (4*(2+k^2)-4*√2) | ||
| α = √(3-2obj-4x2)/2 | ||
| β = k*α | ||
|
|
||
| X = MOI.add_variables(model, 6) | ||
| x = MOI.add_variables(model, 3) | ||
|
|
||
| vov = MOI.VectorOfVariables(X) | ||
|
|
||
| cX = MOI.add_constraint( | ||
| model, | ||
| MOI.VectorAffineFunction{Float64}(vov), MOI.PositiveSemidefiniteConeTriangle(3) | ||
| ) | ||
|
|
||
| cx = MOI.add_constraint( | ||
| model, | ||
| MOI.VectorAffineFunction{Float64}(MOI.VectorOfVariables(x)), MOI.SecondOrderCone(3) | ||
| ) | ||
|
|
||
| c1 = MOI.add_constraint( | ||
| model, | ||
| MOI.VectorAffineFunction( | ||
| MOI.VectorAffineTerm.(1:1, MOI.ScalarAffineTerm.([1., 1., 1., 1.], [X[1], X[3], X[end], x[1]])), | ||
| [-1.0] | ||
| ), | ||
| MOI.Zeros(1) | ||
| ) | ||
|
|
||
| c2 = MOI.add_constraint( | ||
| model, | ||
| MOI.VectorAffineFunction( | ||
| MOI.VectorAffineTerm.(1:1, MOI.ScalarAffineTerm.([1., 2, 1, 2, 2, 1, 1, 1], [X; x[2]; x[3]])), | ||
| [-0.5] | ||
| ), | ||
| MOI.Zeros(1) | ||
| ) | ||
|
|
||
| objXidx = [1:3; 5:6] | ||
| objXcoefs = 2*ones(5) | ||
| MOI.set(model, MOI.ObjectiveFunction{MOI.ScalarAffineFunction{Float64}}(), | ||
| MOI.ScalarAffineFunction(MOI.ScalarAffineTerm.([objXcoefs; 1.0], [X[objXidx]; x[1]]), 0.0)) | ||
| MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE) | ||
|
|
||
| sol = MOI.optimize!(model) | ||
|
|
||
| # fetch solution | ||
| x = sol.primal | ||
| s = sol.slack | ||
| y = sol.dual | ||
|
|
||
| println("x -> ", round.(x; digits=3)) | ||
| println("s -> ", round.(s; digits=3)) | ||
| println("y -> ", round.(y; digits=3)) | ||
|
|
||
| # perturbations in the parameters | ||
| dA = ones(11, 9) | ||
| db = ones(11) | ||
| dc = ones(9) | ||
|
|
||
| # differentiate and get the gradients | ||
| dx, dy, ds = backward_conic!(model, dA, db, dc) | ||
|
|
||
| println("dx -> ", round.(dx; digits=3)) | ||
| println("ds -> ", round.(ds; digits=3)) | ||
| println("dy -> ", round.(dy; digits=3)) | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters