From aaae8cca9603fb766f957fb43723b6b23be041e8 Mon Sep 17 00:00:00 2001 From: odow Date: Wed, 1 Sep 2021 15:18:29 +1200 Subject: [PATCH 1/6] [docs] add tutorial on latency --- docs/make.jl | 1 + docs/src/tutorials/latency.md | 267 ++++++++++++++++++++++++++++++++++ 2 files changed, 268 insertions(+) create mode 100644 docs/src/tutorials/latency.md diff --git a/docs/make.jl b/docs/make.jl index c0ebaad576..c494be7b06 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -30,6 +30,7 @@ makedocs( "tutorials/mathprogbase.md", "tutorials/bridging_constraint.md", "tutorials/manipulating_expressions.md", + "tutorials/latency.md", ], "Manual" => [ "manual/standard_form.md", diff --git a/docs/src/tutorials/latency.md b/docs/src/tutorials/latency.md new file mode 100644 index 0000000000..de918d3f3e --- /dev/null +++ b/docs/src/tutorials/latency.md @@ -0,0 +1,267 @@ +# Latency + +MathOptInterface suffers the "time-to-first-solve" problem of start-up latency. + +This hurts both the user- and developer-experience of MathOptInterface. In the +first case, because simple models have a multi-second delay before solving, and +in the latter, because our tests take so long to run! + +This page contains some advice on profiling and fixing latency-related problems +in the MathOptInterface.jl repository. + +## Background + +Before reading this part of the documentation, you should familiarize yourself +with the reasons for latency in Julia and how to fix them. + + * Read the blogposts on julialang.org on [precompilation](https://julialang.org/blog/2021/01/precompile_tutorial/) + and [SnoopCompile](https://julialang.org/blog/2021/01/snoopi_deep/) + * Read the [SnoopCompile](https://timholy.github.io/SnoopCompile.jl/stable/) + documentation. + * Watch Tim Holy's [talk at JuliaCon 2021](https://www.youtube.com/watch?v=rVBgrWYKLHY) + * Watch the [package development workshop at JuliaCon 2021](https://www.youtube.com/watch?v=wXRMwJdEjX4) + +## Causes + +There are a three main causes of latency in MathOptInterface: + + 1. A large number of types + 2. Lack of method ownership + 3. Type-instability in the bridge layer + +### A large number of types + +Julia is very good at specializing method calls based on the input type. Each +specialization has a compilation cost, but the benefit of faster run-time +performance. + +The best-case scenario is for a method to be called a large number of times with +a single set of argument types. The worst-case scenario is for a method to be +called a single time for a large set of argument types. + +Because of MathOptInterface's _function-in-set_ formulation, we fall into the +worst-case situation. + +This is a fundamental limitation of Julia, so there isn't much we can do about +it. However, if we can precomile MathOptInterface, much of the cost can be +shifted from start-up latency to the time it takes to precompile a package on +installation. + +However, there are two things which make MathOptInterface hard to precomile... + +### Lack of method ownership + +Lack of method ownership happens when a call is made using a mix of structs and +methods from different modules. Because of this, no single module "owns" the +method that is being dispatched, and so it cannot be precompiled. + +!!! tip + This is a slightly simplified explanation. Read the [precompilation tutorial](https://julialang.org/blog/2021/01/precompile_tutorial/) + for a more in-depth discusison on back-edges. + +Unfortunately, the design of MOI means that this is a frequent occurence! We +have a bunch of types in `MOI.Utilities` that wrap types defined in external +packages (i.e., the `Optimizer`s), which implement methods of functions defined +in `MOI` (e.g., `add_variable`, `add_constraint`). + +Here's a simple example of method-ownership in practice: +```julia +module MyMOI + struct Wrapper{T} + inner::T + end + optimize!(x::Wrapper) = optimize!(x.inner) +end + +module MyOptimizer + using ..MyMOI + struct Optimizer end + MyMOI.optimize!(x::Optimizer) = 1 +end + +using SnoopCompile +model = MyMOI.Wrapper(MyOptimizer.Optimizer()) +julia> tinf = @snoopi_deep MyMOI.optimize!(model) +InferenceTimingNode: 0.008256/0.008543 on InferenceFrameInfo for Core.Compiler.Timings.ROOT() with 1 direct children +``` +The result is that there was one method that required type inference. If we +visualize `tinf`: +```julia +using ProfileView +ProfileView.view(flamegraph(tinf)) +``` +we see a flamegraph with a large red-bar indicating that the method +`MyMOI.optimize(MyMOI.Wrapper{MyOptimizer.Optimizer})` cannot be precompiled. + +To fix this, we need to designate a module to "own" that method (i.e., create a +back-edge). The easiest way to do this is for `MyOptimizer` to call +`MyMOI.optimize(MyMOI.Wrapper{MyOptimizer.Optimizer})` during +`using MyOptimzier`. Let's see that in practice: +```julia +module MyMOI + struct Wrapper{T} + inner::T + end + optimize(x::Wrapper) = optimize(x.inner) +end + +module MyOptimizer + using ..MyMOI + struct Optimizer end + MyMOI.optimize(x::Optimizer) = 1 + # The syntax of this let-while loop is very particular: + # * `let ... end` keeps everything local to avoid polluting the MyOptimizer + # namespace + # * `while true ... break end` runs the code once, and forces Julia to + # compile the inner loop, rather than interpret it. + let + while true + model = MyMOI.Wrapper(Optimizer()) + MyMOI.optimize(model) + break + end + end +end + +using SnoopCompile +model = MyMOI.Wrapper(MyOptimizer.Optimizer()) +julia> tinf = @snoopi_deep MyMOI.optimize(model) +InferenceTimingNode: 0.006822/0.006822 on InferenceFrameInfo for Core.Compiler.Timings.ROOT() with 0 direct children +``` +There are now `0` direct children that required type inference because the +method was already stored in `MyOptimizer`! + +Unfortunately, this trick only works if the call-chain is fully inferrable. If +there are breaks (due to type instability), then the benefit of doing this is +reduced. And unfortunately for us, the design of MathOptInterface has a lot of +type instabilities... + +### Type instability in the bridge layer + +Most of MathOptInterface is pretty good at ensuring type-stability. However, a +key component is not type stable, and that is the bridging layer. + +In particular, the bridging layer defines [`Bridges.LazyBridgeOptimizer`](@ref), +which has fields like: +```julia +struct LazyBridgeOptimizer + constraint_bridge_types::Vector{Any} + constraint_node::Dict{Tuple{Type,Type},ConstraintNode} + constraint_types::Vector{Tuple{Type,Type}} +end +``` +This is because the `LazyBridgeOptimizer` needs to be able to deal with any +_function-in-set_ type passed to it, and we also allow users to pass additional +bridges that they defined in external packages. + +So to recap, MathOptInterface suffers package latency because: + + 1. there are a large number of types and functions... + 2. and these are split between multiple modules, including external packages... + 3. and there are type-instabilities like those in the bridging layer. + +## Resolutions + +There are no magic solutions to reduce latency. [Issue #1313](https://github.com/jump-dev/MathOptInterface.jl/issues/1313) +tracks progress on reducing latency in MathOptInterface. + +A useful script is the following (replace GLPK as needed): +```julia +using MathOptInterface, GLPK +const MOI = MathOptInterface + +function example_diet(optimizer, bridge) + category_data = [ + 1800.0 2200.0; + 91.0 Inf; + 0.0 65.0; + 0.0 1779.0 + ] + cost = [2.49, 2.89, 1.50, 1.89, 2.09, 1.99, 2.49, 0.89, 1.59] + food_data = [ + 410 24 26 730; + 420 32 10 1190; + 560 20 32 1800; + 380 4 19 270; + 320 12 10 930; + 320 15 12 820; + 320 31 12 1230; + 100 8 2.5 125; + 330 8 10 180 + ] + bridge_model = if bridge + MOI.instantiate(optimizer; with_bridge_type=Float64) + else + MOI.instantiate(optimizer) + end + model = MOI.Utilities.CachingOptimizer( + MOI.Utilities.UniversalFallback(MOI.Utilities.Model{Float64}()), + MOI.Utilities.AUTOMATIC, + ) + MOI.Utilities.reset_optimizer(model, bridge_model) + MOI.set(model, MOI.Silent(), true) + nutrition = MOI.add_variables(model, size(category_data, 1)) + for (i, v) in enumerate(nutrition) + MOI.add_constraint( + model, + MOI.SingleVariable(v), + MOI.GreaterThan(category_data[i, 1]), + ) + MOI.add_constraint( + model, + MOI.SingleVariable(v), + MOI.LessThan(category_data[i, 2]), + ) + end + buy = MOI.add_variables(model, size(food_data, 1)) + MOI.add_constraint.(model, MOI.SingleVariable.(buy), MOI.GreaterThan(0.0)) + MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE) + f = MOI.ScalarAffineFunction(MOI.ScalarAffineTerm.(cost, buy), 0.0) + MOI.set(model, MOI.ObjectiveFunction{typeof(f)}(), f) + for (j, n) in enumerate(nutrition) + f = MOI.ScalarAffineFunction( + MOI.ScalarAffineTerm.(food_data[:, j], buy), + 0.0, + ) + push!(f.terms, MOI.ScalarAffineTerm(-1.0, n)) + MOI.add_constraint(model, f, MOI.EqualTo(0.0)) + end + MOI.optimize!(model) + term_status = MOI.get(model, MOI.TerminationStatus()) + @assert term_status == MOI.OPTIMAL + MOI.add_constraint( + model, + MOI.ScalarAffineFunction( + MOI.ScalarAffineTerm.(1.0, [buy[end-1], buy[end]]), + 0.0, + ), + MOI.LessThan(6.0), + ) + MOI.optimize!(model) + @assert MOI.get(model, MOI.TerminationStatus()) == MOI.INFEASIBLE + return +end + +if length(ARGS) > 0 + bridge = get(ARGS, 2, "") != "--no-bridge" + println("Running: $(ARGS[1]) $(get(ARGS, 2, ""))") + @time example_diet(GLPK.Optimizer, bridge) + @time example_diet(GLPK.Optimizer, bridge) + exit(0) +end +``` + +You can create a flame-graph via +```julia +using SnoopComile +tinf = example_diet(GLPK.Optimizer, true) +using ProfileView +ProfileView.view(flamegraph(tinf)) +``` + +Here's how things looked in mid-August 2021: +![flamegraph](https://user-images.githubusercontent.com/8177701/129137091-fb78c4a0-2208-4418-968f-24ff15370351.png) + +There are a few opportunities for improvement (non-red flames, particularly on +the right). But the main problem is a large red (non-precompilable due to method +ownership) flame. From e457c42adad4ff0b5430df9469b9e0a9188066ab Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Wed, 1 Sep 2021 17:13:44 +1200 Subject: [PATCH 2/6] Update latency.md --- docs/src/tutorials/latency.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/src/tutorials/latency.md b/docs/src/tutorials/latency.md index de918d3f3e..107382d190 100644 --- a/docs/src/tutorials/latency.md +++ b/docs/src/tutorials/latency.md @@ -1,3 +1,12 @@ +```@meta +CurrentModule = MathOptInterface +DocTestSetup = quote + using MathOptInterface + const MOI = MathOptInterface +end +DocTestFilters = [r"MathOptInterface|MOI"] +``` + # Latency MathOptInterface suffers the "time-to-first-solve" problem of start-up latency. From 0952b5bb3e90a86d98204abce1528ccaba575a5f Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Wed, 1 Sep 2021 17:15:40 +1200 Subject: [PATCH 3/6] Update latency.md --- docs/src/tutorials/latency.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/tutorials/latency.md b/docs/src/tutorials/latency.md index 107382d190..167b70cb0c 100644 --- a/docs/src/tutorials/latency.md +++ b/docs/src/tutorials/latency.md @@ -32,7 +32,7 @@ with the reasons for latency in Julia and how to fix them. ## Causes -There are a three main causes of latency in MathOptInterface: +There are three main causes of latency in MathOptInterface: 1. A large number of types 2. Lack of method ownership @@ -105,7 +105,7 @@ we see a flamegraph with a large red-bar indicating that the method To fix this, we need to designate a module to "own" that method (i.e., create a back-edge). The easiest way to do this is for `MyOptimizer` to call `MyMOI.optimize(MyMOI.Wrapper{MyOptimizer.Optimizer})` during -`using MyOptimzier`. Let's see that in practice: +`using MyOptimizer`. Let's see that in practice: ```julia module MyMOI struct Wrapper{T} From d84cba7c01a334441985c8ee41caaddd5a890508 Mon Sep 17 00:00:00 2001 From: odow Date: Thu, 2 Sep 2021 09:05:28 +1200 Subject: [PATCH 4/6] Updates --- docs/src/tutorials/latency.md | 65 ++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/docs/src/tutorials/latency.md b/docs/src/tutorials/latency.md index de918d3f3e..fedba3d45a 100644 --- a/docs/src/tutorials/latency.md +++ b/docs/src/tutorials/latency.md @@ -43,11 +43,11 @@ Because of MathOptInterface's _function-in-set_ formulation, we fall into the worst-case situation. This is a fundamental limitation of Julia, so there isn't much we can do about -it. However, if we can precomile MathOptInterface, much of the cost can be +it. However, if we can precompile MathOptInterface, much of the cost can be shifted from start-up latency to the time it takes to precompile a package on installation. -However, there are two things which make MathOptInterface hard to precomile... +However, there are two things which make MathOptInterface hard to precompile... ### Lack of method ownership @@ -57,7 +57,7 @@ method that is being dispatched, and so it cannot be precompiled. !!! tip This is a slightly simplified explanation. Read the [precompilation tutorial](https://julialang.org/blog/2021/01/precompile_tutorial/) - for a more in-depth discusison on back-edges. + for a more in-depth discussion on back-edges. Unfortunately, the design of MOI means that this is a frequent occurence! We have a bunch of types in `MOI.Utilities` that wrap types defined in external @@ -67,20 +67,21 @@ in `MOI` (e.g., `add_variable`, `add_constraint`). Here's a simple example of method-ownership in practice: ```julia module MyMOI - struct Wrapper{T} - inner::T - end - optimize!(x::Wrapper) = optimize!(x.inner) +struct Wrapper{T} + inner::T end +optimize!(x::Wrapper) = optimize!(x.inner) +end # MyMOI module MyOptimizer - using ..MyMOI - struct Optimizer end - MyMOI.optimize!(x::Optimizer) = 1 -end +using ..MyMOI +struct Optimizer end +MyMOI.optimize!(x::Optimizer) = 1 +end # MyOptimizer using SnoopCompile model = MyMOI.Wrapper(MyOptimizer.Optimizer()) + julia> tinf = @snoopi_deep MyMOI.optimize!(model) InferenceTimingNode: 0.008256/0.008543 on InferenceFrameInfo for Core.Compiler.Timings.ROOT() with 1 direct children ``` @@ -99,32 +100,33 @@ back-edge). The easiest way to do this is for `MyOptimizer` to call `using MyOptimzier`. Let's see that in practice: ```julia module MyMOI - struct Wrapper{T} - inner::T - end - optimize(x::Wrapper) = optimize(x.inner) +struct Wrapper{T} + inner::T end +optimize(x::Wrapper) = optimize(x.inner) +end # MyMOI module MyOptimizer - using ..MyMOI - struct Optimizer end - MyMOI.optimize(x::Optimizer) = 1 - # The syntax of this let-while loop is very particular: - # * `let ... end` keeps everything local to avoid polluting the MyOptimizer - # namespace - # * `while true ... break end` runs the code once, and forces Julia to - # compile the inner loop, rather than interpret it. - let - while true - model = MyMOI.Wrapper(Optimizer()) - MyMOI.optimize(model) - break - end +using ..MyMOI +struct Optimizer end +MyMOI.optimize(x::Optimizer) = 1 +# The syntax of this let-while loop is very particular: +# * `let ... end` keeps everything local to avoid polluting the MyOptimizer +# namespace +# * `while true ... break end` runs the code once, and forces Julia to compile +# the inner loop, rather than interpret it. +let + while true + model = MyMOI.Wrapper(Optimizer()) + MyMOI.optimize(model) + break end end +end # MyOptimizer using SnoopCompile model = MyMOI.Wrapper(MyOptimizer.Optimizer()) + julia> tinf = @snoopi_deep MyMOI.optimize(model) InferenceTimingNode: 0.006822/0.006822 on InferenceFrameInfo for Core.Compiler.Timings.ROOT() with 0 direct children ``` @@ -162,7 +164,8 @@ So to recap, MathOptInterface suffers package latency because: ## Resolutions -There are no magic solutions to reduce latency. [Issue #1313](https://github.com/jump-dev/MathOptInterface.jl/issues/1313) +There are no magic solutions to reduce latency. +[Issue #1313](https://github.com/jump-dev/MathOptInterface.jl/issues/1313) tracks progress on reducing latency in MathOptInterface. A useful script is the following (replace GLPK as needed): @@ -254,7 +257,7 @@ end You can create a flame-graph via ```julia using SnoopComile -tinf = example_diet(GLPK.Optimizer, true) +tinf = @snoopi_deep example_diet(GLPK.Optimizer, true) using ProfileView ProfileView.view(flamegraph(tinf)) ``` From 7ec2d3afd34b95339f0912947dce23657e2619bf Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Thu, 2 Sep 2021 10:39:55 +1200 Subject: [PATCH 5/6] Update latency.md --- docs/src/tutorials/latency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/tutorials/latency.md b/docs/src/tutorials/latency.md index 252b02018b..a932793f0b 100644 --- a/docs/src/tutorials/latency.md +++ b/docs/src/tutorials/latency.md @@ -68,7 +68,7 @@ method that is being dispatched, and so it cannot be precompiled. This is a slightly simplified explanation. Read the [precompilation tutorial](https://julialang.org/blog/2021/01/precompile_tutorial/) for a more in-depth discussion on back-edges. -Unfortunately, the design of MOI means that this is a frequent occurence! We +Unfortunately, the design of MOI means that this is a frequent occurrence! We have a bunch of types in `MOI.Utilities` that wrap types defined in external packages (i.e., the `Optimizer`s), which implement methods of functions defined in `MOI` (e.g., `add_variable`, `add_constraint`). From 81f68b52738fb48aba5d7cfdb2a8b012decf9aee Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Thu, 2 Sep 2021 11:30:59 +1200 Subject: [PATCH 6/6] Update latency.md --- docs/src/tutorials/latency.md | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/docs/src/tutorials/latency.md b/docs/src/tutorials/latency.md index a932793f0b..b46de1aaeb 100644 --- a/docs/src/tutorials/latency.md +++ b/docs/src/tutorials/latency.md @@ -214,19 +214,11 @@ function example_diet(optimizer, bridge) MOI.set(model, MOI.Silent(), true) nutrition = MOI.add_variables(model, size(category_data, 1)) for (i, v) in enumerate(nutrition) - MOI.add_constraint( - model, - MOI.SingleVariable(v), - MOI.GreaterThan(category_data[i, 1]), - ) - MOI.add_constraint( - model, - MOI.SingleVariable(v), - MOI.LessThan(category_data[i, 2]), - ) + MOI.add_constraint(model, v, MOI.GreaterThan(category_data[i, 1])) + MOI.add_constraint(model, v, MOI.LessThan(category_data[i, 2])) end buy = MOI.add_variables(model, size(food_data, 1)) - MOI.add_constraint.(model, MOI.SingleVariable.(buy), MOI.GreaterThan(0.0)) + MOI.add_constraint.(model, buy, MOI.GreaterThan(0.0)) MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE) f = MOI.ScalarAffineFunction(MOI.ScalarAffineTerm.(cost, buy), 0.0) MOI.set(model, MOI.ObjectiveFunction{typeof(f)}(), f)