Add Lazy/Heuristic MIP callbacks

[blegat]

This PR defines the necessary attributes and submittables for lazy and heuristic callbacks.
The docstrings are based on #670.

Closes #670

cc @mtanneau

[odow]

So it seems like each callback has three attributes

• an attribute to set the callback function
• an attribute to pass the submittable
• an attribute to query the solution

Should we standardize naming on

• [NewCallback]Callback, e.g., LazyConstraintCallback
• [NewCallback], e.g., LazyConstraint
• [NewCallback]VariablePrimal, e.g., LazyConstraintVariablePrimal

[codecov-io]

Codecov Report

Merging #782 into master will decrease coverage by 0.01%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master     #782      +/-   ##
==========================================
- Coverage    95.1%   95.09%   -0.02%     
==========================================
  Files          80       80              
  Lines        8518     8519       +1     
==========================================
  Hits         8101     8101              
- Misses        417      418       +1

Impacted Files	Coverage Δ
src/attributes.jl	85.05% <50%> (-0.99%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c393870...620fae9. Read the comment docs.

[mtanneau]

In terms of interface, Gurobi and CPLEX are good examples that we can follow.

Gurobi has a generic GRBcbget function to query information. You say what you want and where in the solution process you are. You get an error if you query something you're not allowed to (e.g. ask for the integer incumbent during pre-solve). To submit stuff (e.g. a cut or a heuristic solution), there are specific functions for each case. The whole thing is wrapped into a single callback function (the mycallback function in the example).

CPLEX follows the same idea (query what from where in the optimization), but has specific functions for querying/posting information from/to the solver. Only a generic callback function is registered.

Pointers to the docs:

• CPLEX
  • Overview
  • What can be queried from where
• Gurobi
  • Callback codes
  • Callback example (in C)
  • GRBcbget function

[blegat]

One approach would be to have only one callback MIPCallback and you would decide what to do based on whether you can get a fractional and/or an integer solution and whether you can submit a lazy constraint and/or a heuristic solution. That would require adding can_get and can_submit. supports does not do it because supports cannot depend on the state of the solver.

A second approach would be to have FractionalNodeCallback where you are at a fractional node and IntegerNodeCallback where you are at an integer node. What you can get is fixed so no need for can_get but you can check what you can submit with can_submit.

The third approach is having LazyCallback and HeuristicCallback where both what you can get and what you can submit is fixed (this is the current approach of this PR).

So to summarize in a table:

	Integer Solution	Fractional Solution
Lazy Constraint	LazyCallback
Heuristic Solution		HeuristicCallback

• The first solution is having a single callback for the 4 cells.
• The second solution is having a callback for each column.
• The third solution is having a callback for each diagonal cell.

[mtanneau]

IMO, MIPCallback is much preferable. (this approach is what I refer to as "generic callback", not to be confused with solver-independent callback).

• It is lighter: you only ever use MOI.get, MOI.submit and possibly MOI.add and MOI.set.
  This is also more in line with MOI's general philosophy.
• It is more flexible than specific callbacks. For instance, all informational callbacks would be automatically covered.
  Importantly, one can define a new kind of callback by just defining their own attributes.
• It's the direction that the main solvers are taking, and thus would make the wrapping more intuitive.

FYI, here is a table of what MIP solvers currently support. Y is yes, N is no, ? is I don't know. Docs are linked in the names.

Solver	Cbc	CPLEX	GLPK	Gurobi	SCIP	Xpress
Generic callback	?	Y	Y	Y	N/?	N
Heuristics	?	Y	Y	Y	?	Y
Lazy constraint	?	Y	Y	Y	?	Y
User cuts	?	Y	?	Y	?	Y

With respect to can_get and can_submit, there are two cases that should be distinguished:

1. I'm using a MIP solver that does not support lazy constraints, and I'm trying to add one.
   This should raise an "MOI-level" error, because I'm trying to do something the solver doesn't support. In particular, the compiler can infer it.
2. I'm using Gurobi, and I want to add a lazy constraint at the root.
   This should raise a "Gurobi-level" error, and typically cannot be inferred at compile time (unless you type every single callback code).

This distinction is the same as, say querying the solution value after calling optimize!. For instance, if Gurobi ran into some error, you get a Gurobi error when asking for the optimal solution.

Back to MIPCallback. Solvers with generic callbacks will provide a where argument to the callback function, indicating what information can and cannot be queried/submitted (the correspondence would be indicated in the solver's docs).

One question I don't have an answer to is: what about solvers that don't do generic callbacks (e.g. XPress)? How do you break a generic callback into usercut callback / heuristic callback, etc?
This approach also puts a toll on solvers' wrappers. But callbacks are very solver-specific anyway, so it makes sense that most of the heavy lifting is done by solvers themselves.

[blegat]

It is more flexible than specific callbacks. For instance, all informational callbacks would be automatically covered.
Importantly, one can define a new kind of callback by just defining their own attributes.

Not sure to follow. Solvers can already define new callbacks by defining new attributes, this PR will not change the situation in that respect.
For informational callbacks, this is what they should do. How would informational callbacks fit in MIPCallback ?

here is a table of what MIP solvers currently support

Thanks a lot !

With respect to can_get and can_submit, there are two cases that should be distinguished:

The case 1. is already covered by supports. For case 2., if there is enough info in the callback_data (let's say where is also in callback_data), then we might be able to implement can_get(::Optimizer, SomeAttribute(callback_data) and can_submit(::Optimizer, SomeSubmittable(callback_data). This is still run-time but it allows to avoid having solvers throwing errors.

Back to MIPCallback. Solvers with generic callbacks will provide a where argument to the callback function, indicating what information can and cannot be queried/submitted (the correspondence would be indicated in the solver's docs).

I agree, this could be included in callback_data.

One question I don't have an answer to is: what about solvers that don't do generic callbacks (e.g. XPress)? How do you break a generic callback into usercut callback / heuristic callback, etc?

You could put in the callback_data that this is a usercut, etc... so that the wrapper can redirect appropriately and have enough info to answer can_submit, ...

[mtanneau]

How would informational callbacks fit in MIPCallback ?

To implement an informational callback, just define a MIPCallback that only queries the information you want (as long as the solver you're using supports querying it).
I guess my point was: if MOI has a generic callback, then we don't need to change anything the day someone wants to support some fancy feature. I think it would make the codebase easier to maintain.

if there is enough info in the callback_data (let's say where is also in callback_data), then we might be able to implement can_get(::Optimizer, SomeAttribute(callback_data) and can_submit(::Optimizer, SomeSubmittable(callback_data). This is still run-time but it allows to avoid having solvers throwing errors.

Wouldn't this be duplicating code? Solvers already perform this check, and raise an error if needed.
Implementing can_get(::Optimizer, ...) and can_submit(::Optimizer, ...) essentially does the same, but at the solver's MOI wrapper level.
Why not pass on the call to the solver, and intercept the error instead?

[blegat]

Suppose you have some algo to generate a heuristic solution but it takes some time. With can_submit you can check before running the algo if it's worth it

[mtanneau]

You could put in the callback_data that this is a usercut, etc... so that the wrapper can redirect appropriately and have enough info to answer can_submit, ...

The problematic part is more about registering the callback in the first place.
Say you define (at MOI level) a generic callback function my_callback. That callback function is then passed on to the Xpress.jl wrapper, which passes it to the Xpress object.
What callback do you register with Xpress? You'd have to pretty much cover all possible cases...

One possibility (but quite cumbersome) would be to have Xpress.jl register a callback function for every possible specific callback in Xpress, each of them calling my_callback with the right argument.

[blegat]

To correct something I said in earlier posts: In fact, we don't need can_submit, supports is already appropriate. When you write MOI.supports(optimizer, MOI.LazyConstraint(callback_data)), the optimizer can see in callback_data whether lazy constraints can be submitted inside the callback (since it sees which type of callback it is) hence it can return false or true depending on this. This does not depend on the state of optimizer but only on the value of the submittable so it satisfy the definition of supports and we don't need to add can_submit.

Thinking more about it, I prefer solution 2 of #782 (comment). That is, similarly to MOI.TerminationStatus which tells why the solvers stopped. The type of the callback would say why the solver called, e.g.

• It found a feasible solution and asks whether you have any last word before accepting it (e.g. by rejecting it or submitting lazy constraints you were hidding from him). Corresponds to
  • where == GRB_CB_MIPSOL for Gurobi.
  • context == CPX_CALLBACKCONTEXT_CANDIDATE for CPLEX.
  • XPRSaddcbpreintsol for Xpress.
  • glp_ios_reason == GLP_IROWGEN for GLPK.
• It found a infeasible solution and asks whether you want to help (e.g. by submitting heuristic solutions or user cuts). Corresponds to
  • where == GRB_CB_MIPNODE && GRB_CB_MIPNODE_STATUS == GRB_OPTIMAL for Gurobi.
  • context == CPX_CALLBACKCONTEXT_RELAXATION for CPLEX.
  • XPRSaddcboptnode for Xpress.
  • glp_ios_reason == GLP_IHEUR for GLPK (in which case, the supports return true for HeuristicSolution and false for UserCut).
  • glp_ios_reason == GLP_ICUTGEN for GLPK (in which case, the supports return false for HeuristicSolution and true for UserCut).

This type of fallbacks kind of defines what the user can MOI.get but one can do try-catch to be sure to be solver independent.
For submitting, the user can do supports to see if it's worth doing all the computation to get something to submit, e.g.

if MOI.supports(optimizer, MOI.UserCut(callback_data))
    func, set = # Long computation
    MOI.submit(optimizer, MOI.UserCut(callback_data), (func, set))
end

@mtanneau You suggest using rather solution 3. So a single callback and you have a way to query why it was called (or from where it was called but it seems kind of the same), let's say by MOI.get(optimizer, MOI.WhyDidYouCall()).
It seems rather similar in practice:

• In solution 2, for each why, you define a different a different callback.
• In solution 3, for each why, you define a different value of MOI.WhyDidYouCall.

However, there are several reasons for which IMO solution 2 is more appropriate for solver independent callbacks in MOI:

• In solution 3, it's impossible to have a solver-independent enum that covers all possible why (or where) to return for MOI.WhyDidYouCall so we would need to return different types which is slightly inconsistent with the rest of MOI where MOI.get usually returns enums, functions, sets or numbers. Some types will be defined in MOI and the solver specific ones will be defined in the solver wrappers.
• In solution 3, having to deal with everything in the same callback would give a spagetti code difficult to maintain. If you have a code for, say a Gurobi specific callback in addition to your code for solver-indepedent ones, you have to add it to the mix. And since the type returned by MOI.WhyDidYouCall is defined in the Gurobi wrapper, you have to use a trick to only execute this ifelse condition if the Gurobi package is loaded. While in solution 2, you can MOI.set the solver-specific callback next to using Gurobi; solver = with_optimizer(Gurobi.Optimizer) in a gurobi.jl file that you load when you use Gurobi.
• In solution 3, we want to define one callback to rule them all. But to be consistent, that means NLP callbacks should also be part of it, that means that MOI.WhyDidYouCall could return NeedObjectiveValueAt, NeedGradientAt, NeedHessianAt. Here we clearly see that having everything in the same function is limiting since we need to mix the NLP autodiff code with the code for generating lazy constraints etc... all in the same functions. While it is feasible in theory to create a function grouping all different callbacks in the wrappers, it is inconvenient as it would need to recreate this function everytime a callback is added.

I am not saying that solution 3 was a bad design decision for Gurobi. The argument above only apply for solver-independent callbacks for MOI. Not for solver-dependent ones in C.
For instance, I suppose that it might be convenient to have a single function in C so that you can share values with static variables, this clearly does not apply to Julia where you can easily defined closures, etc...

One possibility (but quite cumbersome) would be to have Xpress.jl register a callback function for every possible specific callback in Xpress, each of them calling my_callback with the right argument.

Yes, that would be the way if we go for solution 3.

[rschwarz]

FYI, here is a table of what MIP solvers currently support. Y is yes, N is no, ? is I don't know. Docs are linked in the names.

It's not clear to me what you mean by "generic callback" (but it might be covered by SCIP's event handlers). Other than that, I'm confident that SCIP provides functionality that is equivalent (and goes beyond) what other solvers offer in terms of callbacks. However, the plugin system in SCIP typically requires that the user implement multiple callback methods and only imperfect mappings are possible (as we did with the MPB-wrapper of SCIP).
See SCIP docs and related SCIP.jl issue: #94. Because SCIP's plugins are so idiosyncratic, I think it's best not to try and make solver-independent callbacks work now.

[mtanneau]

I still want to believe in generic callbacks, but... I don't see any of it being practical (nor consensual) anytime soon.

So @blegat I'm overturning my previous comments, and I support having a small set of specific callbacks that expose a limited set of functionalities. That should be enough to satisfy most users and hopefully be merged soon.

It's not clear to me what you mean by "generic callback"

@rschwarz same terminology as CPLEX' generic callback.
Basically, a "generic callback" wraps-up all possible callbacks you may think of.

[blegat]

PR updated based on the discussion above as well as discussion with @ccoffrin, @harshangrjn and @kaarthiksundar.

[mtanneau]

Continuing the discussion (@blegat this is mostly aligned with what you're proposing).

I would suggest to go with the following minimal set of functionalities which should be supported by most (if not all) MIP solvers:

• Heuristic callback: submit a heuristic solution. The user can query the current value of the primal solution, which is (generally) a fractional point.
• User cut callback: submit a cut that's violated by the current fractional point. The user can query the current value of the solution, which is in general fractional. The current solution may be integer (e.g., if the current LP relaxation yields an integer point).
• Lazy constraint callback: submit a constraint that is violated by the current point, and may cut some integer point. The user can query the current value of the primal solution, which may be fractional (e.g., at a node) or integer (e.g., at an incumbent).

To motivate this, here's a compilation of what (some) MIP solvers supports in terms of callback functionality.
H is heuristic (submit a heuristic solution), U is user cut (cut a fractional point), L is lazy constraint (cut an integer or fractional point).

Solver	Node relaxation	Incumbent
CPLEX	H, U	H, L
GLPK	H, L, U	X
Gurobi	H, L, U	L
Xpress	H, U	?

For instance: Gurobi allows to submit heuristic solutions, user cuts and lazy constraints after solving a node relaxation, and only lazy constraints when an incumbent is found.

Notes & caveats.

• CPLEX:
  • Docs do not specify in which context one can post a heuristic solution, so I put it in both cases.
  • No explicit mention of "lazy constraint" in the docs (none that I could find). For rejecting and incumbent, the function is CPXcallbackrejectcandidate which can submit constraints that cut off the integer point. No guarantee these "lazy constraints" will be enforced though
• GLPK:
  • Incumbent callback (GLP_IBINGO) is for information only.
    I don't know whether a node callback is called every time an incumbent is found by one of GLPK's internal heuristics (and this is not mentioned in the docs).
• Gurobi:
  • Heuristic solution can only be added when at a node relaxation.
  • Lazy constraints may be added at node relaxation, or when an incumbent is found. In both cases, the user has access to the value of the current primal solution (either fractional or integer).
• Xpress:
  • I could not find any mention of "lazy constraints" in Xpress docs. Either lazy constraints can be added through XPRSaddcuts, or there are no "lazy constraint" equivalent in Xpress, or there's a function I missed.
  • Docs (Further info getconstraintconstant for entire b vector #3) say cuts can only be added at a node relaxation
  • Incumbent solution can be rejected (see here), but it's not clear whether there's a procedure to include a violated cut from an incumbent.

[rschwarz]

A question for understanding: Is the idea that the user can only add a single callback function for each type (lazy constraint, heuristic) via the new attribute?

So, if I have a model where I want to use represent two separate constraints via lazy constraints, I would have to create a wrapper callback that checks violation and adds cuts for either of them?

[blegat]

A question for understanding: Is the idea that the user can only add a single callback function for each type (lazy constraint, heuristic) via the new attribute?

Yes, in the current design there is only one callback. If you set another callback it overwrites the previous one. The wrapper approach would be a solution.

[blegat]

@mtanneau I agree with your latest comment, let's define HeuristicCallback, UserCutCallback and LazyConstraintCallback. That corresponds to solution 3 of #782 (comment)
I have removed RejectSolution, it can be added later.

I have updated the PR, last call for objections.

[mtanneau]

Couple of comments dealing with callback-wise conventions.