Serializing a Warm Problem

[adishavit]

I have a big model that takes a while to compile and solve. I am using Guroi as my solver.
However, successive warm solves (after changing a few cp.Parameter()s) run very quickly.

How can I serialize my large warm model offline for later use with the warm flag?

[akshayka]

We don't yet have a serialization strategy, but it's on our todo list.

In the meantime, can you try using Python pickle?

[adishavit]

Pickle doesn't work after a call to solve().
Here's a minimal example:

import pickle

import cvxpy as cp

n = 50
x = cp.Variable(n, integer=True)
y = cp.Variable(n, integer=True)
q = cp.Parameter(value=5)
o = cp.Maximize(sum(x))
p = cp.Problem(o, [x <= y, y <= q])
p.solve()  # <=  REMOVE THIS AND IT WORKS

with open('problem.pickle', 'wb') as f:
    pickle.dump(p, f, pickle.HIGHEST_PROTOCOL)

This fails with:

Traceback (most recent call last):
  File "...problem_pickling.py", line 14, in <module>
    pickle.dump(p, f, pickle.HIGHEST_PROTOCOL)
TypeError: can't pickle PyCapsule objects

Any suggestions?

[adishavit]

I have made some progress on this, but it requires some hacks, and is still missing some functionality to work seamlessly.
You may see a working sample gist here: problem_pickling.py.

First, note that besides being unable to pickle Problems with internal [Gurobi] binaries (as in the comment above), some cp.Problems cannot be pickled at all - even before solving - due to exceeding the recursion depth (this is apparently related to the internal index class and self references).

Hack 1: To bypass this, I must re-generate the unsolved initial problem/model again when I need to load my model.
Let's call my initial problem original_problem and the new, identical, unsolved problem warm_problem.

However, both models must have an identical structure and also all constrained variables, expressions and parameters etc. must have the same identical IDs.
For this I use:
Hack 2 : I must reset the global ID_COUNTER.count (from cvxpy.lin_ops.lin_utils) to the same value (e.g. 0) before creating the models - otherwise the variable ids won't match when unpacking the model solution. See line 27 in the attached.

Amazingly after solving the restored problem, all its associated variables (i.e. the last x) now contain the correct values as (perhaps) expected, and subsequent [warm] solves may proceed normally.

Unfortunately (and this is the only major issue for me with this approach), any parameters associated with the restored problem, i.e. _q are still disconnected from the model and have to be "re-linked" by id to find the correct Parameter for changing in the next run. See line 94 in the attached.

I am wondering if there is something that I'm missing that will ensure that _q gets linked to the correct parameter after loading the model - just like x. (the issue is probably related to the fact that I must serialize Problem.parameters__cache__ and that Problem.parameters() uses the @perf.compute_once facility, but I'm not sure what is going on there and how to fix this.)