Manage Gurobi environments in GurobiDirect

[simonbowly]

Fixes #2408

Summary/Motivation:

There are currently several limitations in the GurobiDirect interface:

1. Some Gurobi parameters cannot be used with the current approach, as they require explicitly creating a gurobipy Env object. This includes connection parameters for compute servers, token servers, and instant cloud (can be worked around via a license file, but this isn't always an ideal approach) and special parameters such as MemLimit.
2. There is no clean way to close Gurobi models and environments, which leaves license tokens in-use and compute server connections open longer than a user need them.
3. A user cannot retry acquiring a Gurobi license token (important in shared license environments) since the GurobiDirect class caches errors in global state.

Changes proposed in this PR:

Introduces a constructor flag manage_env for GurobiDirect (defaults to False), and two public methods .close() and .close_global(). If users set manage_env=True:

• GurobiDirect explicitly creates a Gurobi environment bound to the solver instance. This enables Gurobi resources to be properly freed by the solver object:

with SolverFactory('gurobi', solver_io='python', manage_env=True) as opt:
    opt.solve(model)
# All Gurobi models and environments are freed

• Calling .close() achieves the same result as the context manager:

opt = SolverFactory('gurobi', solver_io='python', manage_env=True)
try:
    opt.solve(model)
finally:
    opt.close()
# All Gurobi models and environments are freed

• Internally, solver options are passed to the Env constructor (instead of the Model, as is currently done) to allow environment-level connection parameters to be used:

options = {
    "CSManager": "<url>",
    "CSAPIAccessID": "<access-id>",
    "CSAPISecret": "<api-key>",
}
with SolverFactory('gurobi', solver_io='python', manage_env=True, options=options) as opt:
    opt.solve(model)  # Solved on compute server
# Compute server connection terminated

If manage_env=False (the default) is set, then users will get the old behaviour, which uses the Gurobi default/global environment. There are some minor changes:

• Calling .close(), or exiting the context properly disposes of all models created by the solver

with SolverFactory('gurobi', solver_io='python') as opt:
    opt.solve(model)
# Gurobi models created by `opt` are freed; the default/global Gurobi environment is still active

• Calling .close_global() disposes of models created by the solver, and disposes the Gurobi default environment. This will free all Gurobi resources assuming the user did not create any other models (e.g. via another GurobiDirect object with manage_env=False):

opt = SolverFactory('gurobi', solver_io='python')
try:
    opt.solve(model)
finally:
    opt.close_global()
# Gurobi models created by `opt` are freed, the default/global Gurobi environment is closed

Finally, the available() call no longer stores errors globally and repeats them back if users retry the check. So users can do the following to queue requests if they are using a shared license (regardless of whether manage_env is set to True or False):

with SolverFactory('gurobi', solver_io='python') as opt:
    while not available(exception_flag=False):
        time.sleep(1)
    opt.solve(model)

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[mrmundt]

Hi @simonbowly - We are very excited about this PR but need to have a more targeted discussion as a dev team to discuss backwards compatibility implications. We will get back to you ASAP and thoroughly appreciate your contribution!

[simonbowly]

Thanks @mrmundt! Are the current jenkins test failures relevant? They seem to be only codecov related.

Just one note I should add re: the tests. Some of the tests I added are really integration tests that rely on Gurobi single-use license behaviour, so they are skipped in CI. I can look at adding a few more unit tests if the changes seem reasonable.

[jsiirola]

@simonbowly: yes: the Jenkins failure is legitimate:

    def test_set_environment_parameters(self):
        # Solver options should handle parameters which must be set before the
        # environment is started (i.e. connection params, memory limits). This
        # can only work with a managed env.
    
        with SolverFactory(
            "gurobi_direct",
            manage_env=True,
            options={"ComputeServer": "/url/to/server"},
        ) as opt:
            # Check that the error comes from an attempted connection, not from setting
            # the parameter after the environment is started.
            with self.assertRaisesRegex(ApplicationError, "Could not resolve host"):
>               opt.solve(self.model)
E               AssertionError: "Could not resolve host" does not match "Could not create Model for <class 'pyomo.solvers.plugins.solvers.gurobi_direct.GurobiDirect'> solver plugin - gurobi message=Could not create Model - gurobi message=Unrecognized response (status 404, POST http:///url/to/server/api/v1/cluster/jobs, 'Cannot resolve hostname 'url' in explicit-proxy re')
E               
E               Set parameter TokenServer to value "*****"
E               Set parameter ComputeServer to value "/url/to/server"
E               "

pyomo/solvers/tests/checks/test_gurobi_direct.py:73: AssertionError

[simonbowly]

Thanks @jsiirola. I pushed a fix to that test and rebased.

[simonbowly]

I took one more pass to hopefully sort out that failing conda test 🤞

[simonbowly]

@blnicho I think the only remaining failing test is unrelated - TestKestrel.test_doc on linux/3.9/conda? The test_gurobi_direct cases are all passing in that run.

[michaelbynum]

@michaelbynum sorry to leave this hanging for so long, thanks for merging changes in the meantime!

I've added additional tests in 7c16c62 which cover similar behaviour to the single-use tests but via patching gurobipy functions.

One last thing I'm not sure on - where is the appropriate place to document the new option? There aren't solver-specific pages in the docs, is the docstring of GurobiDirect a suitable place?

I think the docstring of GurobiDirect is a good place.

[michaelbynum]

@simonbowly - One more question. Would it be better to allow users to pass in an instance of gurobipy.Env to the constructor rather than passing a flag to create one under the hood? Would that be more flexible at all? I'm happy either way, but I wanted to throw the idea out there.

[simonbowly]

Would it be better to allow users to pass in an instance of gurobipy.Env to the constructor rather than passing a flag to create one under the hood? Would that be more flexible at all?

I did think about that initially. The distinction would be:

import gurobipy as gp

with gp.Env(params={...}) as env, pyo.SolverFactory("gurobi_direct", env=env) as opt:
    opt.solve(model)

vs.

with pyo.SolverFactory("gurobi_direct", options={...}, manage_env=True) as opt:
    opt.solve(model)

where params in the first case and options in the second case are identical parameter dictionaries. I went with the second approach, reasoning that:

1. Interacting directly with the underlying solver library didn't seem in keeping with pyomo's design, and the dual context is a little more unwieldy.
2. The first approach doesn't gain anything: the Env object accepts the same parameter dictionary and provides the same closable context.

Unless there is a use case for creating one Env and sharing it across solvers? The canonical thing to do there though seems to be to re-use the solver for multiple models.

[simonbowly]

Added documentation with 317b8c5. That's everything I wanted to add, please review when time allows :-)

[simonbowly]

@blnicho I see this didn't make it into the release, did you want me to make any changes?

[blnicho]

@simonbowly we don't have any change requests at this time. We ran out of time to get a second review on this and needed to move forward with the release to meet a deadline for a downstream project.

[michaelbynum]

There was a pypy-3.9 test failure. I restarted that job. If it passes, we will merge.

Sorry for the delay, @simonbowly. It is my fault.

[simonbowly]

No problem at all! I really appreciate the time you all spent discussing & reviewing this PR, thanks again!

[andyvk85]

Very great job folks! Thank you!