Julia/JuMP wrapper

[odow]

The standard way that Julia wraps solvers is to make a HiGHS.jl package in a Git repository of the same name. This package handles installing binaries on users computers and provides two layers of wrappers:

1. C -> Julia. Effectively what you have in src/interfaces
2. Julia -> MathOptInterface. This is what is needed to use HiGHS from JuMP.

Are you okay if we move forward with this? Or do want the wrapper in src/interfaces?

We could either host the wrapper in JuliaOpt, or we could host it in ERGO-Code.

cc. @mtanneau

[jajhall]

Hi Oscar, I 've assigned Michael to respond to this - he's written all the HiGHS interfaces, and will communicate fluently with you. I've also chatted to Mathieu.

As for hosting, so long as both parties can edit the HiGHS.jl package, I don't particularly mind. WOuld it be easier for the Julia/JuMP community if it was hosted in JuliaOpt?

Julian

[odow]

Would it be easier for the Julia/JuMP community if it was hosted in JuliaOpt?

There's no difference for us. We just need a public Github repository. If you prefer, you could create ERGO-Code/HiGHS.jl and give me commit access to set it up.

[mtanneau]

I can definitely help with the Julia part.

[odow]

I can definitely help with the Julia part.

Yeah, if we have a BinaryBuilder for HiGHS, I'm in favor of deprecating Clp and adding a warning on start-up that COIN-OR is deprecating Clp in favor of HiGHS. This should give HiGHS a lot of users and stress-test it a bit. It would also resolve our Clp bugs/missing C wrapper issues.

[matbesancon]

I can also help on the Julia interface once the repository is set up

[feldmeier]

Hi all, I am making sure at the moment that the C -> Julia interface is up to date.

Would a JuMP interface build on top of the C-> Julia interface, or on C directly? Is there any advantage of one over the other approach?

[matbesancon]

Would a JuMP interface build on top of the C-> Julia interface, or on C directly?

The MathOptInterface (the glue between the JuMP interface and solvers) is usually built on top of the C interface. SCIP.jl is a good example:

• the src/wrapper folder is the C interface translated to Julia (it is generated using Clang.jl)
• the src/MOI_wrapper implements the MathOptInterface functions

[rschwarz]

Is there any advantage of one over the other approach?
I guess it makes sense to keep the (low-level?) C->Julia interface, if it's reasonably complete.

Some users might just want to pass matrix/vector data into the LP solver?

[matbesancon]

This is a rough template of what would be necessary for Julia users (not including the JuMP interface yet):
https://github.com/matbesancon/HiGHS.jl
The API is generated using Clang.jl pretty smoothly

(this repo can be moved to the ERGO-CODE organization of course)

[mtanneau]

Would a JuMP interface build on top of the C-> Julia interface, or on C directly?

Generally, we build the MOI wrapper on top of the Julia wrapper.

Is there any advantage of one over the other approach?

Performance-wise, there should be virtually no difference; the C->Julia wrapping is pretty tight.
Ease-of-coding-wise, it's hugely simpler to use the Julia functions than carrying ccalls around. It also means that if anything changess in the C interface, we only have to change the C->Julia part.

Finally, I'd expect those use cases:

• Using HiGHS to solve one LP through JuMP (bulk of users). The workflow then is typically one build-solve-get solution pass, and the C->Julia conversion is definitely not the bottleneck.
• Using HiGHS through MOI (though most of the time JuMP can do the job fine) for use within some optimization routine (e.g. a Benders or column-generation algorithm). The main difference is that here, you may have several modify-then-resolve operations.
• Using HiGHS directly through its Julia interface: this should be marginal.

[jajhall]

All this looks good to me. Just one point on "modify-then-resolve operations". You'll see that a warning message is displayed when many of the methods you'll need are called. These shouldn't be around for much longer. The SCIP interface (obviously) needs them, and once SCIP+HiGHS is solving MIPs reliably - and obviously it'll be giving these methods a real thrashing - I'll remove them.

I'm also conscious that we need some documentation on how to run HiGHS silently - it'll come soon enough!

[matbesancon]

The HiGHS.jl wrapper works with BinaryBuilder.jl, users can just add the package like any other Julia package without cmake-ing themselves.

[matbesancon]

I'm also conscious that we need some documentation on how to run HiGHS silently - it'll come soon enough!

@jajhall it would be great to expose the different enums and structs in the C API, so they can be generated automatically in wrappers

[feldmeier]

I thought about passing the enums through the C interface, but this might block interfaces to some programming languages, hence why I convert them to integers and them convert them back to enums (see e.g. the C# interface).

As far as I am aware, wrappers (e.g. for Julia) need some "individual" coding anyways, since the methods of the C interfaceneed to be wrapped in a class (so the users don't have to deal with the highs pointer).

I'm very open for suggestions though!

What do you mean by exposing the options?

[matbesancon]

I thought about passing the enums through the C interface, but this might block interfaces to some programming languages

We could define the types in a different header file, with highs_c_api.h and c_types.h or similar.

I think most binding generators can handle at least enums, which are converted to integers. Worst case, these which do not support types can only wrap around highs_c_api.h.

wrappers (e.g. for Julia) need some "individual" coding anyways, since the methods of the C interface need to be wrapped in a class

See jump-dev/HiGHS.jl#1, the methods of the C interface can be wrapped directly, I then added a very thin wrapper around the pointer which takes care of cleaning memory, based on how SCIP.jl does it

What do you mean by exposing the options?

Sorry, I meant types

[mlubin]

I think most binding generators can handle at least enums, which are converted to integers.

FWIW, enums are not safe to include in library APIs unless you're very careful about compiler options. The C standard does not define the sizeof(enum).

[matbesancon]

ok yes, we should have a way to communicate the constants though

[mtanneau]

This is how Mosek.jl does it

[matbesancon]

they indicate the file is generated automatically, which is both neat and impressive, do you know how they get it?

[mtanneau]

I don't know, if someone does it will be @ulfworsoe

[ulfworsoe]

Mosek.jl is generated from an abstract description of the MOSEK C interface. We already had the description (which is how we make all the low-level interfaces), so it was easy to do, but in most cases it's probably not the simplest way.

[odow]

Closing because we have HiGHS.jl in a good state now. We can use other issues to follow up on specific points.