[FileFormats.MPS] slowing reading of .mps file

[JunchengLiamLi]

The read_from_file() function is quite slow on my .mps file. It takes 33 seconds to build a model from the .mps file, while the C_API of HiGHS (call from Julia) takes only about the 8 seconds. Please see the below:

@time model_JuMP = read_from_file("2025-1126-SCUC-noncon.mps");
33.343285 seconds (236.12 M allocations: 14.443 GiB, 25.32% gc time)

@time HiGHS.Highs_readModel(model_HiGHS,"2025-1126-SCUC-noncon.mps");
7.832455 seconds (1.50 k allocations: 78.234 KiB, 0.05% compilation time)

Just wondering if there is any way to build a JuMP.Model faster from .mps files.

Many thanks!

[odow]

There are known overheads associated with read_from_file. See my recent talk on MOI: https://youtu.be/M31xoZGyj9w?si=Ma60aWbEAV4RAePt&t=1667

We could perhaps be a bit faster, but the times aren't really directly comparable between JuMP and HiGHS because JuMP is doing significantly more work.

A better question to ask is: why do you care that reading this file takes a few seconds? In most cases this shouldn't be a significant runtime part of any optimization-related workflows.

[JunchengLiamLi]

Using MOI.readModel() is indeed faster than JuMP.read_from_file(). As shown in the following:
julia> @time dest = MOI.FileFormats.Model(format = MOI.FileFormats.FORMAT_MPS);

julia> @time MOI.read_from_file(dest, "2025-1126-SCUC-noncon.mps");
22.695831 seconds (150.83 M allocations: 9.657 GiB, 29.85% gc time)

A better question to ask is: why do you care that reading this file takes a few seconds? In most cases this shouldn't be a significant runtime part of any optimization-related workflows.

That is because in this scenario, the reading of mps files is used in a self-defined primal heuristic that aims to find a good feasible solution as soon as possible, instead of building a model, passing to solver and waiting for the result. Moreover, this heuristic could be revoked multiple times in one solution procedure. Therefore, a 20-second improvement in the speed of reading from .mps file every time is not trivial.

I think that MOI could offer unique values in this type of usage, where both computational speed and flexibility of manipulating the model are desired. Tools like Pyomo are very flexible but could be slow, while the C API of HiGHS is fast but it is not easy to manipulate the model with it. By manipulating the model, I mean adding new binary variables to model as well as new constraints bonding the new variables and existing binary variables.

[odow]

That is because in this scenario, the reading of mps files is used in a self-defined primal heuristic

Why is reading an MPS file part of the heuristic?

[JunchengLiamLi]

Why is reading an MPS file part of the heuristic?

Two reasons:

1. use another solver to solve the sub-MIP. For example, modify the sub-MIP based heuritisic in HiGHS so that SCIP is used to solve the sub-MIP problem
2. easier to change the model substaintially. For example, it seems to me that the following requires quite some work to implement with the C API of HiGHS:

• Step 1: add some binary variables to the model.
• Step 2: add constraints, each of which binds one new variable with an existing variable.
• Step 3: add one constraint requiring that the new variables sum to 1.

It seems to me it is a lot easier to implemt this in MOI/JuMP than HiGHS.jl (using the C API), which makes the code easier to maintain and less prone to bugs.

[odow]

I have so many questions.

You're editing the sub-MIP heuristic in HiGHS by changing the C++ code, and as part of that, you're writing the MPS file to disk, starting Julia, reading the MPS file, doing a bunch of stuff, then writing back a file of solutions, which you then load back into C++ and keep going?

Is the purpose just research to help prototype new heuristic algorithms? It can't be performance critical, right?

Would it be sufficient to have access to the heuristic callback for HiGHS?

We just generally haven't prioritised file I/O because you should avoid it at all costs.

[odow]

Can you share the MPS file?

[odow]

I made a small improvement in #2892

[JunchengLiamLi]

I have so many questions.

You're editing the sub-MIP heuristic in HiGHS by changing the C++ code, and as part of that, you're writing the MPS file to disk, starting Julia, reading the MPS file, doing a bunch of stuff, then writing back a file of solutions, which you then load back into C++ and keep going?

Is the purpose just research to help prototype new heuristic algorithms? It can't be performance critical, right?

Would it be sufficient to have access to the heuristic callback for HiGHS?

We just generally haven't prioritised file I/O because you should avoid it at all costs.

Oh, I just realised that I probably did not explain it well.

I am actually using the heuristic callback for HiGHS, with its C API called from Julia. I completely agree with you that it does not make sense to change the C++ code and call Julia at the same time.

The reasons why I am using files I/O at the moment:

1. I am struggling with building model from the C function Highs_passLp() and thus using the naive alternative of writing and reading model from the disk. I will certainly try to make Highs_passLp() work and replace the file I/O method.
2. I am modifying the model heavily at some point of the heuristic algorithm, which is seemingly quite hard to implement. That is why I resort to MOI/JuMP for that particular step. I do not know how to build a MOI model from a highs instance in C, apart from the file I/O approach.

[JunchengLiamLi]

Can you share the MPS file?

No, the file is confidential. However, I can share a mps file with similar size, created from UnitCommitment.jl

[odow]

Is this really just asking for the JuMP-level heuristic callback support to be added to HiGHS.jl? jump-dev/HiGHS.jl#313

[JunchengLiamLi]

Well, this is a different one. Just imagine that if I want to use cuPDLPx https://github.com/MIT-Lu-Lab/cuPDLPx inside a heuristic callback of HiGHS. Just thought using the mps file would be a convenient way to communicate between cuPDLPx and HiGHS. You might ask why not pass the original mps file to cuPDLPx. That is because I want cuPDLPx to solve a problem that has already been reduced by some components in HiGHS (e.g., MIP presolving). Just wondering if that makes sense.