[RFC] Should we use measured or reported solver runtimes?

[siddharth-krishna]

The current benchmarking scripts measure the wall clock time (note: we are switching to time.perf_counter in an in-flight PR) taken by the call to linopy.solvers.solve_problem, which is a function in the Python library that acts as a uniform solver interface and handles parsing the solver responses and results. This means the measured runtime includes not just the time taken to solve the LP/MILP problem, but also the time taken by the solver to parse the input file and write the output file, or to check that the license (if any) is valid. Note that since all our benchmarks are LP/MPS files already, this measured time does not include the time taken by linopy to serialize the problem to a file.

An alternative is to use the reported runtime for solving from the solver. Some solvers expose this in their APIs:

Gurobipy: model.Runtime
PySCIPOpt: model.getSolvingTime()
HiGHSpy: highs.getRunTime()

This runtime presumably excludes the time for input/output and license checks.

Arguments for using measured runtime:

1. It is closer to the end-user's experience: if a solver takes an extra 10s because it reads input files inefficiently or has to consult a remote license server, then the user will feel that the solver is 10s slower when they use it on their problems.
2. It does not require trusting that solvers do not (un)intentionally report inaccurate solving runtimes. (This can be mitigated to some extent by comparing reported runtimes to measured runtimes while analyzing benchmark results.)
3. If we use reported runtime, we will have to check that all solvers use a consistent definition of runtime (e.g. all of them exclude the time to parse inputs).

Arguments for using reported runtime:

1. It measures the core of what the solver does (solving the problem), and our performance history page reflects improvements to solver algorithms, not just faster input file parsers.
2. The measured runtime includes any time spent by linopy in the solver interface. For instance, if a solver does not have a Python API or linopy doesn't use one (e.g. CBC), then this runtime includes the time taken by linopy to parse the result/solution file.

We invite everyone to discuss the above in this issue. (@jajhall, @mattmilten, @pfetsch, @jjhforrest, and your teams, if you'll forgive the 2nd notification from me today, we would love to hear your thoughts on this as well.)

For now, we plan to record both reported and measured runtime in the results, and check if there are any big discrepancies in our v1 benchmark run next week. But we will await the discussion here before deciding which runtime to use in the website's analyses and graphs.

[mattmilten]

What the benchmark is supposed to measure is the solving or run time of the different solvers. Time to read the model, write the solution, etc., should not be included for better comparisons.
So I vote for reported runtime as main metric. Having the measured total time in addition surely cannot hurt. We should certainly revisit this discussion after the first set of results is in and we may see significant differences or not.

[pfetsch]

I support the view of @mattmilten .

[siddharth-krishna]

@jajhall, does highspy.getRunTime() return the total time spent by HiGHS (e.g. including time to parse input file) or just the runtime of the solver?

[jajhall]

It's just the run-time of the solver

[odow]

I'm actually in favour of total measured runtime. My view is that these benchmarks are end-user focused. There's probably minimal difference between measured and reported, so why not do something that is definitely uniform across all solvers. As a sanity check you could verify that parsing/loading/etc isn't a significant part. But equally: if it is, that could be a performance bug that can be fixed.