Using qpsolvers_benchmark with CI

[ottapav]

My intention is to have a script that will be running on the server that will run benchmarks, generate reports, and export figures automatically. I wrote the script in batch, which eventually calls:

• python qpsolvers_benchmark\benchmark.py maros_meszaros_dense_posdef run
• python qpsolvers_benchmark\benchmark.py maros_meszaros_dense_posdef plot runtime high_accuracy --solvers cvxopt ecos osqp qpswift quadprog scs nppro

The problem is that the first command is waiting for input from the command line to be written in the report. The second command opens a plot window that will remain open. Is there a command to export figures and an option that will suppress the command line input such that script can be executed on the server, generating results without need of human interaction?

[stephane-caron]

The problem is that the first command is waiting for input from the command line to be written in the report.

This one shouldn't be an issue: you can already bypass this behavior using the --author argument.

The second command opens a plot window that will remain open.

This one can be easily changed. It mainly results from this instruction:

https://github.com/qpsolvers/qpsolvers_benchmark/blob/ae64b476c69a2ec429490b8c0c4ffdb740283a04/qpsolvers_benchmark/plot_metric.py#L78

If instead of plt.show we called plt.savefig it would save the figure to a file instead. For instance, we could switch to this second behavior when a --savefig argument is passed to the CLI. I've started a PR for it at #66, let me know your thoughts.

[ottapav]

@stephane-caron Sounds great! Does the figure be saved with or without a title? I would prefer not to include the title in the saved figure.

[stephane-caron]

In the long run we will want an easier way for benchmark users to create custom plots: #67. Let's continue the discussion in the PR for short-term feedback.