Skip to content

Errors in the v1 benchmark run #193

Description

@siddharth-krishna

We are tracking and investigating the benchmark instances and solvers that returned an error in this spreadsheet:
https://docs.google.com/spreadsheets/d/18bb8xqZhydMzGq7jXmAsmFk5PNzjwoTUMws5lHWHqJ4/edit?usp=sharing

The latest versions of each solver, used in the main ranking in the home dashboard, have 20 errors:

Image

Most of these appear to be errors with the corresponding solvers, and we have opened issues to follow up with the solver developers.

Here are some other known errors/issues with previous solver versions (2023 and before):

  • SCIP from years 2022 and 2023 returns an unknown termination condition when it terminates due to reaching the duality gap tolerance. This is a bug in linopy which has been fixed by us in the latest version, but the latest linopy is not compatible with old SCIP versions.
  • CBC from 2023 also returns warning status for some instances, and again we have fixed this in the latest linopy, but because of the above issue the 2023 conda env uses an older linopy version. When the above is fixed, this will also be fixed.
  • GLPK reports an unknown termination condition when it reaches gap limit. Linopy needs to be fixed to report this correctly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions