Skip to content

Integrate benchmarking#200

Merged
johannahaffner merged 16 commits intopatrick-kidger:devfrom
johannahaffner:pytest-benchmark
Dec 27, 2025
Merged

Integrate benchmarking#200
johannahaffner merged 16 commits intopatrick-kidger:devfrom
johannahaffner:pytest-benchmark

Conversation

@johannahaffner
Copy link
Collaborator

@johannahaffner johannahaffner commented Dec 24, 2025

This follows up on #128.

Two small things:

  • the script stays - having the benchmarks output this would entail customising the fixture provided by pytest-benchmark, and plotting is also something one would do every few runs, rather than every run - in particular whenever one would like to look at a comparison between solvers or implementation with others.
  • We'll continue to require benchmarking runs to specify pytest benchmarks --benchmark-only - if we'd put --benchmark-only into benchmarks/conftest.py then it would get collected when pytest is invoked and it would override our default --benchmark-skip.

And one weird thing: if I remove the OrderedDict, I still get results jumbled between solvers. But as you (@patrick-kidger) note, it should not be needed. Playing with it for a bit I can remove it in two of the three places, but the underlying logic is not apparent to me, nor am I happy to trust that this would work for a different number of solvers / problems / something.
I think the pragmatic thing to do here is to simply document the puzzling necessity of OrderedDict in this script. (Low-key itching to check whether this is an upstream thing...)

@johannahaffner
Copy link
Collaborator Author

Tests are failing due to JAX 0.8.2 being used with 3.12, and a different version of JAX being used with 3.10, for which JAX has recently dropped support.

@johannahaffner johannahaffner marked this pull request as ready for review December 25, 2025 16:41
Copy link
Owner

@patrick-kidger patrick-kidger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks reasonable to me, with only one comment, can we target dev instead of main?

@johannahaffner johannahaffner changed the base branch from main to dev December 27, 2025 18:59
@johannahaffner
Copy link
Collaborator Author

Now failing checks because dev includes a module that triggers the error being addressed in patrick-kidger/lineax#184. Tests pass and benchmarks run locally (benchmarks are skipped in CI and no other tested code is added).

@johannahaffner johannahaffner merged commit b1a908b into patrick-kidger:dev Dec 27, 2025
0 of 2 checks passed
@johannahaffner johannahaffner deleted the pytest-benchmark branch December 27, 2025 19:37
patrick-kidger pushed a commit that referenced this pull request Feb 8, 2026
* Implement pytest-benchmark based setup for systematic performance evaluation of Optimistix' solvers.

* version bump for sif2jax requirements

* add semi-recent matplotlib version to specify a minimum

* no more monkeypatching

* set EQX_ON_ERROR with os.environ

* give a reason for skipping compilation tests

* Add L-BFGS solvers to benchmark suite

* clarify what --benchmark-autosave will do.

* remove strict dtype promotion rules - benchmarks are not tests, so we don't need them here. We would otherwise have to use context management for any comparison to Optax minimisers.

* state purpose of --scipy flag more clearly.

* improve contribution guidelines, inline decorator, specify pyright errata

* pyproject.toml from main

* add sif2jax

* move benchmark dependencies to tests group

* add benchmark-skip option

* add example to contributing guidelines, document OrderedDict workaround, adapt to sif2jax usage of properties.

---------

Co-authored-by: Johanna Haffner <johanna.haffner@bsse.ethz.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants