DOC: update testing docs for alternative backends

[fancidev]

Is your feature request related to a problem? Please describe.

In a recent PR from me (gh-20766), 6/14 commits and 6/9 CI runs are used to address Array API incompatibility. This created unnecessary burden on the CI infrastructure and lengthens the development cycle.

Describe the solution you'd like.

I’d like to improve the situation by running those compatibility tests locally.

Ideally, I’d like to have the following:

1. An option in dev.py to run all the backends for xp-compatible tests. (Or maybe, do this by default.) Update: the option (-b) already exists.
2. Update the dev doc to advertise this option. Update: DOC/DEV: mention -b option in contributor guide on testing #20849
3. Maybe update requirements.txt to include the alternative backends.

Describe alternatives you've considered.

The first thing I did after seeing the CI errors was to install Jax, array_api_strict, and pytorch. I looked at the CI log to learn how to install them. (Basically with the [cpu] option.) Then I played a bit with each of them in a Jupyter Notebook. This helps understanding some of the errors, and I tested that the fixes at least work with the numpy backend. However, because I don’t know how to run the tests with other backends, I leave to the CI to run them.

Additional context (e.g. screenshots, GIFs)

No response

[rgommers]

1. An option in dev.py to run all the backends for xp-compatible tests.

This exists: -b all. Documentation from python dev.py test --help:

--array-api-backend  -b  ARRAY_BACKEND  Array API backend ('all', 'numpy', 'pytorch', 'cupy', 'array_api_strict', 'jax.numpy').

2. Update the dev doc to advertise this option.

That SGTM. Just a single-sentence mention though - we should really be leaning on the self-documenting interface and refer to --help for each dev.py subcommand.

3. Maybe update requirements.txt to include the alternative backends.

It's not really possible to install all alternative backends in a single env. array-api-strict gets you a way to the goal here, and is already included in requirements/test.txt and in environment.yml.

Installing PyTorch, CuPy or JAX can be tricky - especially if you want more than one in the same env (they often conflict with each other). I don't see a reasonable way to include them as optional requirements. What can be done instead is provide lock files for:

• scipy-dev-torch
• scipy-dev-cupy
  -scipy-dev-jax

and maybe other flavors:

• scipy-dev-torch-cu12
• scipy-dev-all (it's doable as a lock file, but requires manual tweaks often to create the lock file so it's not doable by simply listing all packages as dependencies)

[rgommers]

I think we probably can and should enable testing with array-api-strict by default now, if it is installed, rather than gating that on an environment variable. That will be an improvement.

[lucascolley]

Just a single-sentence mention though - we should really be leaning on the self-documenting interface and refer to --help for each dev.py subcommand.

A sentence linking out to https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#adding-tests should do it.

[fancidev]

Thanks for the info @rgommers ! So the trick is to place --help after test in python dev.py test --help. Then the options are right there! I have previously tried python dev.py --help test and didn't see the details.

The -b option works nicely by detecting the installed backends and running different backends only for the tests that are marked with xp compatible. This is greatly helpful!

[lucascolley]

I've updated the title for the actionable point here. There is also:

I think we probably can and should enable testing with array-api-strict by default now, if it is installed, rather than gating that on an environment variable. That will be an improvement.