Test all Python versions with Hatch 1.8.0

[aazuspan]

The new Hatch release will auto-install missing Python versions when using an environment matrix for testing! I did a quick test by adding:

[[tool.hatch.envs.test.matrix]]
python = ["3.8", "3.9", "3.10", "3.11", "3.12"]

to the pyproject.toml, updating my hatch, and running hatch run test:all, and it successfully ran the whole test matrix (takes a while the first time, but it looks like it caches installs for future runs). This solves all the trouble I've had in the past getting tox to find the right Python installations.

@grovduck I don't think this is a high priority, or something we necessarily even want to run every time, but it would be nice to have the option to test old Python versions and catch those sneaky typing incompatibilities before they hit the CI. I figured you might be interested in this for other projects that you've migrated to Hatch as well.

[grovduck]

@aazuspan, absolutely - great find! After struggling for a bit to figure out how to update pipx and hatch, I think I have this running as well and can see my new virtualenvs in C:\Users\<usename>\AppData\Local\hatch\env\virtual\scikit-learn-knn-regression (is this what you mean about caching installs?).

If we do leave the test matrix in pyproject.toml, what would be the hatch command to just run against one Python (e.g. Python 3.10)? I haven't dug deeply enough, but it looks like this replaces tox functionality rather than using tox functionality?

[aazuspan]

(is this what you mean about caching installs?)

Yeah, I hadn't looked into the mechanism, but that makes sense that it's just storing it as part of the environment.

it looks like this replaces tox functionality rather than using tox functionality?

Exactly!

what would be the hatch command to just run against one Python (e.g. Python 3.10)?

It looks like you can modify which versions in the matrix run with + and -, so the command hatch run +py=3.10 test:all would just trigger the 3.10 run, or hatch run -py=3.10 test:all would trigger everything but 3.10.

I'm kind of assuming that testing with one Python version will still be the default since testing with the whole matrix can get pretty slow, so maybe it would make sense to keep the test env as-is, and add a new env to handle the matrix testing? My first thought is to configure it so you could run a command like hatch run matrix:test or hatch run test_matrix:all to run the full suite. What do you think?

[grovduck]

I'm kind of assuming that testing with one Python version will still be the default since testing with the whole matrix can get pretty slow, so maybe it would make sense to keep the test env as-is, and add a new env to handle the matrix testing? My first thought is to configure it so you could run a command like hatch run matrix:test or hatch run test_matrix:all to run the full suite. What do you think?

Yeah, I think that's a great idea. It would be nice to do the singular Python version test with hatch run test:all, but have the availability of the full matrix in a different testing "namespace". As far as the two naming options, I'd be happy with either one.