manage dependencies with rapids-dependency-file-generator

[jameslamb]

Contributes to #115

When #115 is complete, this project will need access to some of the same dependency information in several different places:

• setup.py / pyproject.toml (for wheel-building)
• conda recipe
• CI configurations

This proposes using rapids-dependency-file-generator to keep those sets of dependencies consistent across those locations, and to automate some of the work of updating them. We've found this to be really helpful across other RAPIDS projects.

For an example of this, see rapidsai/cugraph#4521. All the changes to pyproject.toml and conda env YAML files there were driven by just updating dependencies.yaml and running rapids-dependency-file-generator --clean.

Other changes proposed here:

• adding a !=3.30.0 constraint on CMake
  • to avoid skip cmake 3.30.0 across RAPIDS build-planning#80, which I did run into while building locally with CMake 3.30.0
• only running the other build/test steps after pre-commit passes
  • to ensure jobs here don't take time on a GPU runner when they're doomed to fail because of linting errors

Notes for Revieweris

In a future PR, I'll propose switching from a setup.py file to a pyproject.toml, and automatically updating that pyproject.toml via this rapids-dependency-file-generator mechanism. That way, constraints like legate-core=24.06.* won't need to be duplicated in multiple places.

[RAMitchell]

LGTM

[jameslamb]

Oh hey thanks! I hadn't added you as a reviewer yet or taken this out of draft yet because it's not actually working. If I end up having to change anything substantial, I'll ask for another review.

[jameslamb]

The only change I had to make to get this working after you'd reviewed was switching from the pattern mamba env create && source activate to just mamba env update.

I think that should be non-controversial and low-risk, so I'm gonna merge this.