Migrate as much as possible to pyproject.toml

[jakirkham]

Description

• Adds tomli dependency for Python pre-3.11
• Moves package data handling to MANIFEST.in
• Symlinks and packages README
• Specifies build backend & moves to newer setuptools backend
• Pulls all metadata into pyproject.toml
• Migrates setup.cfg content to pyproject.toml
• Simplifies setup.py to the minimal amount of logic necessary

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[jakirkham]

cc @vyasr

[jakirkham]

We may need to update some things on CI (like isort)

Edit: Looks like we need tomli in more places

[vyasr]

This looks great! I'm in favor of almost all of these changes. However, I would like to test this out with the wheel building before we merge anything just to be safe. Burn down is coming up fast and build infrastructure is fragile enough by nature that I'm wary of making changes that could hold up that effort. We're pretty close to stabilizing our logic for wheels so I think we should be able to test this with wheel builds within the next week or so.

[vyasr]

Happy to iterate on this PR until then though of course.

[jakirkham]

Yeah the changes here should be useful for wheel building. Trying out the changes here would be welcome. Though likely still needs a couple fixes.

[bdice]

This is awesome @jakirkham. It fixes a lot of minor problems in the rmm packaging that have been troublesome on occasion. I have a couple small requests. I also fixed the merge conflicts in the dependencies.yaml file.

Resolved

[bdice]

Thanks @jakirkham! Just a few suggestions with packaging, then this will be good to go.

[jakirkham]

Thanks Vyas! 🙏

Would it work with a symlink?

[jakirkham]

Included a couple notes about other things that can move to pyproject.toml potentially

[vyasr]

Would it work with a symlink?

My guess is that a symlink would work for most RAPIDS packages where the C++ is contained in a separate directory in the tree. In this instance I think you would end up with a circular symlink because the C++ is contained at the top-level directory (perhaps something that we should consider changing in rmm anyway). In particular, you need the CMakeLists.txt file contained at the root of the repo.

[jakirkham]

Thanks Vyas! 🙏

Had a couple small follow up points around packaging of text files

[jakirkham]

Would it work with a symlink?

My guess is that a symlink would work for most RAPIDS packages where the C++ is contained in a separate directory in the tree. In this instance I think you would end up with a circular symlink because the C++ is contained at the top-level directory (perhaps something that we should consider changing in rmm anyway). In particular, you need the CMakeLists.txt file contained at the root of the repo.

Ok think we should punt on this. Was mostly curious if there was an easy answer, but it seems this needs more thought. So yeah we can raise an issue and skip

[jakirkham]

Perhaps it is weird to review my PR. In any event, Vyas these changes look good to me. Thank you for working on them! 🙏

[bdice]

Two non-blocking suggestions. Otherwise LGTM!

[vyasr]

/merge

[jakirkham]

Thanks all! 🙏