Use rapids-cmake parallel testing feature

[robertmaynard]

Description

Converts librmm over to use rapids-cmake new GPU aware parallel testing feature, which allows tests to run across all the GPUs on a machine without oversubscription.

This will allow developers to run ctest -j<N> and ctest will figure out given the current machine how many tests it can run in parallel given the current GPU set ( currently 4 tests per GPU ).

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[robertmaynard]

Things to be done before moving out of draft:

• rapids-cmake needs to merge Add rapids_test allowing projects to run gpu tests in parallel rapids-cmake#328

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (branch-23.02@c5c02fc). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files

@@              Coverage Diff               @@
##             branch-23.02   #1183   +/-   ##
==============================================
  Coverage                ?   0.00%           
==============================================
  Files                   ?       6           
  Lines                   ?     414           
  Branches                ?       0           
==============================================
  Hits                    ?       0           
  Misses                  ?     414           
  Partials                ?       0           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ajschmidt8]

Big fan of this PR! @bdice and I were planning on circling back to the GH Actions test scripts in the future to figure out how we could make some of these scripts less verbose.

[robertmaynard]

Moving to 23.04

[harrism]

Some RMM tests allocate significant memory. If tests are run in parallel on small-memory GPUs this might be a problem.

[robertmaynard]

Some RMM tests allocate significant memory. If tests are run in parallel on small-memory GPUs this might be a problem.

If we have a list of high memory usage tests we can mark them as requiring the entire GPU which will remove this failure spot ( It will still allow multiple tests to run on multi-gpu systems )

[bdice]

Tests are failing due to a lack of ctest. Maybe this needs cmake in the test dependencies of librmm? If ctest is the only way to run tests now, we should specify it in the meta.yaml. If there are still test binaries that are useful on their own and ctest is "optional," then perhaps specify it in the dependencies.yaml section for C++ tests?

[robertmaynard]

Tests are failing due to a lack of ctest. Maybe this needs cmake in the test dependencies of librmm? If ctest is the only way to run tests now, we should specify it in the meta.yaml. If there are still test binaries that are useful on their own and ctest is "optional," then perhaps specify it in the dependencies.yaml section for C++ tests?

I think it has to go in the dependencies.yaml since we are just changing how our CI invokes the tests. The tests themselves can still be run without ctest if one doesn't want parallel execution support.

[robertmaynard]

Looks to be reducing our c++ test time by 45sec ( 4min to 3.15min ). rmm has a couple of tests ( DEVICE_MR ) that have long execution times ( 40+sec ) and high memory usage that cause high serialization.

[vyasr]

Looks to be reducing our c++ test time by 45sec ( 4min to 3.15min ). rmm has a couple of tests ( DEVICE_MR ) that have long execution times ( 40+sec ) and high memory usage that cause high serialization.

I'm expecting that we'll get much more mileage out of this feature in cudf (and other downstream RAPIDS repos) where the tests are much more amenable to parallelization (lower util).

[vyasr]

Tests are failing due to a lack of ctest. Maybe this needs cmake in the test dependencies of librmm? If ctest is the only way to run tests now, we should specify it in the meta.yaml. If there are still test binaries that are useful on their own and ctest is "optional," then perhaps specify it in the dependencies.yaml section for C++ tests?

I think it has to go in the dependencies.yaml since we are just changing how our CI invokes the tests. The tests themselves can still be run without ctest if one doesn't want parallel execution support.

One minor caveat is that for cudf I plan to use ctest to configure tests with suitable environment variables to manage the preload library and other associated settings, so those tests will run differently when run via ctest than when the executables are invoked directly. Obviously that specific example is only relevant for cudf, but it does illustrate the possibility that we will have instances where ctest will run a test differently from a direct invocation. We may want to document that expectation if it translates to a recommended way to run tests (ctest over direct executable).

[vyasr]

Mostly LGTM.

[robertmaynard]

/merge