[Breaking] Upgrade cuDF and RMM to 0.18 nightlies; require RMM 0.18+ for RMM plugin

[hcho3]

Also update the RMM plugin code to account for the updated signature of the template class rmm::mr::thrust_allocator<T> (rapidsai/rmm#647).

cc @harrism @jrhemstad

[pseudotensor]

Do things like this preserve backwards compatibility? i.e. anyone using xgboost with RMM has to use nightly (unstable) rapids 0.18? Or any rapids will work?

[hcho3]

@pseudotensor As it stands, no, this PR breaks backward compatibility.

[pseudotensor]

Seems not good idea to track nightly breaking changes. xgboost should be more stable, have deprecation stages, etc. Ya, seems easy to import rmm and check version. Would be nice if rmm had standard version . But one can do:

>>> from rmm._version import get_versions
>>> get_versions()['version']
'0.14.0'

But shouldn't do this globally at import level, like @trivialfis fixed for other imports of cudf etc. Instead can use pkg_info etc. or whatever method was used there in xgboost that followed tensorflow.

[hcho3]

Seems not good idea to track nightly breaking changes. xgboost should be more stable, have deprecation stages, etc

@pseudotensor
I agree with you in general. However, the RMM plugin is still marked as experimental and so it is exempt from the usual deprecation policy.

@harrism For now, we will make this particular breaking change. I will file a feature request for detecting the RMM version.

[pseudotensor]

Seems not good idea to track nightly breaking changes. xgboost should be more stable, have deprecation stages, etc

@pseudotensor
I agree with you in general. However, the RMM plugin is still marked as experimental and so it is exempt from the usual deprecation policy.

@harrism For now, we will make this particular breaking change. I will file a feature request for detecting the RMM version.

Even if experimental in xgboost it doesn't follow xgboost has to track unstable nightly versions of the experimental feature. That's your choice.

[hcho3]

@pseudotensor

Some considerations:

• RMM plugin is by default turned off from the usual XGBoost installation (pip install xgboost). Only the XGBoost Conda package from the rapidsai channel will use it.
• There's the issue of not having a way to obtain the RMM version from the C++ layer. Your suggestion works in Python but not in C++. I will request for a C++ macro that encodes the version string.

So in this particular occasion, I vote for the breaking change.

[hcho3]

@pseudotensor @harrism I created rapidsai/rmm#663 to request the version string in C++. The requested feature will help us (XGBoost devs) to handle breaking changes more gracefully in the future.

[trivialfis]

Looks good to me. I will go with the breaking change. Requiring nightly dependencies when using nightly xgboost seems fine to me.

[trivialfis]

Also please remove the CI tag when squashing commits as it's touching the core of xgboost.

[hcho3]

Got it. Let me also add a [breaking] prefix, so that later I can remind myself that RMM 0.18+ is required to enable the RMM plugin.