Implement better default parameters for RandomForestClassifier

[btepera]

For sufficiently large datasets, cuML's implementation of RandomForestClassifier can produce significantly lower accuracy scores than scikit-learn when using the default parameters. The main culprits seem to be max_depth and max_features.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score
import cuml

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz"

data = pd.read_csv(url, header=None)

X, y = data.iloc[:, :-1], data.iloc[:, -1]
print(X.shape)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = RandomForestClassifier(n_estimators=100, n_jobs=-1)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print("sklearn RF accuracy: " + str(accuracy_score(y_test, y_pred)))

clf_gpu = cuml.ensemble.RandomForestClassifier(n_estimators=100)
clf_gpu.fit(X_train, y_train)
y_pred = clf_gpu.predict(X_test)
print("cuML RF default accuracy: " + str(accuracy_score(y_test, y_pred)))

clf_gpu_mod = cuml.ensemble.RandomForestClassifier(n_estimators=100, max_depth=30, max_features=1.0)
clf_gpu_mod.fit(X_train, y_train)
y_pred = clf_gpu_mod.predict(X_test)
print("cuML RF accuracy with greater max_depth and max_features: " + str(accuracy_score(y_test, y_pred)))

(581012, 54)
sklearn RF accuracy: 0.955259330654114
cuML RF default accuracy: 0.7143016961696341
cuML RF accuracy with greater max_depth and max_features: 0.9637616928994949

cuML's default max_depth is just 16, compared to sklearn which has unlimited depth. Would be great if cuML had better or "smarter" defaults, particularly when working with larger datasets.

[adam2392]

I wonder how much of a slowdown would we get if we allowed trees to be built to purity?

My current limited understanding of the RF implementation is that GPU is slowed down if we don't statically allocate memory (I.e. know depth beforehand), but is it possible to just continuously allocate new memory as the tree is getting built? E.g. build 16 depths at a time and then stack the tree and trim it at the end. Can we allocate memory asynchronously?

Maybe someone on the team with a much better understanding can comment on the engineering challenges there.

[hcho3]

Goal

• Understand the performance characteristics of cuML RF
• Choose a better default for hyperparameters for cuML RF

Tasks

• Identify representative data sets
• Run benchmark for fitting sklearn RF and cuml RF models. Measure both accuracy and training time.
  • Identify two or three hyperparameters and do a sweep. Example: max_depth
  • Hyperparameter matrix should include the current default values, to assess how optimal the current choice of defaults are.

[csadorf]

To clarify this issue: we should strictly focus on accuracy improvements within the scope of this issue. Potential performance regressions should be addressed separately.