class imbalance, sparse data, and random survival forest

[seaspray09]

We have been performing survival analyses using random survival forests (RSF) where we have apparent class imbalance (for example, a 5:95 event to nonevent ratio) and potential sparse data. Our interests are in finding and evaluating potential risk predictors for clinical outcomes described in such datasets.

My questions are as as follows:

1. What are your thoughts and approach on how class imbalance should be considered when performing RSF calculations. If we have low outcome event rates (such as a 5% event rate), would this be an issue in performing RSF calculations? What might be some recommended adjustments to the RSF calculations/code hyperparameters that one may need to take into account when evaluating data with class imbalance so that one can have some assurance of the predictors that are found along with the variable rank order? Does the imbalanced() function of randomForestSRC apply only to random forest (RF) or also to RSF?

2. For performing RSF on sparse data, would RSF be suitable for finding and evaluating risk predictors in such a dataset? The dataset is a mix of dense and sparse data. What might be some adjustments that you would recommend, if necessary, to allow for proper use of RSF to discover and evaluate risk predictors from such a dataset?

[ishwaran]

Technically, this is a much different scenario than the class imbalanced data problem, which applies to classification with imbalanced frequency, whereas your scenario is survival analysis in the presence of a high censoring rate. High censoring is typical in medical settings and does not require any change of the methodology for random survival forests. Therefore you do not need to make any modifications for risk assessments and so forth. However, I would recommend that you use the Brier score for performance metric wherever possible since the C-index can be biased in high censoring cases. The function get.brier.survival could be useful for that. See also the vignette on random survival forests: https://luminwin.github.io/randomForestSRC/articles/survival.html

[seaspray09]

Thank you for providing this invaluable insight Dr. Ishwaran. As a follow-up, I was wondering how the mechanism within random survival forests accounts for the presence of a high censoring rate. Is it due to the use of the log-rank test statistic to select the cutpoints for predictors at the branch points, or something else? Also, how permissible would calculations based on the log-rank statistic be under situations where proportional hazard assumptions be violated?

[ishwaran]

It's not necessarily the RSF mechanism that is at play, but rather that the survival framework itself accounts for both censored data and uncensored data and this what makes it possible to deal with high censoring rates. Estimators used in survival analysis, Nelson-Aalen or KM, are specifically designed to handle censoring.

The problem with class imbalanced data analysis is that the Bayes rule is used for classification. The underlying estimators used to estimate the conditional distribution are just fine (just like in survival analysis), but the problem is classification is based on the Bayes rule which will have extremely poor performance in high imbalanced data.

It is permissible to use log-rank when proportional hazards are violated. The PBC data set is a great example which shows how well log-rank can work in such settings. I think you can construct examples where you can do better under modifications, using different splitting rules, etc., but as a general rule it should work just fine.

[seaspray09]

Thank you for providing these additional clarifications, Dr. Ishwaran.

Since survival frameworks account for highly censored data, is there therefore not a need to be concerned about high class imbalance when looking for predictors of a disease outcome using RSF?

Specifically, for a scenario that is highly censored (for example 100 disease events among 2000 participants over a certain time period):

1. What would be your recommendations for performing feature selection from input data containing approximately 30 to 40 variables followed by assessing importance among predictors?

2. What may be appropriate ways to interpret "important predictors" in this situation?

To find important predictors towards our intended disease event outcome, we have tried to run RSF using randomForestSRC as follows:

rfsrc(Surv(time, event)~., data, ntree=1000, family="surv", err.block=1, tree.err=TRUE, importance=TRUE).

The output is yielding performance error rates of approximately 0.40, so this is also of concern.

[ishwaran]

It's just like I said, there's nothing really special about the high censoring rate case excepting that the C-index may be biased. Since variable importance uses C-index, this could be a problem, however variable importance is a difference, and it's likely this bias is cancelled out in the differencing operation. I use high censored data all the time (the usual scenario in medical applications) and have not noticed anything unusual. Currently we do not provide variable importance measures using the Brier score, for example, which could potentially remove such bias if it existed.

The high performance error rate suggests you are not able to do achieve high performance. That's just how it is sometimes for a machine learning problem. Did you try tuning the forest parameters, like nodesize and mtry? That might help.

[suengeek]

Dear Prof. Ishwaran,
I am doing a survival random forest for predicting cvd events, however, when I run the codes for calculating brier score, here comes an error:
###obtain Brier score using KM and RSF censoring distribution estimators
bs.km <- get.brier.survival(obj1, cens.mode = "km")$brier.score
bs.rsf <- get.brier.survival(obj1, cens.model = "rfsrc")$brier.score

###plot the brier score
plot(bs.km, type = "s", col = 2)
lines(bs.rsf, type ="s", col = 4)
legend("bottomright", legend = c("cens.model = km", "cens.model = rfsrc"), fill = c(2,4))

and here is the warning for 'bs.rsf':
Error in generic.predict.rfsrc(object, newdata, m.target = m.target, importance = importance, :
x-variables in test data do not match original training data.
I double checked the data frame and codes, but I didn't find the reason, could you help me?

[ishwaran]

That warning is issued when the test data features do not match the training features, typically this occurs when you have factors in the test data that take different levels than observed in the training data. Without a reproducible example, however, it is not possible for me to diagnose this further. Can you try to create a reproducible example?

[suengeek]

Dear, This is not a problem in the testing process, I just followed your illustration. It happens in the training process.

obj1 <- rfsrc(Surv(Mace.survival,Mace.indicator)~., data = dta,
ntree = 1000, nodesize = 57,set.seed(-123456),importance="permute")
get.cindex(obj1$yvar[,1], obj1$yvar[,2], obj1$predicted.oob)

bs.km <- get.brier.survival(obj1, cens.mode = "km")$brier.score
bs.rsf <- get.brier.survival(obj1, cens.model = "rfsrc")$brier.score
Error in generic.predict.rfsrc(object, newdata, m.target = m.target, importance = importance, :
x-variables in test data do not match original training data

[YinanHuang11]

Currently, we are evaluating the comparative time to cardiovascular (CV) adverse events in patients after taking drug A vs. drug B vs. drug C, based on about 60+ baseline predictors. We will be using your randomForestSRC.

Shall we perform any feature reduction? Of note, these CV outcomes are rare events, of more than 3000 patients, only about less than 200 developed CV events, including 5 for drug A, 62 for drug B and 62 for drug C?

I read from your prior post that Brier score is recommended in the scenario of imbalanced data. Any other evaluation metrics should also be considered, e.g., AUC or calibration plots?

[ishwaran]

The Brier score seems like a good idea