subsample and VIMP confidence intervals

[cmnelson1]

Hi all.

I am currently using randomForestSRC version 3.2.3. I am re-running a survival forest, extracting permutation VIMPS and using the subsample function to obtain confidence intervals. I am using this code to extract the CI:
set.seed(12345)
oo_1 <- subsample(RFmodel)
vimpCI <- extract.subsample(oo_1)$var.jk.sel.Z

I had previously run the same forest on the same data set (and same seed) and extracted the VIMP confidence intervals in the same way (using the same seed). However, I was using an older version of R and older version of the package (and, unfortunately do not know the version---my computer had to be re-imaged and all software reloaded). The confidence intervals I am getting now are much wider and the list of "significant" variables is much shorter than those on my previous runs with an earlier version of the package.

I have read through the release notes and do not see why this would be. The performance of the forest is almost identical and the VIMPs extracted using RFmodel$importance are identical.

Was there a change to the subsample function that might explain this difference? And, if so, what is that change? And is there a way to replicate what I extracted previously so I understand the difference?

Thank you!

Mindy

[ishwaran]

Yes, there were some changes that were made.

First make sure that in the training forest you have set the importance to permute, impotance="permute" otherwise by default anti-VIMP will be used in the subsample call. The latter is faster but not as accurate.

Second, I believe we have made a change to the sub ratio value which is the size of the data set that is subsampled. This was made in conjunction with using the subsampled variance estimator as the default estimator as opposed to the jack-knife estimator. You should check to see if $var.sel.Z returns values that look more appropriate.

[cmnelson1]

Thank you Dr. Ishwaran for your thoughts. I did have importance set to "permute" for the RF. I have checked the $var.sel.Z output for VIMPs and CI and they are actually the same as the jack-knife estimator output.

I assume that you reset the subratio parameter to a value that you thought was a reasonable default for prediction. I would not otherwise know what might be reasonable for our data. The current settings certainly creates a more sparse list of important variables than what the previous setting extracted. The subratio documentation says the default is set to the inverse square root of the sample size. Do you happen to know if this is the old default or the new default? I think I will play around with different values just to see if I can recreate our old list. Even so, it would be hard for me to judge which list would be the "most appropriate" list. The top 8 variables are exactly the same for the two different runs (old run/old settings and new run/new settings), so maybe I should just focus on those since they are the most robust.

Thank you again for your help.