Differences in permutation variable importance between ranger and randomForestSRC

[mickeycampbell]

Hello,

Looking through the VIMP vignette, I have come to understand (to the extent that I am comprehending the mathematical notation correctly...) that rfsrc()'s variable importance measure in the context of a regression problem where importance = "permute" should be the difference in OOB MSE that occurs when a predictor variable's values are randomly permuted in comparison to that when the original values are retained.

Assuming this is correct, then that appears to be the same measure used in the ranger package, as described here.

However, when I run the same dataset through ranger() and rfsrc() while attempting to mirror the two models' parameters (e.g., same number of trees, mtry, nodesize, etc.), I get substantially different variable importance values. The variable importance from rfsrc() values are, on average, about 50% lower than those of ranger(). The order of the importance is very similar, which is reassuring. And the model performance is nearly identical, in terms of OOB R^2^ and MSE. I would expect, due to the randomness of random forests, some inherent and minor differences, but the systematically lower values that emerge from rfsrc() (or, conversely, the systematically higher values from ranger()) have me a bit puzzled.

FWIW, I also explored variable importance from randomForest(), but understand that those values are normalized by the standard deviation of the MSE differences, making direct comparison to the outputs of ranger() and rfsrc() a bit more challenging.

I will attach the dataset I am using in case anyone wants to try to reproduce exactly what I am finding. bio is the response variable (biomass, in Mg/ha), and all other fields are airborne lidar structural predictor variables.

biomass.csv

Here is a quick code snippet to highlight my approach:

library(ranger)
library(randomForestSRC)

# read in the data
df <- read.csv("e:/mickey/temp/biomass.csv")

# build ranger model
mod.rng <- ranger(
  formula = bio ~ ., 
  data = df, 
  importance = "permutation",
  mtry = 10, 
  min.node.size = 1, 
  sample.fraction = 0.7
)

# build randomForestSRC model
mod.src <- rfsrc(
  formula = bio ~ ., 
  data = df, 
  importance = "permute",
  mtry = 10, 
  nodesize = 1, 
  sampsize = round(0.7 * nrow(df))
)

# get variable importance
vi.rng <- mod.rng$variable.importance |> as.data.frame()
vi.src <- mod.src$importance |> as.data.frame()
vi <- merge(vi.rng, vi.src, by = "row.names")
colnames(vi) <- c("variable", "vi.ranger", "vi.randomForestSRC")
vi <- vi[order(vi$vi.ranger, decreasing = T),]
row.names(vi) <- 1:nrow(vi)

# print first 10 rows
head(vi, 10)

Here is what is returned:

          variable vi.ranger vi.randomForestSRC
1           rumple  78.83899           27.97426
2  sum_tree_volume  61.25834           27.42259
3             zq75  59.89472           15.97686
4             ziqr  59.28462           14.12050
5               L1  53.76214           12.22002
6            zmean  50.31988           10.88286
7    sum_tree_area  46.38478           24.69914
8             zq70  40.98573           10.18309
9             zq80  37.78599           18.87974
10             lai  36.81310           16.63616

Thanks to anyone who can provide some insights!