Test performance is better than training performance for imbanaced rf

[Mosen111]

Dear Professor Ishwaran,
I have been working with an unbalanced classification model and have encountered an unexpected problem. Specifically, with different splitting schemes, I consistently observe that the test performance is significantly better than the training performance on all metrics. When i run a ranfom forest in the test data, the performance is very similar to the model in the train data, but the "predict" function returnes much better test performance results. Could you please share your opinion on this result and my code below? that would be much appreciated.

rf <- imbalanced(gender ~ ., data = traindata, ntree = 3000)
test_rf <- predict(rf, newdata = test_data, outcome = "test")

[ishwaran]

The option outcome=test is a very special form of prediction where the trained forest is being recalculated by replacing the terminal node estimator with new values from the test data.

You should instead use the canonical call:

rf <- imbalanced(gender ~ ., data = traindata, ntree = 3000)
test_rf <- predict(rf, newdata = test_data)

[Mosen111]

thanks a lot for your quick reply. After removing "outcome=test", the discrepency between two models got even larger. Here is the training performance:

" Sample size: 3096
Frequency of class labels: 1853, 1243
Number of trees: 3000
Forest terminal node size: 6
Average no. of terminal nodes: 228.8913
No. of variables tried at each split: 9
Total no. of variables: 39
Resampling used to grow trees: swor
Resample size used to grow trees: 1957
Analysis: RFQ
Family: class
Splitting rule: auc random
Number of random split points: 10
Imbalanced ratio: 1.4907
(OOB) Brier score: 0.19822228
(OOB) Normalized Brier score: 0.7928891
(OOB) AUC: 0.75122857
(OOB) PR-AUC: 0.66721375
(OOB) G-mean: 0.68971392
(OOB) Requested performance error: 0.31028608

Confusion matrix:

      predicted

observed Female Male class.error
Female 1283 570 0.3076
Male 389 854 0.3130

  (OOB) Misclassification rate: 0.3097545"

######## the test performance:

"Sample size of test (predict) data: 635
Number of grow trees: 3000
Average no. of grow terminal nodes: 228.8913
Total no. of grow variables: 39
Resampling used to grow trees: swor
Resample size used to grow trees: 1957
Analysis: RFQ
Family: class
Imbalanced ratio: 1.4708
Brier score: 0.09622492
Normalized Brier score: 0.38489968
AUC: 0.99429724
PR-AUC: 0.99192903
G-mean: 0.96423281
Requested performance error: 0.03576719

Confusion matrix:

      predicted

observed Female Male class.error
Female 357 21 0.0556
Male 4 253 0.0156

       Misclassification error: 0.03937008"

####### the performance of a rf trained on the test data:
"Sample size: 635
Frequency of class labels: 378, 257
Number of trees: 5000
Forest terminal node size: 6
Average no. of terminal nodes: 48.515
No. of variables tried at each split: 9
Total no. of variables: 39
Resampling used to grow trees: swor
Resample size used to grow trees: 401
Analysis: RFQ
Family: class
Splitting rule: auc random
Number of random split points: 10
Imbalanced ratio: 1.4708
(OOB) Brier score: 0.21031388
(OOB) Normalized Brier score: 0.84125551
(OOB) AUC: 0.71668417
(OOB) PR-AUC: 0.61673131
(OOB) G-mean: 0.67842147
(OOB) Requested performance error: 0.32157853

Confusion matrix:

      predicted

observed Female Male class.error
Female 243 135 0.3571
Male 73 184 0.2840

  (OOB) Misclassification rate: 0.3275591"

I would appreciate any comments. Thanks in advance.

[ishwaran]

From your output, the training performance as measured by the g-mean is

(OOB) G-mean: 0.68971392

The performance using the trained forest on the test data has g-mean performance

(OOB) G-mean: 0.67842147

Looks pretty close to me