Confusion matrix normalization

[asardaes]

I have been using the caret package for my thesis, which is why I'm messing with the code and reporting the issues I find, I hope that's ok. This time I was looking at the confusionMatrix function for train objects. I did a few tests before I changed anything.

Let's take a simple example, there are 150 observations in the iris dataset, so using 5-fold CV would result in training sets with 30 observations each

data(iris)
TrainData <- iris[,1:4]
TrainClasses <- iris[,5]

knnFit1 <- train(TrainData, TrainClasses,
                 method = "knn",
                 preProcess = c("center", "scale"),
                 tuneLength = 10,
                 trControl = trainControl(method = "cv", number = 5))

cell_cols <- grep("^cell", colnames(knnFit1$resampledCM))
data_per_fold <- apply(knnFit1$resampledCM[ , cell_cols], 1, sum)
all(data_per_fold == 30)

## [1] TRUE

Now let's calculate the available confusion matrices

CM.none <- confusionMatrix(knnFit1, norm = "none")
CM.avg <- confusionMatrix(knnFit1, norm = "average")
CM.ovr <- confusionMatrix(knnFit1, norm = "overall")

I understand none and overall, although I don't agree entirely with the description in the documentation. To me, none actually shows the average counts for each cell across resamples (what supposedly average does):

CM.none$table

##             Reference
## Prediction   setosa versicolor virginica
##   setosa       10.0        0.0       0.0
##   versicolor    0.0        9.6       0.8
##   virginica     0.0        0.4       9.2

sum(CM.none$table)

## [1] 30

And the overall one is just the same table but in percentages:

CM.ovr$table

##             Reference
## Prediction      setosa versicolor virginica
##   setosa     33.333333   0.000000  0.000000
##   versicolor  0.000000  32.000000  2.666667
##   virginica   0.000000   1.333333 30.666667

all.equal(CM.none$table / sum(CM.none$table) * 100,
          CM.ovr$table)

## [1] TRUE

But I simply don't understand what average does.

CM.avg$table

##             Reference
## Prediction   setosa versicolor virginica
##   setosa        200          0         0
##   versicolor      0        192        16
##   virginica       0          8       184

This is why I'm proposing these changes...

---

With the changes

• none calculates aggregated un-normalized counts
• average calculates average counts across resamples
• overall calculates percentual average counts (stays the same)

confusionMatrix(knnFit1, norm = "none")

## Cross-Validated (5 fold) Confusion Matrix 
## 
## (entries are un-normalized aggregated counts)
##  
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         50          0         0
##   versicolor      0         47         3
##   virginica       0          3        47

confusionMatrix(knnFit1, norm = "average")

## Cross-Validated (5 fold) Confusion Matrix 
## 
## (entries are average cell counts across resamples)
##  
##             Reference
## Prediction   setosa versicolor virginica
##   setosa       10.0        0.0       0.0
##   versicolor    0.0        9.4       0.6
##   virginica     0.0        0.6       9.4

confusionMatrix(knnFit1, norm = "overall")

## Cross-Validated (5 fold) Confusion Matrix 
## 
## (entries are percentual average cell counts across resamples)
##  
##             Reference
## Prediction   setosa versicolor virginica
##   setosa       33.3        0.0       0.0
##   versicolor    0.0       31.3       2.0
##   virginica     0.0        2.0      31.3

Let me know your thoughts.

I don't know if this makes sense for rfe and sbf, but if it does, I could adapt the code so that those functions include the changes.

[topepo]

That sounds fine to me and it would be good to make those changes more universal (i.e. for rfe and sbf). Can you update the man file(s) too?

[asardaes]

I see that rfe and sbf have a lot in common in their confusionMatrix functions, so I put them together (like it was for the print generic) but added some logic. I'm not familiar with those two methods so can you check if it works as expected?

[Sandy4321]

good, this is very important , everybody use it

On Tue, Jan 19, 2016 at 4:31 PM, Alexis Sarda notifications@github.com
wrote:

I see that rfe and sbf have a lot in common in their confusionMatrix
functions, so I put them together (like it was for the print generic) but
added some logic. I'm not familiar with those two methods so can you check
if it works as expected?

—
Reply to this email directly or view it on GitHub
#355 (comment).

[asardaes]

I took the liberty of adding the boot632 reference, mainly because I initially thought the 0.632+ was being used, so I had to look at the code to make sure which one was actually computed.

[topepo]

Thanks!