better confusion matrix

[ja-thomas]

Imorovements for get confusion matrix (and fixed a bug in there).

Closes #1048

[berndbischl]

travis fails, please rebase on master

[ja-thomas]

Some more explaination.

getConfusionMatrix now returns an object that contains a lot more information. The "old" matrix is now in a slot cm$result, I also adapted all the internal usage so that the new object works and doesn't break anything.

We fixed a bug where the column marginals where strangely (or even wrongly) calculated for realtve values.

Also there is the possibility to add row and column sums which was requested in #1048

I added a printer to see the information in a compact way:

> set.seed(1230)
> allinds = 1:150
> train = sample(allinds, 75)
> test = setdiff(allinds, train)
> mod = train("classif.rpart", iris.task, subset = train)
> pred = predict(mod, iris.task, subset = test)
> print(getConfMatrix(pred, relative = TRUE))
Relative confusion matrix (normalized by row/column):
            predicted
true         setosa    versicolor virginica -err.-   
  setosa     1.00/1.00 0.00/0.00  0.00/0.00 0.00     
  versicolor 0.00/0.00 0.83/0.95  0.17/0.18 0.17     
  virginica  0.00/0.00 0.05/0.05  0.95/0.82 0.05     
  -err.-          0.00      0.05       0.18 0.22/0.23


Absolute confusion matrix:
            predicted
true         setosa versicolor virginica -err.-
  setosa         32          0         0      0
  versicolor      0         20         4      4
  virginica       0          1        18      1
  -err.-          0          1         4      5

> print(getConfMatrix(pred, relative = TRUE, sums = TRUE))
Relative confusion matrix (normalized by row/column):
            predicted
true         setosa    versicolor virginica -err.-   
  setosa     1.00/1.00 0.00/0.00  0.00/0.00 0.00     
  versicolor 0.00/0.00 0.83/0.95  0.17/0.18 0.17     
  virginica  0.00/0.00 0.05/0.05  0.95/0.82 0.05     
  -err.-          0.00      0.05       0.18 0.22/0.23


Absolute confusion matrix:
           setosa versicolor virginica -err.- -n-
setosa         32          0         0      0  32
versicolor      0         20         4      4  24
virginica       0          1        18      1  19
-err.-          0          1         4      5  NA
-n-            32         21        22     NA 150

[berndbischl]

1. the last element in the bottom right corner = errs / n.

2. confMatrix --> ConfMatrix. And the structure must be documented

3. really explain that we create a list of mats, that are printed then. that the user knows this.

4. check your asserts

5. i would add "sums" / "-n-" to all matrices

6. digits default should be 2

[ja-thomas]

> allinds = 1:150
> train = sample(allinds, 75)
> test = setdiff(allinds, train)
> mod = train("classif.lda", iris.task, subset = train)
> pred = predict(mod, iris.task, subset = test)
> print(getConfMatrix(pred))
            predicted
true         setosa versicolor virginica -err.-
  setosa         24          0         0      0
  versicolor      0         26         0      0
  virginica       0          1        24      1
  -err.-          0          1         0      1
> print(getConfMatrix(pred, sums = TRUE))
           setosa versicolor virginica -err.- -n-
setosa         24          0         0      0  24
versicolor      0         26         0      0  26
virginica       0          1        24      1  25
-err.-          0          1         0      1  NA
-n-            24         27        24     NA 150
> print(getConfMatrix(pred, relative = TRUE))
Relative confusion matrix (normalized by row/column):
            predicted
true         setosa    versicolor virginica -err.-   
  setosa     1.00/1.00 0.00/0.00  0.00/0.00 0.00     
  versicolor 0.00/0.00 1.00/0.96  0.00/0.00 0.00     
  virginica  0.00/0.00 0.04/0.04  0.96/1.00 0.04     
  -err.-          0.00      0.04       0.00   0.01   


Absolute confusion matrix:
            predicted
true         setosa versicolor virginica -err.-
  setosa         24          0         0      0
  versicolor      0         26         0      0
  virginica       0          1        24      1
  -err.-          0          1         0      1
> print(getConfMatrix(pred, relative = TRUE, sums = TRUE))
Relative confusion matrix (normalized by row/column):
            predicted
true         setosa    versicolor virginica -err.-    -n- 
  setosa     1.00/1.00 0.00/0.00  0.00/0.00 0.00      24  
  versicolor 0.00/0.00 1.00/0.96  0.00/0.00 0.00      27  
  virginica  0.00/0.00 0.04/0.04  0.96/1.00 0.04      24  
  -err.-          0.00      0.04       0.00   0.01    <NA>
  -n-        24        26         25        <NA>      150 


Absolute confusion matrix:
           setosa versicolor virginica -err.- -n-
setosa         24          0         0      0  24
versicolor      0         26         0      0  26
virginica       0          1        24      1  25
-err.-          0          1         0      1  NA
-n-            24         27        24     NA 150

[larskotthoff]

What exactly does the relative confusion matrix show?

[ja-thomas]

for a/b, a is the relative frequency (and error in the margins) normalized by column and b by row. the n are the group sizes.

[ja-thomas]

1. Error: measures quickcheck (@test_base_measures.R#573)

I dont think that has something to do with my changes.

[berndbischl]

1. Error: measures quickcheck (@test_base_measures.R#573)

pls rebase

[ja-thomas]

So like suggested in #1113, we renamed getConfMatrix to calculateConfusionMatrix and deprecated the old one.

Can you check if the deprecation is done correctly, i've never done that before.

Also two mini fixes, padding and the internal function name

[jakobbossek]

Can you check if the deprecation is done correctly, i've never done that before.

Deprecation looks good.

[berndbischl]

1. pls recheck the ConfMatrix docs.

2. add family performance

[ja-thomas]

The tag is already there for family tag for calculateConfusionMatrix. But should it also be added for the ConfMatrix?