SVM with classProbs=TRUE leads to much worse accuracy

[jamona]

Why are the classification accuracies in certain cases much worse while using "svmRadial" together with classProbs=TRUE instead of classProbs=FALSE? Is there any way to use svmRadial with class-probabilities and not screwing up my accuracy?

[topepo]

A reproducible example would be a good place to start the discussion so that we can see how different the results are.

With classProbs = TRUE, kernlab estimates a secondary regression model based on the decision values and uses those to make predictions. That might decrease accuracy.

[jamona]

The results are very different. But since it's more a problem of kernlab, I think you can close this issue. For everyone who wants to use the e1071-svm with caret, here is my code (according to the official caret-tutorial for embedding other libraries):

E1071 to Caret

eSVM <- list(type = "Classification", library = "e1071", loop = NULL)
prm <- data.frame(parameter = c("C"),class = rep("numeric", 1),label = c("Cost"))
eSVM$parameters <- prm

svmGrid <- function(x, y, len = NULL, search = "grid") {
  ## This produces low, middle and high values for sigma 
  ## (i.e. a vector with 3 elements). 
  ## To use grid search:
  if(search == "grid") {
    out <- expand.grid( C = 2 ^((1:len) - 3))
  } else {
    ## For random search, define ranges for the parameters then
    ## generate random values for them
    out <- data.frame(C = 2^runif(len, min = -5, max = 8))
  }
  out
}
eSVM$grid <- svmGrid

svmFit <- function(x, y, wts, param, lev, last, weights, classProbs) {
  svm(x = as.matrix(x), y = y,
       kernel = "radial",
       cost = param$C,
       probability = classProbs)
}

eSVM$fit <- svmFit

svmPred <- function(modelFit, newdata, preProc = NULL, submodels = NULL)
  predict(modelFit, newdata)
eSVM$predict <- svmPred

svmProb <- function(modelFit, newdata, preProc = NULL, submodels = NULL)
  as.data.frame(attr(predict(modelFit, newdata, probability=TRUE),"probabilities"))
eSVM$prob <- svmProb
eSVM$levels <- function(x) lev(x)

[topepo]

But since it's more a problem of kernlab

That's not really the case; SVMs in general do not intrinsically estimate class probabilities so Platt's method (which is what I described) is pretty much used everywhere.

[jamona]

Then it would be interesting why (found several topics on StackOverflow, where the same problem was encountered) using classProbs=TRUE with your kernlab-SVM cuts down the accuracy that much. Using classProbs=TRUE lead to an accuracy-reduction from over 80% to 45%. While using e1071 this is not the case and the accuracy stays nearly constant.

[topepo]

That's more than I've seen. Again a data set I can use to look into the matter would be great.

[topepo]

I'm going to close this; we don't have any other information to help investigate the issue.

[MatthewBolton]

Quick note.

I'm experiencing this issue in 2019. A test using the iris dataset fails to reproduce the issue.

However, using my dataset (y = factor w/ 10 levels)(unfortunately, confidential data), prediction accuracy is dropping from ~97% to ~61% when trained in caret when the only difference is classProbs = TRUE and classProbs = FALSE.

Training of models directly in kernlab produces similar results when probabilistic predictions are made, indicating that as others have suggested, is an issue with how Platt's method works.

Using the same dataset with other classes of learners class probabilities seem to work as expected. Strangely, even though svmRadialSigma/svmRadial/svmCost all exhibit this weird behaviour, svmRadialWeights seems to perform well, when producing raw predictions, but using the same model to make probabilistic predictions, performance again drops.

[statistics88]

Hi I am experiencing this issue using caret package.
I have put a reproducible example here : https://stackoverflow.com/questions/63749263/different-results-for-svm-in-r