Receiving "maximum number of iterations reached ..." message when executing SVM with classProbs=TRUE

[RobertFeyerharm]

Cross posted from my question on StackOverflow:

I'm running a support vector machine with a radial basis kernel function in the R caret package. My code runs without errors or warnings, however it generates a "maximum number of iterations reached ..." message which I interpret as meaning the algorithm didn't converge to a solution.

Using a small college admissions dataset (4 features, n=400) as an example:

# Load data & factor admit variable.
> mydata <- read.csv("http://www.ats.ucla.edu/stat/data/binary.csv")
  mydata$admit <- as.factor(mydata$admit)

# Create levels yes/no to make sure the the classprobs get a correct name.
 levels(mydata$admit) = c("yes", "no")


# Train SVM via 10-fold CV.
set.seed(123)
train_control <- trainControl( method="cv",
    number=10,
    classProbs = TRUE,
    savePredictions = TRUE)

model_rbfsvm<- train(as.factor(admit) ~ .,
    data=mydata,
    trControl=train_control,
    method="svmRadial", 
    tuneGrid=expand.grid(C=c(.000001, .00001, .0001, .001, .01, .1, 1, 10), sigma=c(.00001, .0001, .001, .01, .1, 1, 10)), 
    metric="Accuracy", 
    preProcess=c("center","scale"))

successfully executes but produces the following message (I've abbreviated - it goes on for many lines):
maximum number of iterations reached 4.663775e-05 4.663771e-05maximum number of iterations reached...

It was pointed out to me that when classProbs is set equal to FALSE, the above message is not generated. However the model_rbfsvm$finalModel output is significantly different with classProbs=FALSE vs. classProbs=TRUE:

# classProbs=FALSE 
> model_rbfsvm$finalModel
Support Vector Machine object of class "ksvm" 

SV type: C-svc  (classification) 
 parameter : cost C = 10 

Gaussian Radial Basis kernel function. 
 Hyperparameter : sigma =  0.1 

Number of Support Vectors : 258 

Objective Function Value : -2377.643 
Training error : 0.2775 

# classProbs=TRUE 
> model_rbfsvm$finalModel
Support Vector Machine object of class "ksvm" 

SV type: C-svc  (classification) 
 parameter : cost C = 1 

Gaussian Radial Basis kernel function. 
 Hyperparameter : sigma =  0.1 

Number of Support Vectors : 263 

Objective Function Value : -248.3275 
Training error : 0.2825 
Probability model included. 

[topepo]

That message does occur sometimes. It doesn't have anything to do with train but is related to kernlab. That said, I've always gone by the guideline that, if I look at the results and see if the number of iterations is "good enough", then I ignore it. Otherwise there isn't much that can be done.

There can be a significant impact on the model when class probabilities have been requested and it has been mentioned here and other places. SVMs do not normally produce class probabilities and a secondary model is used on the usual output of the SVM model that translates them to be "probability like". Sometimes this doesn't do a very good job and performance changes (i.e. drops). I'm not surprised that you get different answers with and without that option.

[RobertFeyerharm]

Thanks topepo, very helpful. Yes, I didn't realize that the class probabilities option for SVM requires an additional model.