sklearn and shogun's multiclass SVC produce different result

[cyberyu]

I had this problem similar to someone else reported on Github

https://gist.github.com/olinguyen/7fd3bf7642ae952a7803579cb9561b5b

Also, I added the Multi-class SVM from shogun and compared three classifiers. I didn't really benchmark C and epsilon, but I assumed that C value should not be that different across packages.

The reason led me to investigate was: I had some other real work using MKL (Multiple kernel learning) in Shogun. I learned the optimal weights of combined kernel, using that optimal kernel, the performance in Shogun is always lower than the same kernel applied on Sklearn SVC classifier.

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
 
# import some data to play with
iris = datasets.load_iris()
 
num_samples = len(iris.data)
 
np.random.seed(seed=42)
 
idx = np.arange(num_samples)
np.random.shuffle(idx)
 
spl = int(num_samples * 0.70)
 
X_train = iris.data[idx][:spl, :2] 
y_train = iris.target[idx][:spl]
X_test = iris.data[idx][spl:, :2]
y_test = iris.target[idx][spl:]
 
C = 1.0  # SVM regularization parameter
epsilon = 0.1

lsvc = svm.LinearSVC(C=C).fit(X_train, y_train)
sklearn_pred = lsvc.predict(X_test)
print("sklearn predictions:", sklearn_pred)
 
# Shogun  liblinear
 
classifier = LibLinear()
#strategy = MulticlassOneVsOneStrategy()
strategy = MulticlassOneVsRestStrategy()
 
mc_classifier = LinearMulticlassMachine(strategy, RealFeatures(X_train.T), classifier, MulticlassLabels(y_train.astype('d')))
mc_classifier.train()
 
y_pred = mc_classifier.apply_multiclass(RealFeatures(X_test.T))
print("Shogun LibLinear predictions:", y_pred.get_labels())

# Shogun SVM
linkernel = LinearKernel(RealFeatures(X_train.T),RealFeatures(X_train.T))  
svm = MulticlassLibSVM(C, linkernel, MulticlassLabels(y_train.astype('d')))
svm.set_epsilon(epsilon)
svm.train()
labels_predict = svm.apply_multiclass(RealFeatures(X_test.T)) 

print("Shogun Multiclass SVM predictions:", labels_predict.get_labels())


print("True Labels:", y_test)

[vigsterkr]

mmm the random (seeds etc) is for sure different so that for sure will end in slightly different results.

afaik in case of sklearn the default is L2 reg with l1-loss right? and it's solving the dual? just because that's the default for LibLinear in case of shogun... i.e. make sure that they are the same solvers.

but again PRNG should give you different results - as expected.

[cyberyu]

Thanks for the reply. The real problem is Shogun performance on this is very poor: it always give me naive predictions. I doubt my code may not construct the kernels correctly, but since I am using Linear kernel and the data set is very popular, I wonder where did I do wrong. I searched around and didn't find enough materials on how to correctly initialize this in Python. (I am an experienced SVM person and I understand how SVM works, but it seems really tricky to get satisfying results in python using a simple data set) My understanding is Liblinear from Shogun (L2 loss) is a different formulation than Sklearn's SVC (hinge loss), but more closer to Sklearn's LinearSVC which default loss function is also L2. And Shogun's Multiclass SVM, and Shogun's MKL may use hinge loss by default. Solving the dual or primal should not expect very different results if the problem is convex, anyway, I dont expect results to be exactly the same, but why my Shogun code gives such a poor performance? Did I miss any kernel normalization or centering process in my code? Below is the output generate from my code above (Shogun SVM cannot learn the class 1):

sklearn predictions: [0 2 2 0 2 2 2 2 2 0 2 2 2 1 1 2 1 2 1 0 1 2 2 0 1 2 2 0 2 0 2 2 2 1 2 1 2
1 2 0 2 1 0 1 2]
Shogun LibLinear predictions: [0. 2. 2. 0. 2. 2. 2. 2. 2. 0. 2. 2. 2. 2. 2. 2. 2. 2. 2. 0. 2. 2. 2. 0.
2. 2. 2. 0. 2. 0. 2. 2. 2. 2. 2. 2. 2. 2. 2. 0. 2. 2. 0. 2. 2.]
Shogun Multiclass SVM predictions: [2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.]
True Labels: [0 2 2 0 1 1 2 1 2 0 2 1 2 1 1 1 0 1 1 0 1 2 2 0 1 2 2 0 2 0 1 2 2 1 2 1 1
2 2 0 1 2 0 1 2]

[vigsterkr]

@cyberyu thnx heaps for your detailed issue description! i'll try to look at it and get back to you ASAP

[gf712]

Hey @cyberyu
I don't think it's a bug, you are just using the wrong getter. y_pred.get_multiclass_confidences gets you the confidence values for each example, i.e. y_pred.get_multiclass_confidences(0) for the first row in the prediction set. @vigsterkr I am not sure if get_values should return a matrix with these values? If so it can be fixed quite easily.

[karlnapf]

@gf712 it would be good to update the examples to extract the right values in a sensible way. Mind doing that?

[gf712]

@karlnapf I just realised that the issue is actually something else, I was just looking at the way the confidence values were extracted and everything was always zero..
But I'll have a look at the confidence values extraction anyway.

[abnerzyx]

@cyberyu
`
import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets

iris = datasets.load_iris()

num_samples = len(iris.data)

np.random.seed(seed=42)

idx = np.arange(num_samples)
np.random.shuffle(idx)

spl = int(num_samples * 0.70)

X_train = iris.data[idx][:spl, :2]
y_train = iris.target[idx][:spl]
X_test = iris.data[idx][spl:, :2]
y_test = iris.target[idx][spl:]

C = 1.0
epsilon = 0.1

lsvc = svm.LinearSVC(C=C).fit(X_train, y_train)
sklearn_pred = lsvc.predict(X_test)
print("sklearn predictions:", sklearn_pred)

classifier = LibLinear(L2R_L2LOSS_SVC_DUAL)
classifier.set_bias_enabled(True)
classifier.set_C(C, C)
#strategy = MulticlassOneVsOneStrategy()
strategy = MulticlassOneVsRestStrategy()

mc_classifier = LinearMulticlassMachine(strategy, RealFeatures(X_train.T), classifier, MulticlassLabels(y_train.astype('d')))
mc_classifier.train()

y_pred = mc_classifier.apply_multiclass(RealFeatures(X_test.T))
print("Shogun LibLinear predictions:", y_pred.get_labels())

np.array_equal(y_pred.get_labels(),sklearn_pred)
`

[karlnapf]

So was there a problem in the end that needs fixing?

[abnerzyx]

@karlnapf No need to fix, this is a parameter setting problem.The result of my code is consistent.