Merge pull request #3281 from sanuj/cookbook_svm

add kernel svm cookbook

doc/cookbook/source/examples/classifier/kernel_svm.rst

@@ -0,0 +1,57 @@
+=============================
+Kernel Support Vector Machine
+=============================
+
+Kernel Support Vector Machine is a binary classifier which finds a data-separating hyperplane in a Hilbert space induced by a positive definite kernel. The hyperplane is chosen to maximize the margins between the two classes. The loss function that needs to be minimized is:
+
+.. math::
+
+    \max_{\bf \alpha} \sum_{i=0}^{N-1} \alpha_i - \sum_{i=0}^{N-1}\sum_{j=0}^{N-1} \alpha_i y_i \alpha_j y_j  k({\bf x}_i, {\bf x}_j)
+
+subject to:
+
+.. math::
+
+    0 \leq \alpha_i \leq C, \sum_{i=0}^{N-1} \alpha_i y_i = 0
+
+where :math:`N` is the number of training samples, :math:`{\bf x}_i` are training samples, :math:`k` is a kernel, :math:`\alpha_i` are the weights, :math:`y_i` is the corresponding label where :math:`y_i \in \{-1,+1\}` and :math:`C` is a pre-specified regularization parameter.
+
+Implementation of Kernel SVM in Shogun uses LibSVM :cite:`chang2011libsvm`.
+
+See :cite:`scholkopf2002learning` and Chapter 6 in :cite:`cristianini2000introduction` for a detailed introduction.
+
+-------
+Example
+-------
+
+Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and :sgclass:`CBinaryLabels` as
+
+.. sgexample:: kernel_svm.sg:create_features
+
+In order to run :sgclass:`CLibSVM`, we need to initialize a kernel like :sgclass:`CGaussianKernel` with training features and some parameters like :math:`C` and epsilon i.e. residual convergence parameter which is optional.
+
+.. sgexample:: kernel_svm.sg:set_parameters
+
+We create an instance of the :sgclass:`CLibSVM` classifier by passing it regularization coefficient, kernel and labels.
+
+.. sgexample:: kernel_svm.sg:create_instance
+
+Then we train and apply it to test data, which here gives :sgclass:`CBinaryLabels`.
+
+.. sgexample:: kernel_svm.sg:train_and_apply
+
+We can extract :math:`\alpha` and :math:`b`.
+
+.. sgexample:: kernel_svm.sg:extract_weights_bias
+
+Finally, we can evaluate test performance via e.g. :sgclass:`CAccuracyMeasure`.
+
+.. sgexample:: kernel_svm.sg:evaluate_accuracy
+
+----------
+References
+----------
+:wiki:`Support_vector_machine`
+
+.. bibliography:: ../../references.bib
+    :filter: docname in docnames

examples/meta/src/classifier/kernel_svm.sg

@@ -0,0 +1,40 @@
+CSVFile f_feats_train("../../data/classifier_binary_2d_nonlinear_features_train.dat")
+CSVFile f_feats_test("../../data/classifier_binary_2d_nonlinear_features_test.dat")
+CSVFile f_labels_train("../../data/classifier_binary_2d_nonlinear_labels_train.dat")
+CSVFile f_labels_test("../../data/classifier_binary_2d_nonlinear_labels_test.dat")
+
+#![create_features]
+RealFeatures features_train(f_feats_train)
+RealFeatures features_test(f_feats_test)
+BinaryLabels labels_train(f_labels_train)
+BinaryLabels labels_test(f_labels_test)
+#![create_features]
+
+#![set_parameters]
+real C = 1
+real epsilon = 0.001
+GaussianKernel gauss_kernel(features_train, features_train, 15)
+#![set_parameters]
+
+#![create_instance]
+LibSVM svm(C, gauss_kernel, labels_train)
+svm.set_epsilon(epsilon)
+#![create_instance]
+
+#![train_and_apply]
+svm.train()
+BinaryLabels labels_predict = svm.apply_binary(features_test)
+#![train_and_apply]
+
+#![extract_weights_bias]
+RealVector alphas = svm.get_alphas()
+real b = svm.get_bias()
+#![extract_weights_bias]
+
+#![evaluate_accuracy]
+AccuracyMeasure eval()
+real accuracy = eval.evaluate(labels_predict, labels_test)
+#![evaluate_accuracy]
+
+# additional integration testing variables
+RealVector output = labels_predict.get_labels()