add cookbook page for svr

doc/cookbook/source/examples/regression/support_vector_regression.rst

@@ -0,0 +1,48 @@
+=========================
+Support Vector Regression
+=========================
+
+Support vector regression is a regression model inspired from support vector machines. The solution can be written as:
+
+.. math::
+    f({\bf x})=\sum_{i=1}^{N} \alpha_i k({\bf x}, {\bf x}_i)+b
+
+where :math:`{\bf x}` is the new data point, :math:`{\bf x}_i` is a training sample, :math:`N` denotes number of training samples, :math:`k` is a kernel function, :math:`\alpha` and :math:`b` are determined in training.
+
+See :cite:`scholkopf2002learning` for a more detailed introduction. :sgclass:`LibSVR` performs support vector regression using LibSVM :cite:`chang2011libsvm`.
+
+-------
+Example
+-------
+
+Imagine we have files with training and test data. We create `CDenseFeatures` (here 64 bit floats aka RealFeatures) and :sgclass:`CRegressionLabels` as
+
+.. sgexample:: support_vector_regression.sg:create_features
+
+Choose an appropriate :sgclass:`CKernel` and instantiate it. Here we use a :sgclass:`CGaussianKernel`.
+
+.. sgexample:: support_vector_regression.sg:create_appropriate_kernel
+
+We create an instance of :sgclass:`CLibSVR` classifier by passing it the kernel, labels, solver type and some more parameters. More solver types are available in :sgclass:`CLibSVR`. See :cite:`chang2002training` for more details.
+
+.. sgexample:: support_vector_regression.sg:create_instance
+
+Then we train the regression model and apply it to test data to get the predicted :sgclass:`CRegressionLabels`.
+
+.. sgexample:: support_vector_regression.sg:train_and_apply
+
+After training, we can extract :math:`\alpha`.
+
+.. sgexample:: support_vector_regression.sg:extract_alpha
+
+Finally, we can evaluate the :sgclass:`CMeanSquaredError`.
+
+.. sgexample:: support_vector_regression.sg:evaluate_error
+
+----------
+References
+----------
+:wiki:`Support_vector_machine`
+
+.. bibliography:: ../../references.bib
+    :filter: docname in docnames

doc/cookbook/source/references.bib

@@ -33,4 +33,30 @@ @article{ueda2000smem
   pages={2109--2128},
   year={2000},
   publisher={MIT Press}
+}
+@article{chang2011libsvm,
+  title={LIBSVM: a library for support vector machines},
+  author={C.C. Chang and C.J. Lin},
+  journal={ACM Transactions on Intelligent Systems and Technology (TIST)},
+  volume={2},
+  number={3},
+  pages={27},
+  year={2011},
+  publisher={ACM}
+}
+@article{chang2002training,
+  title={Training v-support vector regression: theory and algorithms},
+  author={C.C. Chang and C.B. Lin},
+  journal={Neural Computation},
+  volume={14},
+  number={8},
+  pages={1959--1977},
+  year={2002},
+  publisher={MIT Press}
+}
+@book{scholkopf2002learning,
+  title={Learning with kernels: support vector machines, regularization, optimization, and beyond},
+  author={B. Sch{\"o}lkopf and A.J. Smola},
+  year={2002},
+  publisher={MIT press}
 }

examples/meta/src/regression/support_vector_regression.sg

@@ -0,0 +1,39 @@
+CSVFile f_feats_train("../../data/regression_1d_sinc_features_train.dat")
+CSVFile f_feats_test("../../data/regression_1d_sinc_features_test.dat")
+CSVFile f_labels_train("../../data/regression_1d_sinc_labels_train.dat")
+CSVFile f_labels_test("../../data/regression_1d_sinc_labels_test.dat")
+
+#![create_features]
+RealFeatures features_train(f_feats_train)
+RealFeatures features_test(f_feats_test)
+RegressionLabels labels_train(f_labels_train)
+RegressionLabels labels_test(f_labels_test)
+#![create_features]
+
+#![create_appropriate_kernel]
+real width = 1
+GaussianKernel kernel(width)
+#![create_appropriate_kernel]
+
+#![create_instance]
+real svm_c = 1
+real svr_param = 0.1
+LibSVR svr(svm_c, svr_param, kernel, labels_train, enum LIBSVR_SOLVER_TYPE.LIBSVR_EPSILON_SVR)
+#![create_instance]
+
+#![train_and_apply]
+svr.train(features_train)
+RegressionLabels labels_predict = svr.apply_regression(features_test)
+#![train_and_apply]
+
+#![extract_alpha]
+RealVector alpha = svr.get_alphas()
+#![extract_alpha]
+
+#![evaluate_error]
+MeanSquaredError eval()
+real mse = eval.evaluate(labels_predict, labels_test)
+#![evaluate_error]
+
+# integration testing variables
+RealVector output = labels_test.get_labels()