Merge pull request #3299 from sanuj/mkl_regression

add cookbook for mkl regression

doc/cookbook/source/examples/regression/multiple_kernel_learning.rst

@@ -0,0 +1,56 @@
+========================
+Multiple Kernel Learning
+========================
+
+Multiple kernel learning (MKL) is based on convex combinations of arbitrary kernels over potentially different domains.
+
+.. math::
+
+    {\bf k}(x_i,x_j)=\sum_{i=1}^{K} \beta_k {\bf k}_i(x_i, x_j)
+
+where :math:`\beta_k > 0`, :math:`\sum_{k=1}^{K} \beta_k = 1`, :math:`K` is the number of sub-kernels, :math:`\bf{k}` is a combined kernel, :math:`{\bf k}_i` is an individual kernel and :math:`{x_i}_i` are the training data.
+
+Regression is done by using :sgclass:`CSVMLight`. `See the SVR cookbook <http://shogun.ml/cookbook/latest/examples/regression/support_vector_regression.html>`_ for more details.
+
+
+See :cite:`sonnenburg2006large` for more information about MKL.
+
+-------
+Example
+-------
+
+Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and :sgclass:`CRegressionLabels` as
+
+.. sgexample:: multiple_kernel_learning.sg:create_features
+
+Then we create indvidual kernels like :sgclass:`CPolyKernel` and :sgclass:`CGaussianKernel` which will be later combined in one :sgclass:`CCombinedKernel`.
+
+.. sgexample:: multiple_kernel_learning.sg:create_kernel
+
+We create an instance of :sgclass:`CCombinedKernel` and append the :sgclass:`CKernel` objects.
+
+.. sgexample:: multiple_kernel_learning.sg:create_combined_train
+
+:sgclass:`CMKLRegression` needs an SVM solver as input. We here use :sgclass:`SVMLight`. We create an object of :sgclass:`SVMLight` and :sgclass:`CMKLRegression`, provide the combined kernel and labels before training it.
+
+.. sgexample:: multiple_kernel_learning.sg:train_mkl
+
+After training, we can extract :math:`\beta` and SVM coefficients :math:`\alpha`.
+
+.. sgexample:: multiple_kernel_learning.sg:extract_weights
+
+We set the updated kernel and predict :sgclass:`CRegressionLabels` for test data.
+
+.. sgexample:: multiple_kernel_learning.sg:mkl_apply
+
+Finally, we can evaluate the :sgclass:`CMeanSquaredError`.
+
+.. sgexample:: multiple_kernel_learning.sg:evaluate_error
+
+----------
+References
+----------
+:wiki:`Multiple_kernel_learning`
+
+.. bibliography:: ../../references.bib
+    :filter: docname in docnames

doc/cookbook/source/references.bib

@@ -86,3 +86,12 @@ @inproceedings{shalev2011shareboost
   pages={1179--1187},
   year={2011}
 }
+@article{sonnenburg2006large,
+  title={Large scale multiple kernel learning},
+  author={S. Sonnenburg and G. R{\"a}tsch and C. Sch{\"a}fer and B. Sch{\"o}lkopf},
+  journal={The Journal of Machine Learning Research},
+  volume={7},
+  pages={1531--1565},
+  year={2006},
+  publisher={JMLR.org}
+}

examples/meta/src/regression/multiple_kernel_learning.sg

@@ -0,0 +1,51 @@
+CSVFile f_feats_train("../../data/regression_1d_sinc_features_train.dat")
+CSVFile f_feats_test("../../data/regression_1d_sinc_features_test.dat")
+CSVFile f_labels_train("../../data/regression_1d_sinc_labels_train.dat")
+CSVFile f_labels_test("../../data/regression_1d_sinc_labels_test.dat")
+
+#![create_features]
+RealFeatures features_train(f_feats_train)
+RealFeatures features_test(f_feats_test)
+RegressionLabels labels_train(f_labels_train)
+RegressionLabels labels_test(f_labels_test)
+#![create_features]
+
+#![create_kernel]
+PolyKernel poly_kernel(10,2)
+GaussianKernel gauss_kernel_1(2)
+GaussianKernel gauss_kernel_2(3)
+#![create_kernel]
+
+#![create_combined_train]
+CombinedKernel combined_kernel()
+combined_kernel.append_kernel(poly_kernel)
+combined_kernel.append_kernel(gauss_kernel_1)
+combined_kernel.append_kernel(gauss_kernel_2)
+combined_kernel.init(features_train, features_train)
+#![create_combined_train]
+
+#![train_mkl]
+SVRLight binary_svm_solver()
+MKLRegression mkl(binary_svm_solver)
+mkl.set_kernel(combined_kernel)
+mkl.set_labels(labels_train)
+mkl.train()
+#![train_mkl]
+
+#![extract_weights]
+RealVector beta = combined_kernel.get_subkernel_weights()
+RealVector alpha = mkl.get_alphas()
+#![extract_weights]
+
+#![mkl_apply]
+combined_kernel.init(features_train, features_test)
+RegressionLabels labels_predict = mkl.apply_regression()
+#![mkl_apply]
+
+#![evaluate_error]
+MeanSquaredError error()
+real mse = error.evaluate(labels_predict, labels_test)
+#![evaluate_error]
+
+# additional integration testing variables
+RealVector output = labels_predict.get_labels()