Merge pull request #93 from smohsinali/docs

Add how to run rkhs tutorial to the documnetation

docs/source/algorithms_and_datasets.rst

@@ -53,6 +53,16 @@ Whether it makes sense to do so is indicated in the column CV.
             <td align="center">Python</td>
             <td aligh="center">no</td>
         </tr>
+        <tr>
+            <td align="center"><a href="algorithms_and_datasets.html#braninhar6camel">RKHS</a></td>
+            <td align="center">1(-)</td>
+            <td align="center">1/-</td>
+            <td align="center">-</td>
+            <td align="center">-</td>
+            <td align="center">&lt; 1s</td>
+            <td align="center">Python</td>
+            <td aligh="center">no</td>
+        </tr>
         <tr>
             <td align="center"><a href="algorithms_and_datasets.html#braninhar6camel">Camelback function</a></td>
             <td align="center">2(-)</td>
@@ -143,20 +153,22 @@ Description
 
 .. _braninhar6camel:
 
-Branin, Hartmann 6d and Camelback Function
+Branin, RKHS, Hartmann 6d and Camelback Function
 ------------------------------------------
 
 This benchmark already comes with the basic *HPOlib* bundle.
 
 **Dependencies:** None |br|
 **Recommended:** None
 
-Branin, Camelback and the Hartmann 6d function are three simple test functions,
+Branin, RKHS, Camelback and the Hartmann 6d function are three simple test functions,
 which are easy and cheap to evaluate. More test functions can be found
-`here <http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm>`_
+`here <http://www-optima.amp.i.kyoto-u.ac.jp/member/student/hedar/Hedar_files/TestGO_files/Page364.htm>`_.
 |br|
 Branin has three global minima at (-pi, 12.275), (pi, 2.275), (9.42478, 2.475) where f(x)=0.397887.
 |br|
+RKHS has single global minima at x=0.89235 where f(x)=5.73839.
+|br|
 Camelback has two global minima at (0.0898, -0.7126) and (-0.0898, 0.7126) where f(x) = -1.0316
 |br|
 Hartmann 6d is more difficult with 6 local minima and one global optimum at

docs/source/install.rst

@@ -93,20 +93,20 @@ Installing inside an virtualenv
     .. code:: bash
 
         ls optimizers/smac
-            smac_2_06_01-dev_parser.py   smac_2_06_01-dev.py   smac_2_06_01-dev_src    smac_2_06_01-devDefault.cfg
+            smac_2_10_00-dev_parser.py   smac_2_10_00-dev.py   smac_2_10_00-dev_src    smac_2_10_00-devDefault.cfg
 
 5.  You can now run, e.g. smac with 200 evaluations on the branin function:
 
     .. code:: bash
 
         cd benchmarks/branin
-        HPOlib-run -o ../../optimizers/smac/smac -s 23
+        HPOlib-run -o ../../optimizers/smac/smac_2_10_00-dev -s 23
 
     This takes depending on your machine ~2 minutes. You can now plot the results of your first experiment:
 
     .. code:: bash
 
-        HPOlib-plot FIRSTRUN smac_2_06_01-dev_23_*/smac_*.pkl -s `pwd`/Plots/
+        HPOlib-plot FIRSTRUN smac_2_10_00-dev_23_*/smac_*.pkl -s `pwd`/Plots/
 
     You can test the other optimizers (spearmint will take quite longer 30min):
 
@@ -119,13 +119,13 @@ Installing inside an virtualenv
 
     .. code:: bash
 
-        HPOlib-plot SMAC smac_2_06_01-dev_23_*/smac_*.pkl TPE hyperopt_august2013_mod_23_*/hyp*.pkl SPEARMINT spearmint_april2013_mod_23_*/spear*.pkl -s `pwd`/Plots/
+        HPOlib-plot SMAC smac_2_10_00-dev_23_*/smac_*.pkl TPE hyperopt_august2013_mod_23_*/hyp*.pkl SPEARMINT spearmint_april2013_mod_23_*/spear*.pkl -s `pwd`/Plots/
 
     and to check the general performance on this super complex benchmark:
 
     .. code:: bash
 
-        HPOlib-plot BRANIN smac_2_06_01-dev_23_*/smac_*.pkl hyperopt_august2013_mod_23_*/hyp*.pkl spearmint_april2013_mod_23_*/spear*.pkl -s `pwd`/Plots/
+        HPOlib-plot BRANIN smac_2_10_00-dev_23_*/smac_*.pkl hyperopt_august2013_mod_23_*/hyp*.pkl spearmint_april2013_mod_23_*/spear*.pkl -s `pwd`/Plots/
 
 Usage without installation
 ==========================
@@ -136,14 +136,14 @@ If you decide to not install HPOlib, you need to download the optimizer code by
 
     cd optimizers
     wget http://www.automl.org/hyperopt_august2013_mod_src.tar.gz
-    wget http://www.automl.org/smac_2_06_01-dev_src.tar.gz
+    wget http://www.automl.org/smac_2_10_00-dev_src.tar.gz
     wget http://www.automl.org/spearmint_april2013_mod_src.tar.gz
 
     tar -xf hyperopt_august2013_mod_src.tar.gz
     mv hyperopt_august2013_mod_src tpe/
 
-    tar -xf smac_2_06_01-dev_src.tar.gz
-    mv smac_2_06_01-dev_src.tar.gz smac/
+    tar -xf smac_2_10_00-dev_src.tar.gz
+    mv smac_2_10_00-dev_src.tar.gz smac/
 
     tar -xf spearmint_april2013_mod_src.tar.gz
     mv spearmint_april2013_mod_src spearmint/

docs/source/manual.rst

@@ -25,45 +25,75 @@ benchmarks or create your own. Each benchmarks comes with an algorithm and
 (if necessary) a wrapper and data. If you want to use one of the benchmarks
 listed here, follow these steps:
 
-Let's say you want to run the logistic regression:
-
-1.  Read the description for :ref:`logistic regression <logreg>`
-    and install the dependencies (which are `THEANO <http://deeplearning.net/software/theano/>`_
-    and `scikit-data <http://jaberg.github.io/skdata/>`_) and maybe recommended
-    software.
-2.  Download the benchmark:
-
-        :bash:`wget www.automl.org/logistic.tar.gz`
-
-3.  Unpack:
-
-        :bash:`tar -xf logistic.tar.gz`
-
-4.  Inside the root directory you will find a script :bash:`wrappingLogistic.py`,
-    three directories with the name of the optimizers (plus one directory for
-    random search), two other directories, named :bash:`cv` and :bash:`nocv`
-    and a script :bash:`theano_teardown.py`. Now choose if you want to run the experiment with
-    crossvalidation or without. Change into the :bash:`cv` directory if you
-    want to use crossvalidation, if not, change into the :bash:`nocv`
-    directory. There you will find a :bash:`config.cfg` and which contains
-    information about how long to run the experiment, how many cross validation
-    folds to use etc.
-5.  Be sure you are connected to the internet, because logistic regression uses
-    `scikit-data <http://jaberg.github.io/skdata/>`_) to download data, when
-    called for the first time. Then run:
-
-        :bash:`HPOlib-run /path/to/optimizers/<tpe/hyperopt|smac|spearmint|tpe/random> [-s seed] [-t title]`
-
-    from inside the :bash:`cv` or :bash:`nocv` folder to run one optimizer for
-    as many evaluations as stated in :bash:`config.cfg`,
-    (100 times in this example).
-6.  More information about the :bash:`config.cfg` can be found :ref:`here
-    <adjust_settings>`.
-
-**NOTE:** Since calculations are done with the THEANO library, you can also run
-this benchmark on a NVIDIA GPU. This is switched off by default, but you can
-change this with the THEANO flags. You find them in :bash:`config.cfg` and
-information on how to set the THEANO flags :ref:`here <configure_theano>`.
+Let's say you want to run the Reproducing Kernel Hilbert space (`RKHS <https://github.com/iassael/function-rkhs>`_) function:
+
+1.  RKHS is located with other benchmarks inside :bash:`HPOlib/benchmarks` folder. To run the benchmark first go inside that folder.
+
+    .. code:: bash 
+        
+        cd HPOlib/benchmarks/rkhs
+
+2.  Inside this folder you can run one the optimizers (smac, tpe or spearmint) on RKHS function using HPOlib :
+
+    .. code:: bash
+
+        HPOlib-run -o ../../optimizers/smac/smac_2_10_00-dev -s 23
+        HPOlib-run -o ../../optimizers/tpe/h -s 23
+        HPOlib-run -o ../../optimizers/spearmint/spearmint_april2013 -s 23
+
+    Or more generally
+
+    .. code:: bash
+
+        HPOlib-run /path/to/optimizers/<tpe/hyperopt|smac|spearmint|tpe/random> [-s seed] [-t title]
+
+
+    By default optimizers will run 200 evaluations on the function. For smac and tpe this will take about 2 mins but for spearmint it will be longer than 45 mins, so change :bash:`number_of_jobs` parameter in :bash:`config.cfg` file in same folder to 50 or less.
+
+    .. code:: cfg
+
+        [SMAC]
+        p = params.pcs
+
+        [TPE]
+        space = space.py
+
+        [SPEARMINT]
+        config = config.pb
+
+        [HPOLIB]
+        console_output_delay = 2.0
+        function = python ../rkhs.py
+        number_of_jobs = 200 #Change this to 50.
+        result_on_terminate = 1000
+
+3.  Now you can plot results for the experiment in different ways:
+
+    Plot the results of only one optimizer:
+
+    .. code:: bash
+
+        HPOlib-plot FIRSTRUN smac_2_10_00-dev_23_*/smac_*.pkl -s `pwd`/Plots/
+
+    The Plots can be found inside folder named :bash:`Plots` in current working directory (:bash:`HPOlib/benchmarks/rkhs`) 
+
+    .. image:: MeanTrace_fr.png
+
+    and if you have run all optimizers and want to compare their results:
+
+    .. code:: bash
+
+        HPOlib-plot SMAC smac_2_10_00-dev_23_*/smac_*.pkl TPE hyperopt_august2013_mod_23_*/hyp*.pkl SPEARMINT spearmint_april2013_mod_23_*/spear*.pkl -s `pwd`/Plots/
+
+    .. image:: MeanTrace_all.png
+
+    and to check the general performance on this super complex benchmark:
+
+    .. code:: bash
+
+        HPOlib-plot RKHS smac_2_10_00-dev_23_*/smac_*.pkl hyperopt_august2013_mod_23_*/hyp*.pkl spearmint_april2013_mod_23_*/spear*.pkl -s `pwd`/Plots/
+
+    .. image:: MeanTrace_comparison.png
 
 .. <!-- ########################################################################
    HOWTO RUN YOUR OWN BENCHMARKS
@@ -84,7 +114,7 @@ what files you need:
 * One **directory** having the name of the optimizer for each optimizer you want to use.
   Currently, these are :bash:`hyperopt_august2013_mod`,
   :bash:`random_hyperopt2013_mod`,
-  :bash:`smac_2_06_01-dev` and :bash:`spearmint_april2013_mod`.
+  :bash:`smac_2_10_00-dev` and :bash:`spearmint_april2013_mod`.
 * One **search space** for each optimizer. This must be placed in the directory with the name of the optimizer.
   You can convert your searchspace to other formats with
   :ref:`HPOlib_convert <hpolib_convert>` from and to all three different