Merge pull request #238 from bouthilx/release-v0.1.5rc1

Release v0.1.5rc1

.gitignore

@@ -6,6 +6,8 @@
 .spyproject
 .ropeproject
 *.log
+*.pkl
+*.lock
 
 # StarUML documentation
 *.mdj

ROADMAP.md

@@ -3,21 +3,17 @@ Last update July 5th, 2019
 
 ## Next releases - Short-Term
 
-### v0.1.4
+### v0.1.6
 
-#### Trial interruption/suspension/resumption
-
-Handle interrupted or lost trials so that they can be automatically resumed by running workers.
-
-#### Command line tools to monitor experiments
+#### Auto-resolution of EVC
 
-See [#125](https://github.com/Epistimio/orion/issues/125)
+Make branching events automatically solved with sane defaults.
 
-### v0.1.5
+### v0.1.7
 
-#### Auto-resolution of EVC
+#### Preliminary Python API
 
-Make branching events automatically solved with sane defaults.
+Library API to simplify usage of algorithms without Oríon's worker.
 
 ## Next releases - Mid-Term
 

conda/meta.yaml

@@ -23,6 +23,7 @@ requirements:
     - pymongo >=3
     - gitpython
     - filelock
+    - tabulate
 
 test:
   import:

docs/src/conf.py

@@ -94,7 +94,7 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build']
+exclude_patterns = ['_build', '**/pptree.py']
 
 # The name of the Pygments (syntax highlighting) style to use.
 highlight_language = 'python3'
@@ -123,7 +123,7 @@
     # Toc options
     'collapse_navigation': False,
     'sticky_navigation': True,
-    'navigation_depth': 2,
+    'navigation_depth': 4,
     #  'includehidden': False,
     #  'titles_only': False
 }

docs/src/index.rst

@@ -15,8 +15,10 @@
    :maxdepth: 1
 
    user/pytorch
-   user/evc
+   user/monitoring
+   user/searchspace
    user/algorithms
+   user/evc
 
 .. toctree::
    :caption: Examples

docs/src/user/algorithms.rst

@@ -46,6 +46,8 @@ Configuration
 
 Seed for the random number generator used to sample new trials. Default is ``None``.
 
+.. _ASHA:
+
 ASHA
 ----
 

docs/src/user/cli/info.rst

@@ -0,0 +1,61 @@
+``info`` Detailed informations about experiments
+------------------------------------------------
+
+This command gives a detailed description of a given experiment.
+Here is an example of all the sections provided by the command.
+
+.. code-block:: console
+
+   orion info orion-tutorial
+
+.. code-block:: bash
+
+   Commandline
+   ===========
+   --lr~loguniform(1e-5, 1.0)
+
+
+   Config
+   ======
+   pool size: 1
+   max trials: inf
+
+
+   Algorithm
+   =========
+   random:
+       seed: None
+
+
+   Space
+   =====
+   /lr: reciprocal(1e-05, 1.0)
+
+
+   Meta-data
+   =========
+   user: <username>
+   datetime: 2019-07-18 18:57:57.840000
+   orion version: v0.1.5
+
+
+   Parent experiment
+   =================
+   root:
+   parent:
+   adapter:
+
+
+   Stats
+   =====
+   trials completed: 1
+   best trial:
+     /lr: 0.03543491957849911
+   best evaluation: 0.9626
+   start time: 2019-07-18 18:57:57.840000
+   finish time: 2019-07-18 18:58:08.565000
+   duration: 0:00:10.725000
+
+
+The last section contains information about the best trial so far, providing its
+hyperparameter values and the corresponding objective.

docs/src/user/cli/list.rst

@@ -0,0 +1,71 @@
+``list`` Overview of all experiments
+------------------------------------
+
+Once you have launched a certain amount of experiments, you might start to lose track of some of
+them. You might forget their name and whether or not they are the children of some other experiment
+you have also forgotten. In any cases, the ``list`` command for Oríon will help you visualize the
+experiments inside your database by printing them in a easy-to-understand tree-like structure.
+
+Configuration
+~~~~~~~~~~~~~
+As per usual with Oríon, if no configuration file is provided to the ``list`` command, the default
+configuration will be used. You can however provide a particular configuration file through the
+usual ``-c`` or ``--config`` argument. This configuration file needs only to contain a valid
+database configuration.
+
+Basic Usage
+~~~~~~~~~~~
+The most basic usage of ``list`` is to use it without any arguments. This will simply print out
+every experiments inside the database in a tree-like fashion, so that the children of the
+experiments are easily identifiable. Here is a sample output to serve as an example:
+
+.. code-block:: console
+
+         ┌child_2
+     root┤
+         └child_1┐
+                 └grand_child_1
+     other_root
+
+Here, you can see we have five experiments. Two of them are roots, which mean they do not have any
+parents. One of them, ``other_root``, does not have any children and so, it does not have any
+branches coming out of it. On the other hand, the ``root`` experiment has multiple children,
+``child_1`` and ``child_2``, which are printed on the same tree level, and one grand-child,
+``grand_child_1`` which branches from ``child_1``.
+
+The ``--name`` argument
+~~~~~~~~~~~~~~~~~~~~~~~
+The last example showed you how to print every experiments inside the database in a tree. However,
+if you wish to have an overview of the tree of a single experiment, you can add the ``--name``
+argument to the call to ``list`` and only the experiment with the provided name and its children
+will be shown. Here's two examples using the same set of experiments as above:
+
+.. code-block:: bash
+
+    orion list --name root
+
+Output
+
+.. code-block:: console
+
+         ┌child_2
+     root┤
+         └child_1┐
+                 └grand_child_1
+
+Here, the ``other_root`` experiment is not showned because it is not inside the ``root`` experiment
+tree.
+
+.. code-block:: bash
+
+    orion list --name child_1
+
+Output
+
+.. code-block:: console
+
+     child_1┐
+            └grand_child_1
+
+Here, the ``root`` and ``child_2`` experiments are not present because they are not children of
+``child_1``.

docs/src/user/cli/status.rst

@@ -0,0 +1,180 @@
+``status`` Overview of trials for experiments
+---------------------------------------------
+
+When you reach a certain amount of trials, it becomes hard to keep track of them. This is where the
+``status`` command comes into play. The ``status`` command outputs the status of the different
+trials inside every experiment or a specific EVC tree. It can either give you an overview of
+the different trials status, i.e., the number of currently ``completed`` trials and so on, or, it
+can give you a deeper view of the experiments by outlining every single trial, its status and its
+objective.
+
+Basic Usage
+~~~~~~~~~~~
+The most basic of usages is to simply run ``status`` without any other arguments, except for a local
+configuration file if needed. This will then output a list of all the experiments inside your
+database with the count of every type of trials related to them. If an experiment has at least one
+``completed`` trial associated with it, the objective value of the best one will be printed as well.
+Children experiments are printed below their parent and are indicated through a different tab
+alignment than their parent, mainly, one tab further. This continues on for grand-children, and so
+on and so forth. We provide an example output to illustrate this:
+
+.. code-block:: bash
+
+    root
+    ====
+    status       quantity    min example_objective
+    ---------  ----------  -----------------------
+    completed           5                  4534.95
+
+
+      child_1
+      =======
+      status       quantity    min example_objective
+      ---------  ----------  -----------------------
+      completed           5                  4547.28
+
+
+    other_root
+    ==========
+    status       quantity    min example_objective
+    ---------  ----------  -----------------------
+    completed           5                  4543.73
+
+The ``--all`` Argument
+~~~~~~~~~~~~~~~~~~~~~~
+The basic status command combines statistics of all trials for each status. However, if you want to
+see every individual trial, with its id and its status, you can use the ``--all``
+argument which will print out every single trial for each experiment with their full information.
+Here is a sample output using the same experiments and trials as before:
+
+.. code-block:: console
+
+    orion status --all
+
+.. code-block:: bash
+
+    root
+    ====
+    id                                status       min example_objective
+    --------------------------------  ---------  -----------------------
+    bc222aa1705b3fe3a266fd601598ac41  completed                  4555.13
+    59bf4c85305b7ba065fa770805e93cb1  completed                  4653.49
+    da5159d9d36ef44879e72cbe7955347c  completed                  4788.08
+    2c4c6409d3beeb179fce3c83b4f6d8f8  completed                  4752.45
+    6a82a8a55d2241d989978cf3a7ebbba0  completed                  4534.95
+
+
+      child_1
+      =======
+      id                                status       min example_objective
+      --------------------------------  ---------  -----------------------
+      a3395b7192eee3ca586e93ccf4f12f59  completed                  4600.98
+      e2e2e96e8e9febc33efb17b9de0920d1  completed                  4786.43
+      7e0b4271f2972f539cf839fbd1b5430d  completed                  4602.58
+      568acbcb2fa3e00c8607bdc2d2bda5e3  completed                  4748.09
+      5f9743e88a29d0ee87b5c71246dbd2fb  completed                  4547.28
+
+
+    other_root
+    ==========
+    id                                status       min example_objective
+    --------------------------------  ---------  -----------------------
+    aaa16658770abd3516a027918eb91be5  completed                  4761.33
+    68233ce61ee5edfb6fb029ab7daf2db7  completed                  4543.73
+    cc0b0532c56c56fde63ad06fd73df63f  completed                  4753.5
+    b5335589cb897bbea2b58c5d4bd9c0c1  completed                  4751.15
+    a4a711389844567ac1b429eff96964e4  completed                  4790.87
+
+
+
+The ``--collapse`` Argument
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+On the other hand, if you wish to only get an overview of the experiments and the amount of trials
+linked to them without looking through the whole EVC tree, you can use the ``--collapse``
+option. As its name indicates, it will collapse every children into the root experiment and make a
+total count of the amount of trials `in that EVC tree`. As always, we provide an output to
+give you an example:
+
+
+.. code-block:: console
+
+    orion status --collapse
+
+    root
+    ====
+    status       quantity    min example_objective
+    ---------  ----------  -----------------------
+    completed          10                  4534.95
+
+
+    other_root
+    ==========
+    status       quantity    min example_objective
+    ---------  ----------  -----------------------
+    completed           5                  4543.73
+
+
+The ``--name`` Argument
+~~~~~~~~~~~~~~~~~~~~~~~
+If you wish to isolate a single EVC tree and look at their trials instead of listing every
+single experiments, you can use the ``--name`` argument by itself or combine it with the ones above
+to obtain the same results, but constrained. Once again, some examples for each type of scenrario is
+given:
+
+.. code-block:: console
+
+    orion status --name root
+
+.. code-block:: bash
+
+    root
+    ====
+    status       quantity    min example_objective
+    ---------  ----------  -----------------------
+    completed          10                  4534.95
+
+
+      child_1
+      =======
+      status       quantity    min example_objective
+      ---------  ----------  -----------------------
+      completed          10                  4547.28
+
+.. code-block:: console
+
+    orion status --name root --all
+
+.. code-block:: bash
+
+    root
+    ====
+    id                                status       min example_objective
+    --------------------------------  ---------  -----------------------
+    bc222aa1705b3fe3a266fd601598ac41  completed                  4555.13
+    59bf4c85305b7ba065fa770805e93cb1  completed                  4653.49
+    da5159d9d36ef44879e72cbe7955347c  completed                  4788.08
+    2c4c6409d3beeb179fce3c83b4f6d8f8  completed                  4752.45
+    6a82a8a55d2241d989978cf3a7ebbba0  completed                  4534.95
+
+
+      child_1
+      =======
+      id                                status       min example_objective
+      --------------------------------  ---------  -----------------------
+      a3395b7192eee3ca586e93ccf4f12f59  completed                  4600.98
+      e2e2e96e8e9febc33efb17b9de0920d1  completed                  4786.43
+      7e0b4271f2972f539cf839fbd1b5430d  completed                  4602.58
+      568acbcb2fa3e00c8607bdc2d2bda5e3  completed                  4748.09
+      5f9743e88a29d0ee87b5c71246dbd2fb  completed                  4547.28
+
+.. code-block:: console
+
+    orion status --name root --collapse
+
+.. code-block:: bash
+
+    root
+    ====
+    status       quantity    min example_objective
+    ---------  ----------  -----------------------
+    completed          10                  4534.95