MAINT documentation #228

shukon · 2017-04-16T14:40:32Z

Rewriting documentation in general.

replacing example leading-ones with svm
add options-overview
refactor navigation structure (separating manual and quickstart)

This is not to be merged right away, but I would greatly appreciate an opinion on the work so far to see whether it is going in the right direction. Also to know things you are missing in the documentation in general would help.

To read, with sphinx installed:

cd doc/
make html

…c/restructure_examples

codecov-io · 2017-04-16T14:40:34Z

Codecov Report

Merging #228 into development will increase coverage by 0.61%.
The diff coverage is 100%.

@@              Coverage Diff               @@
##           development    #228      +/-   ##
==============================================
+ Coverage        85.28%   85.9%   +0.61%     
==============================================
  Files               42      42              
  Lines             2270    2369      +99     
==============================================
+ Hits              1936    2035      +99     
  Misses             334     334

Impacted Files	Coverage Δ
smac/tae/execute_ta_run_aclib.py	`71.21% <ø> (ø)`	⬆️
smac/optimizer/local_search.py	`95.91% <ø> (ø)`	⬆️
smac/utils/io/output_writer.py	`96.72% <ø> (ø)`	⬆️
smac/tae/execute_ta_run_old.py	`78.33% <ø> (ø)`	⬆️
smac/epm/random_epm.py	`66.66% <ø> (ø)`	⬆️
smac/runhistory/runhistory2epm.py	`83.33% <ø> (ø)`	⬆️
smac/facade/smac_facade.py	`94.32% <ø> (ø)`	⬆️
smac/epm/base_imputor.py	`100% <ø> (ø)`	⬆️
smac/epm/rf_with_instances.py	`100% <ø> (ø)`	⬆️
smac/initial_design/multi_config_initial_design.py	`81.08% <ø> (ø)`	⬆️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7319bd6...eff9fbb. Read the comment docs.

mfeurer · 2017-04-18T06:54:33Z

A great addition would be an API documentation, something similar to sklearn's documentation. I'm not sure if this is on your stack and/or whether this is the right time to start this.

shukon · 2017-04-18T15:30:59Z

@mfeurer I think API documentation would be great and work hand in hand with further improving the docstrings. Sphinx provides automatic generation of documentation from docstrings. However, the current theme does display the generated rst-files nicely. The widely used read-the-docs theme displays the generated API-files rather nicely. I post a screenshot for comparison.

RTD-Theme:

Bootstrap (current theme):

mfeurer · 2017-04-18T15:56:43Z

Yep that's look fine as well + it's actually developed by something like a company if I understand correctly. What's important to me is that there is a global menu on the page, which isn't there in the default layout.

shukon · 2017-04-25T16:05:05Z

You can now check out the API that is generated from SMAC as it is right now. Global menu on the left with collapsible per-page-menus.

mfeurer · 2017-04-25T16:19:14Z

I didn't check the text (PING again once that is ready for review), but I like the layout. I wonder though if it makes sense to divide the longer Sections into shorter ones (i.e. manual and quickstart guide).

mlindauer · 2017-05-09T13:19:38Z

@shukon any updates on this PR? Is it ready for a review iteration?

shukon · 2017-05-10T10:17:49Z

Soon! I'm on it. Fixed API-generation to get rid of the errors, trying to get toc right and rewrite a couple paragraphs.

shukon · 2017-05-15T22:17:34Z

@mlindauer @mfeurer I think this PR is ready for review.

cd doc
make buildapi
make html

updates most sections
Quickstart provides three examples, branin, svm and spear.
The manual so far aims to explain the difference between using SMAC from the cmdline and using SMAC within Python, lists and explains options and file-formats and presents the different TAEs.
finally got the API-doc to work properly and tried to update all docstrings so at least the format is processed correctly. I avoided messing with the content too much (because this PR already contains a big amount of changes), but tried to supplement the most important parts.
adds F.A.Q. (to which I just added 3 questions I had myself at some point, just thought it might make sense).

mlindauer · 2017-05-16T09:58:38Z

Hi,
I fixed some minor issues.

Open issues/todos are:

developer guide section
- contributing new code
- code style
- very brief overview of architecture
the logger output is not consistent in the python and cli example. this is an issue related to the SMAC code and has to be first fixed within SMAC.

mlindauer · 2017-05-17T07:18:11Z

Another issue I found today is that the API is not build anymore.
The build also throws errors such as
/home/lindauer/git/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.optimizer'

mfeurer · 2017-05-24T08:25:22Z

Actually, the first example (using CLI interface to optimize branin) might be intimidating for someone coming from the blackbox optimization literature and is used to an interface like this (see getting started section).

…SMAC3 into doc/restructure_examples

shukon · 2017-05-31T09:55:07Z

@mfeurer Added f_min to branin-example (prior the cmd-line explanation). This feels like the interface you talked about. However, I left the cmd-line in, so you have a direct comparison in the same example. What do you think?

mfeurer · 2017-06-06T12:52:18Z

Much better, thanks a lot. In general, it is a good idea to have a very minimal example, such as this one. Maybe it would be great to simplify the f_min example even further?

mfeurer

I went through the manual and a part of the API documentation. However, I stopped the second because a lot of the docstrings are not rendered yet.

mfeurer · 2017-06-06T12:55:11Z

doc/quickstart.rst

+   :lineno-match:
+
+This way, you can optimize a blackbox-function with minimal effort. However, to
+use the whole of *SMACs* capability, a scenario_ object should be used. There are a


You're explaining the CLI, so there's no scenario-object here. Maybe it would be a great idea to contrast the three different interfaces of SMAC? But that would be something for a new PR. We should get this one merged ASAP.

I'll keep that in mind.

mfeurer · 2017-06-06T12:57:15Z

doc/basic_usage.rst

+The usage of *SMAC* from your Python-code is described in the `SVM-example
+<quickstart.html#svm-example>`_.
+Scenario and PCS are both build within the code. The target algorithm needs to
+be registered with a `Target Algorithm Evaluator (TAE) <tae.html#tae>`_, which communicates


Nope, the default is to simply pass a function.

mfeurer · 2017-06-06T13:07:47Z

doc/options.rst

+The following assumes that the scenario is created via a scenario-file. If it is
+generated within custom code, you might not need *algo* or *paramfile*.
+
+Required:


Would it be possible to dynamically create this list of arguments from the scenario file?

mfeurer · 2017-06-06T13:09:49Z

doc/options.rst

+        * *tuner-timeout* is the maximum amount of CPU-time used for optimization. Default: inf.
+        * *wallclock_limit* is the maximum amount of wallclock-time used for optimization. Default: inf.
+        * *runcount_limit* is the maximum number of algorithm-calls during optimization. Default: inf.
+        * *minR* is the minimum number of calls per configuration. Default: 1


Could you please reword this description? minimum number of calls and maximum number of calls is not very descriptive if a user doesn't know about the intensify procedure.

mfeurer · 2017-06-06T13:10:53Z

doc/options.rst

+        # Forbiddens:
+        {parameter_name_1=value_1, ..., parameter_name_N=value_N}
+
+.. note::


Sure? Can't it read both the old and the new format? Or do you mean that it is more restrictive than the implementation of the new PCS format in SMAC2?

I am not sure what I mean ;). I'll cut the note out.

mfeurer · 2017-06-06T13:18:48Z

doc/apidoc/smac.epm.random_epm.rst

+smac\.epm\.random\_epm module
+=============================
+
+.. automodule:: smac.epm.random_epm


@mlindauer @KEggensperger as a general question, do we want the documentation to show all methods which are inherited from the base class? I think this would be a great addition to the docs.

I wouldn't mind as long as the documentation stays concise

mfeurer · 2017-06-06T13:21:47Z

doc/apidoc/smac.epm.rf_with_instances.rst

+smac\.epm\.rf\_with\_instances module
+=====================================
+
+.. automodule:: smac.epm.rf_with_instances


@sfalkner the docstring for rf_with_instances mentions the parameter max_num_nodes. I'm currently wondering if the tree is build in depth-first or breadth-first fashion as this would change the behaviour?

mfeurer · 2017-06-06T13:22:33Z

doc/apidoc/smac.epm.rfr_imputator.rst

+smac\.epm\.rfr\_imputator module
+================================
+
+.. automodule:: smac.epm.rfr_imputator


@KEggensperger could you please add the return values to the docstring of the impute() function?

I've just pushed a commit to this branch adding some more documentation to the BaseImputor class. See 0d9c36e

mfeurer · 2017-06-06T13:25:05Z

doc/apidoc/smac.facade.smac_facade.rst

@@ -0,0 +1,7 @@
+smac\.facade\.smac\_facade module
+=================================
+


@shukon could you please update the return value of get_trajectory in the docstring? This function actually returns a named tuple now.

mfeurer · 2017-06-06T13:28:42Z

doc/apidoc/smac.intensification.intensification.rst

+smac\.intensification\.intensification module
+=============================================
+
+.. automodule:: smac.intensification.intensification


Is there a reason why for most classes the constructor isn't shown?

Yes, this is intended. It is toggled via this option. Would you prefer it to be shown?

I would prefer to see all methods of a class. @KEggensperger @mlindauer what's your opinion on this?

All methods. What would be a good argument to hide the constructor?

mfeurer · 2017-06-06T13:39:49Z

@shukon also, is there a reason why you reference pages by their file names and don't reference the page by its name?

…te on help-content)

shukon · 2017-06-10T00:06:29Z

also, is there a reason why you reference pages by their file names and don't reference the page by its name?

@mfeurer Because the links where not working the way I wanted before - do the work for you? Especially with the "#"-link.

Anymore requested fixes I missed?

mfeurer · 2017-06-14T07:40:26Z

@mlindauer @shukon @KEggensperger I think we should merge this now as this PR does what it's supposed to do: restructure the docs so that the structure is the way we want it to be. Polishing the API documentation is in my opinion better suited in a new PR to not make this PR too big and make it block the development of SMAC. What do you guys think?

mlindauer

I haven't looked into the current version, but if you think that we can merge it now, I'm happy to approve it.

mfeurer · 2017-06-15T09:19:43Z

I just merged this and opened new issues #265 and #266.

shukon added 8 commits March 24, 2017 21:40

MAINT doc restructure examples in manual

ca2c007

ADD scenario-options to manual

ea15017

Merge branch 'development' of https://github.com/automl/SMAC3 into do…

5374db4

…c/restructure_examples

MAINT further specfiy examples

16a02a0

MAINT extend manual, fix leadingones-example

7e4a4cf

ADD svm-example and update docs

504b914

MAINT refactor files ADD TAE-explanation

39e94ce

MAINT refactor, missing files

9783949

ADD APIdoc, CHANGE doc-theme to rtd

02c0af6

mlindauer added this to the Release 1.0 milestone Apr 26, 2017

ADD scenario-argument help

e98aecb

FIX API-doc errors

71a7c50

shukon and others added 6 commits May 10, 2017 12:38

FIX typo in scenario

7d5046d

FIX toc in doc

863a747

ADD basic usage, introduction

cac1a77

MAINT update docstrings, API-doc, Makefile and general documentation

bad4a34

Merge branch 'development' into doc/restructure_examples

875516a

MAINT restore missing option

45dc9e2

MAINT Fixing a lot of minor things

5fa576b

minor changes in documentation

a7f4c69

shukon and others added 5 commits May 27, 2017 12:58

MAINT add links, remove tae from svm

cb4d5e4

Merge branch 'development' into doc/restructure_examples

64dc9e0

MAINT clarify quickstart

7112f51

Merge branch 'doc/restructure_examples' of https://github.com/automl/…

b8e694b

…SMAC3 into doc/restructure_examples

ADD f_min to branin-example

1ed9a50

Merge branch 'development' into doc/restructure_examples

4a2342c

mfeurer requested changes Jun 6, 2017

View reviewed changes

rewording of documentation

c0241b6

shukon added 6 commits June 10, 2017 00:49

MAINT change gcc to build-essentials

f7b7683

ADD auto-generation of scenario options in docs (along with some upda…

15f4553

…te on help-content)

REMOVE note on forbidden-clauses in PCS

f7acd66

ADD pynisher-mention

8c67d1f

ADD link to example in tae.rst

4876516

MAINT update get_trajectory-docstring

5f20991

shukon and others added 3 commits June 13, 2017 11:56

ADD unit-test for scenario-options-doc

b1f341f

MAINT extend docstring of BaseImputorClass

0d9c36e

fix doc, anaconda has no build-essentials

eff9fbb

mfeurer approved these changes Jun 14, 2017

View reviewed changes

KEggensperger approved these changes Jun 14, 2017

View reviewed changes

mlindauer approved these changes Jun 14, 2017

View reviewed changes

mfeurer merged commit f8c77e9 into development Jun 15, 2017

mfeurer deleted the doc/restructure_examples branch June 15, 2017 09:19

		@@ -0,0 +1,7 @@
		smac\.facade\.smac\_facade module
		=================================

MAINT documentation #228

MAINT documentation #228

Conversation

shukon commented Apr 16, 2017

codecov-io commented Apr 16, 2017 • edited

Codecov Report

mfeurer commented Apr 18, 2017

shukon commented Apr 18, 2017 • edited

mfeurer commented Apr 18, 2017 • edited

shukon commented Apr 25, 2017

mfeurer commented Apr 25, 2017

mlindauer commented May 9, 2017

shukon commented May 10, 2017

shukon commented May 15, 2017 • edited

mlindauer commented May 16, 2017

mlindauer commented May 17, 2017

mfeurer commented May 24, 2017

shukon commented May 31, 2017

mfeurer commented Jun 6, 2017 • edited

mfeurer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfeurer commented Jun 6, 2017

shukon commented Jun 10, 2017 • edited

mfeurer commented Jun 14, 2017

mlindauer left a comment

Choose a reason for hiding this comment

mfeurer commented Jun 15, 2017

codecov-io commented Apr 16, 2017 •

edited

shukon commented Apr 18, 2017 •

edited

mfeurer commented Apr 18, 2017 •

edited

shukon commented May 15, 2017 •

edited

mfeurer commented Jun 6, 2017 •

edited

shukon commented Jun 10, 2017 •

edited