Merge pull request #391 from autonomio/master

v.0.6.3 to Production

.github/ISSUE_TEMPLATE/bug-report.md

@@ -1,24 +1,38 @@
 ---
 name: Bug Report
-about: something needs fixing
+about: I want to report something that is broken
 
 ---
 
-Thanks so much for coming here to raise an issue. Please take a moment to 'check' the below boxes:
+Thank you very much for reporting a bug on Talos. Before you do, please go through the below checklist carefully and make sure to prepare your bug report in a way that facilitates effective handling of the matter.
 
-- [ ] I'm up-to-date with the latest release:
-
-      pip install -U talos
+#### 1) Confirm the below
 
-- [ ] I've confirmed that my Keras model works outside of Talos.
+- [ ] My Python version is 3.5 or higher
+- [ ] I have searched through the issues [Issues](https://github.com/autonomio/talos/issues) for a duplicate
+- [ ] I've tested that my Keras model works as a stand-alone
 
-If you still have an error, please submit **complete trace** and a code with: 
+#### 2) Include the output of:
 
-- output of shape for x and y e.g. (212,12)
-- Talos params dictionary
-- The Keras model wired for Talos
-- Description of extra variables in the model
+`talos.__version__`
 
-You can provide the code in pastebin / gist or any other format you like. 
+#### 3) Explain clearly what you expect to happen
+
+*A description of what you tried to do and what you thought should happen.*
+
+#### 4) Explain what actually happened
+
+*A description of the issue in Talos that you had identified*
+
+#### 5) Provide a code-complete reference
+
+- [ ] My bug report includes an input model
+- [ ] My bug report includes a parameter dictionary
+- [ ] My bug report includes a `Scan()` command
+- [ ] My bug report question includes a link to a sample of the data
+
+NOTE: If the data is sensitive and can't be shared, [create dummy data](https://scikit-learn.org/stable/modules/classes.html#samples-generator) that mimics it.
+
+**A self-contained Jupyter Notebook, Google Colab, or similar is highly preferred and will speed up helping you with your issue.**
 
 -------------------------------------------------------------------------

.github/ISSUE_TEMPLATE/feature-request.md

@@ -0,0 +1,29 @@
+---
+name: Feature Request
+about: I want to suggest a new feature
+
+---
+
+Thanks a lot for suggesting a feature to Talos. Please take a moment to go through the below checklist to provide context in a way that makes it easy to take your request forward.
+
+#### 1) I think Talos should add...
+
+*A description of the feature with as much detail as you believe is valuable*
+
+#### 2) Once implemented, I can see how this feature will...
+
+*Explain how researchers will benefit from having this feature in Talos**
+
+#### 3) I believe this feature is... (choose one)
+
+- [ ] ...critically important
+- [ ] ...must have
+- [ ] ...nice to have
+
+#### 4) Given the chance, I'd be happy to make a PR for this feature...
+
+- [ ] ...definitely
+- [ ] ...possibly
+- [ ] ...unlikely
+
+-------------------------------------------------------------------------

.github/ISSUE_TEMPLATE/support-request.md

@@ -0,0 +1,43 @@
+---
+name: Support
+about: I want to ask for support
+
+---
+
+First off, make sure to check your [support options](https://github.com/autonomio/talos#-how-to-get-support).
+
+The preferred way to resolve usage related matters is through the [docs](https://autonomio.github.io/talos/#/) which are maintained up-to-date with the latest version of Talos.
+
+If you do end up asking for support in a new issue, make sure to follow the below steps carefully.
+
+#### 1) Confirm the below
+
+- [ ] I have looked for an answer in the [Docs](https://autonomio.github.io/talos)
+- [ ] My Python version is 3.5 or higher
+- [ ] I have searched through the issues [Issues](https://github.com/autonomio/talos/issues) for a duplicate
+- [ ] I've tested that my Keras model works as a stand-alone
+
+#### 2) Include the output of:
+
+`talos.__version__`
+
+#### 3) Explain clearly what you are trying to achieve
+
+*A description of your specific use-case and what you hope to achieve with it*
+
+#### 4) Explain what you have already tried
+
+*An outline of the steps that you have already taken so far*
+
+#### 5) Provide a code-complete reference
+
+- [ ] My support question includes an input model
+- [ ] My support question includes a parameter dictionary
+- [ ] My support question includes a `Scan()` command
+- [ ] My support question includes a link to a sample of the data
+
+NOTE: If the data is sensitive and can't be shared, [create dummy data](https://scikit-learn.org/stable/modules/classes.html#samples-generator) that mimics it.
+
+**A self-contained Jupyter Notebook, Google Colab, or similar is highly preferred and will speed up helping you with your issue.**
+
+-------------------------------------------------------------------------

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,14 @@
+## You want to make a PR to Talos
+
+Thanks so much :) First, please take a moment to carefully check through
+the below items:
+
+- [ ] Changes have gone through actual use testing
+- [ ] [Docs](https://autonomio.github.io/talos) are updated where relevant
+- [ ] Code is [PEP8](https://www.python.org/dev/peps/pep-0008/)
+- [ ] All local tests have passed (run ./test.sh in /talos)
+- [ ] Travis tests have passed
+- [ ] Open a pull request
+- [ ] PR is to daily-dev branch
+
+<hr>

.travis.yml

@@ -9,7 +9,6 @@ notifications:
     on_success: never
     on_failure: always
 python:
- - '2.7'
  - '3.5'
  - '3.6'
 install:

README.md

@@ -24,7 +24,7 @@
   <a href="#Examples">Examples</a> •
   <a href="#Install">Install</a> •
   <a href="#Support">Support</a> •
-  <a href="https://autonomio.github.io/docs_talos">Docs</a> •
+  <a href="https://autonomio.github.io/talos/">Docs</a> •
   <a href="https://github.com/autonomio/talos/issues">Issues</a> •
   <a href="#License">License</a> •
   <a href="https://github.com/autonomio/talos/archive/master.zip">Download</a>
@@ -50,25 +50,31 @@ Talos radically transforms ordinary Keras workflows without taking away any of K
 
 Talos is made for data scientists and data engineers that want to remain in **complete control of their Keras models**, but are tired of mindless parameter hopping and confusing optimization solutions that add complexity instead of reducing it. Within minutes, without learning any new syntax, Talos allows you to configure, perform, and evaluate hyperparameter optimization experiments that yield state-of-the-art results across a wide range of prediction tasks. Talos provides the **simplest and yet most powerful** available method for hyperparameter optimization with Keras.
 
-### Key Features
+<hr>
+
+### :wrench: Key Features
 
 Based on what no doubt constitutes a "biased" review (being our own) of more than ~30 hyperparameter tuning and optimization solutions, Talos comes on top in terms of intuitive, easy-to-learn, highly permissive access to critical hyperparameter optimization capabilities. Key features include:
 
 - Single-line optimize-to-predict pipeline `talos.Scan(x, y, model, params).predict(x_test, y_test)`
-- automated hyperparameter optimization
-- model generalization evaluator
-- experiment analytics
-- Random search
+- Automated hyperparameter optimization
+- Model generalization evaluator
+- Experiment analytics
+- Pseudo, Quasi, and Quantum Random search options
 - Grid search
-- Correlation based optimization
-- Pseudo, Quasi, and Quantum Random functions
+- Probabilistic optimizers
+- Single file custom optimization strategies
+- Dynamically change optimization strategy during experiment
+- Support for man-machine cooperative optimization strategy
 - Model candidate generality evaluation
 - Live training monitor
 - Experiment analytics
 
 Talos works on **Linux, Mac OSX**, and **Windows** systems and can be operated cpu, gpu, and multi-gpu systems.
 
-### Examples
+<hr>
+
+### 📈 Examples
 
 Get the below code [here](https://gist.github.com/mikkokotila/4c0d6298ff0a22dc561fb387a1b4b0bb). More examples further below.
 
@@ -84,11 +90,13 @@ The *Simple* example below is more than enough for starting to use Talos with an
 
 [Field Report](https://towardsdatascience.com/hyperparameter-optimization-with-keras-b82e6364ca53)  [~15 mins]
 
-For more information on how Talos can help with your Keras workflow, visit the [User Manual](https://autonomio.github.io/docs_talos).
+For more information on how Talos can help with your Keras workflow, visit the [User Manual](https://autonomio.github.io/talos/).
 
 You may also want to check out a visualization of the [Talos Hyperparameter Tuning workflow](https://github.com/autonomio/talos/wiki/Workflow).
 
-### Install
+<hr>
+
+### 💾 Install
 
 Stable version:
 
@@ -98,18 +106,34 @@ Daily development version:
 
 #### `pip install git+https://github.com/autonomio/talos.git@daily-dev`
 
-### Support
+<hr>
+
+### 💬 How to get Support
 
-Check out [common errors](https://github.com/autonomio/talos/wiki/Troubleshooting) in the Wiki.
+| I want to...                     | Go to...                                                  |
+| -------------------------------- | ---------------------------------------------------------- |
+| **...troubleshoot**           | [Docs] · [Wiki] · [GitHub Issue Tracker]                   |
+| **...report a bug**           | [GitHub Issue Tracker]                                     |
+| **...suggest a new feature**  | [GitHub Issue Tracker]                                     |
+| **...get support**            | [Stack Overflow] · [Spectrum Chat]                         |
+| **...have a discussion**      | [Spectrum Chat]                                            |
 
+[github issue tracker]: https://github.com/automio/talos/issues
+[docs]: https://autonomio.github.io/talos/
+[wiki]: https://github.com/autonomio/talos/wiki
+[stack overflow]: https://stackoverflow.com/questions/tagged/talos
+[spectrum chat]: https://spectrum.chat/talos
 
-Check the [Docs](https://autonomio.github.io/docs_talos) which is generally keeping up with Master (and pip package).
+<hr>
+
+### 📢 Citations
 
-If you want ask a **"how can I use Talos to..."** question, the right place is [StackOverflow](https://stackoverflow.com/questions/ask).
+If you use Talos for published work, please cite:
 
-If you found a bug or want to suggest a feature, check the [issues](https://github.com/autonomio/talos/issues) or [create](https://github.com/autonomio/talos/issues/new/choose) a new issue.
+`Autonomio Talos [Computer software]. (2019). Retrieved from http://github.com/autonomio/talos.`
 
+<hr>
 
-### License
+### 📃 License
 
 [MIT License](https://github.com/autonomio/talos/blob/master/LICENSE)

docs/Analyze.md

@@ -0,0 +1,61 @@
+# Analyze (previously Reporting)
+
+The experiment results can be analyzed through the [Analyze()](https://github.com/autonomio/talos/blob/master/talos/utils/reporting.py) utility. `Analyze()` may be used after Scan completes, or during an experiment (from a different shell / kernel).
+
+## Analyze Use
+
+```python
+r = Reporting('experiment_log.csv')
+
+# returns the results dataframe
+r.data
+
+# returns the highest value for 'val_fmeasure'
+r.high('val_fmeasure')
+
+# returns the number of rounds it took to find best model
+r.rounds2high()
+
+# draws a histogram for 'val_acc'
+r.plot_hist()
+```
+
+Reporting works by loading the experiment log .csv file which is saved locally as part of the experiment. The filename can be changed through dataset_name and experiment_no Scan arguments.
+
+## Analyze Arguments
+
+`Analyze()` has only a single argument `source`. This can be either a .csv file which results `Scan()` or the class object which also results from `Scan()`.
+
+The `Analyze` class object contains several useful properties.
+
+## Analyze Properties
+
+See docstrings for each function for a more detailed description.
+
+**`high`** The highest result for a given metric
+
+**`rounds`**  The number of rounds in the experiment
+
+**`rounds2high`** The number of rounds it took to get highest result
+
+**`low`** The lowest result for a given metric
+
+**`correlate`** A dataframe with Spearman correlation against a given metric
+
+**`plot_line`** A round-by-round line graph for a given metric
+
+**`plot_hist`** A histogram for a given metric where each observation is a permutation
+
+**`plot_corr`** A correlation heatmap where a single metric is compared against hyperparameters
+
+**`plot_regs`** A regression plot with data on two axis
+
+**`plot_box`** A box plot with data on two axis
+
+**`plot_bars`** A bar chart that allows up to 4 axis of data to be shown at once
+
+**`plot_bars`** Kernel Destiny Estimation type histogram with support for 1 or 2 axis of data
+
+**`table`** A sortable dataframe with a given metric and hyperparameters
+
+**`best_params`** A dictionary of parameters from the best model

docs/Asking_Help.md

@@ -0,0 +1,15 @@
+# 💬 How to get Support
+
+| I want to...                     | Go to...                                                  |
+| -------------------------------- | ---------------------------------------------------------- |
+| **...troubleshoot**           | [Docs] · [Wiki] · [GitHub Issue Tracker]                   |
+| **...report a bug**           | [GitHub Issue Tracker]                                     |
+| **...suggest a new feature**  | [GitHub Issue Tracker]                                     |
+| **...get support**            | [Stack Overflow] · [Spectrum Chat]                         |
+| **...have a discussion**      | [Spectrum Chat]                                            |
+
+[github issue tracker]: https://github.com/automio/talos/issues
+[docs]: https://autonomio.github.io/docs_talos
+[wiki]: https://github.com/autonomio/talos/wiki
+[stack overflow]: https://stackoverflow.com/questions/tagged/talos
+[spectrum chat]: https://spectrum.chat/talos

docs/AutoModel.md

@@ -0,0 +1,28 @@
+# AutoModel
+
+`AutoModel` provides a meaningful way to test several network architectures in an automated manner. Currently there are five supported architectures:
+
+- conv1d
+- lstm
+- bidirectional_lstm
+- simplernn
+- dense
+
+`AutoModel` creates an input model for Scan(). Optimized for being used together with `AutoParams()` and expects one or more of the above architectures to be included in params dictionary, for example:
+
+```python
+
+p = {...
+    'networks': ['dense', 'conv1d', 'lstm']
+    ...}
+
+```
+
+## AutoModel Arguments
+
+Argument | Input | Description
+--------- | ------- | -----------
+`task` | str or None | `binary`, `multi_label`, `multi_class`, or `continuous`
+`metric` | None or list | One or more Keras metric (functions) to be used in the model
+
+Setting `task` effects which various aspects of the model and should be set according to the specific prediction task, or set to `None` in which case `metric` input is required.