feat(mathy): cleanup and remove dead code

Mathy had experimental agents and code that were hard to maintain and reduced clarity with the package. Remove all but a single agent (A3C) and refactor the models to use the keras functional API.

* feat(a3c): replace LSTM with SIREN layers
 - add time feature back (to help the value function predict end-of-episode variance in values)
 - remove LSTMs and self-attention mechanism from embedding layer
 - add Sin activated dense layers for inputs, and feed them into a SIREN layer at the end.
 - replace PolicyValueReward heads with SIREN layers

NOTE: The SIREN model trains atleast as well as the LSTMs with self-attention, and is much smaller/faster on CPU.

* feat(episode_memory): add to_window_observations

 - for computing windows of (n) size with (s) stride over the entire EpisodeMemory state. This can be used to turn terminal episode sequences into smalle n-step windows for training.
 - allows dropping partial windows (e.g. at the end of the episode)
 - allows zipping the output with any other keys in the EpisodeMemory class
 - allows including an extra iterable of the same length as the EpisodeMemory to be zipped with the windows.

* chore(zero): drop zero agent and related code

BREAKING CHANGE: This removes the MCTS powered agent. It was a nice agent but the pieces required for it to work effectively have been lost. It's not a primary use-case of mine now, so it's better to drop it and remove any confusion.

* refactor(agents): move to singular agent with generic names

 - PolicyValueModel -> AgentModel
 - Remove embeddings model and make it part of AgentModel

* refactor(model): replace custom keras model with functional

 - allows graphing model architecture for website
 - adds extra validation
 - allows full-model saving (without explicit optimizer save/loading)

* refactor(model): move action masking out of the model

 - Soooo masking doesn't play well with full-model saving and loading, or functional models, or keras strict data validation. Drop it out of the model, and let the action selector do its own masking.
 - TODO: add a loss term for the mask in addition to the policy loss?

* chore: misc cleanup and fix tests

* chore: update tf_siren dep

* chore: fix some tests and remove dead ones

* refactor(core): use mathy_core for CAS needs

This moves a lot of complexity out to another package, which is desirable for a number of reasons:
- when you only need to do CAS things, the mathy_core package can be installed without the need for larger things like TensorFlow or PyTorch which can make the install time much quicker
- the build time for the Mathy repo goes down by however much time it took to compile and test the core mathy code.
- it's easier to browse the code if you're unfamiliar with Mathy because it's not buried in a monorepo under a mountain of other folders

* chore: fix website build

* perf(ci): refactor virtual envs to reduce build time

 - share a root virtualenv to avoid double or triple installing hefty packages like TensorFlow and PyTorch

* chore: remove ci specific setup script

* chore: remove disk space cleanup step

 - it takes about a minute, and it's unclear if it's needed now that we share a root virtual env

* chore: drop keras-self-attention dep

* chore: fix requirements for tf agent

* chore: fix requirements for mkdocs plugin

* chore: install graphviz on ci build machine

* chore: try tensorflow 2.2.0

 - I think this caused optimizer loading problems last time I tried to upgrade, but let's see

* chore: fix import errors

* chore: try to confirm siren fix

* chore: remove kwargs from Sine layer

* chore: remove second pytest invocation for zero

 - TensorFlow didn't mix too well with multiprocessing so I had to run the MP tests separately for the Zero agent.
 - Zero agent is gone, cya extra test invocation

* chore: fix venv path in test scripts

* chore: drop example apps from main package

 - will add them later as mathy org repositories
 - update API docs
 - drop some unused queues

* chore: drop reformer test

* chore: bump tf_siren to 0.0.5 for tf2.2

* chore: drop misc cleanup

* test(cli): better coverage for generating problems

* chore: cleanup from review

* chore: cleanup from review


BREAKING CHANGES: This removes the "zero" agent entirely, and invalidates existing pretrained mathy models.

Author: justindujardin

.github/workflows/main.yml

@@ -13,16 +13,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v1
-      # from: https://bit.ly/2yo0tXG
-      - name: Free Disk Space
-        run: |
-          sudo swapoff -a
-          sudo rm -f /swapfile
-          sudo apt clean
-          docker rmi $(docker image ls -aq)
-          df -h
+      - name: Install Graphviz
+        run: sudo apt-get install graphviz
       - name: Setup Packages
-        run: sh tools/ci-setup.sh
+        run: sh tools/setup.sh
       - name: Build Packages
         run: sh tools/build.sh
       - name: Test Packages

.github/workflows/release.yml

@@ -31,3 +31,6 @@ build
 coverage
 
 libraries/website/site/
+
+
+.ipynb_checkpoints

.gitignore

@@ -1,57 +1,50 @@
 {
-  "name": "mathy_alpha_sm",
-  "version": "0.7.14",
-  "description": "Mathy.ai multi-task small model",
-  "license": "CC BY-SA 3.0",
-  "author": "Justin DuJardin",
-  "email": "justin@dujardinconsulting.com",
-  "url": "https://mathy.ai",
-  "mathy_version": ">=0.6.7,<1.0.0",
-  "prediction_window_size": 16,
-  "dropout": 0.2,
-  "units": 256,
-  "embedding_units": 256,
-  "lstm_units": 128,
-  "topics": [
+  "name":"unnamed_model",
+  "version":"0.0.1",
+  "description":"Mathy.ai trained model",
+  "license":"CC BY-SA 3.0",
+  "author":"Justin DuJardin",
+  "email":"justin@dujardinconsulting.com",
+  "url":"https://mathy.ai",
+  "mathy_version":">=0.7.14,<1.0.0",
+  "seq_len":256,
+  "prediction_window_size":6,
+  "units":256,
+  "embedding_units":256,
+  "topics":[
     "poly",
-    "poly-combine",
     "complex",
-    "binomial",
-    "poly-blockers"
+    "binomial"
   ],
-  "difficulty": null,
-  "model_dir": "mathy_alpha_sm",
-  "model_name": "model",
-  "init_model_from": null,
-  "train": false,
-  "verbose": true,
-  "lr_initial": 0.01,
-  "lr_decay_steps": 100,
-  "lr_decay_rate": 0.96,
-  "lr_decay_staircase": true,
-  "max_eps": 15000,
-  "summary_interval": 100,
-  "gamma": 0.99,
-  "num_workers": 4,
-  "td_lambda": 0.2,
-  "print_training": false,
-  "print_model_call_times": false,
-  "print_mode": "terminal",
-  "normalization_style": "layer",
-  "update_gradients_every": 8,
-  "mcts_sims": 10,
-  "worker_wait": 0.01,
-  "profile": false,
-  "action_strategy": "a3c",
-  "main_worker_use_epsilon": false,
-  "e_greedy_min": 0.01,
-  "e_greedy_max": 0.1,
-  "entropy_loss_scaling": 0.05,
-  "normalize_entropy_loss": true,
-  "normalize_pi_loss": true,
-  "aux_tasks_weight_scale": 1,
-  "teacher_start_evaluations_at_episode": 50,
-  "teacher_evaluation_steps": 20,
-  "teacher_promote_wins": 0.95,
-  "teacher_demote_wins": 0.5
+  "difficulty":null,
+  "model_dir":"training/mathy_alpha_sm",
+  "model_name":"model",
+  "verbose":false,
+  "lr_initial":0.01,
+  "lr_decay_steps":100,
+  "lr_decay_rate":0.96,
+  "lr_decay_staircase":true,
+  "max_eps":15000,
+  "summary_interval":100,
+  "gamma":0.99,
+  "num_workers":3,
+  "td_lambda":0.5,
+  "print_training":false,
+  "print_model_call_times":false,
+  "print_mode":"terminal",
+  "update_gradients_every":8,
+  "worker_wait":0.01,
+  "profile":false,
+  "normalization_style":"layer",
+  "main_worker_use_epsilon":false,
+  "e_greedy_min":0.01,
+  "e_greedy_max":0.04,
+  "entropy_loss_scaling":0.05,
+  "normalize_entropy_loss":true,
+  "normalize_pi_loss":true,
+  "aux_tasks_weight_scale":1.0,
+  "teacher_start_evaluations_at_episode":500,
+  "teacher_evaluation_steps":20,
+  "teacher_promote_wins":0.9,
+  "teacher_demote_wins":0.65
 }

libraries/mathy_alpha_sm/mathy_alpha_sm/model.config.json

@@ -1,5 +1,5 @@
 #!/bin/bash
 set -e
-. .env/bin/activate
+. ../../.env/bin/activate
 echo "Build python package..."
 python setup.py sdist bdist_wheel