Drhuffman12/add team utils part 4 (#59)

* drhuffman12/add_team_utils_part_4 TODO: update/refactor based on newer Breed::Manager's

* drhuffman12/add_team_utils_part_4 code cleanup

* drhuffman12/add_team_utils_part_4 switch to using 'CounterSafe'

* drhuffman12/add_team_utils_part_4 re-enable/add some specs

* drhuffman12/add_team_utils_part_4 reformat

* drhuffman12/add_team_utils_part_4 more tests

* drhuffman12/add_team_utils_part_4 formatting

* drhuffman12/add_team_utils_part_4 wrap up tests for 'MiniNetManager'

* drhuffman12/add_team_utils_part_4 reorg 'history_size' and 'error_stats'

* update README badges

* drhuffman12/add_team_utils_part_4 re-org non-JSON-specific tests out of 'from_json' test folder and into 'cmn' folder.

* drhuffman12/add_team_utils_part_4 refactoring

* drhuffman12/add_team_utils_part_4 param cleanup

* drhuffman12/add_team_utils_part_4 formatting

* drhuffman12/add_team_utils_part_4 rename 'disable_bias' to 'bias_disabled'

* drhuffman12/add_team_utils_part_4 rename 'disable_bias' to 'bias_disabled'

* drhuffman12/add_team_utils_part_4 refactor re RnnSimpleManager

* drhuffman12/add_team_utils_part_4 reformat

* drhuffman12/add_team_utils_part_4 refactor tests re RnnSimpleManager

* drhuffman12/add_team_utils_part_4 reformat

* drhuffman12/add_team_utils_part_4 finish 'RnnSimpleManager' and add 'Manager#train_team' and 'Manager#train_team_using_sequence'; update 'README.md'

* drhuffman12/add_team_utils_part_4 version bump

* drhuffman12/add_team_utils_part_4 use estimate_better_delta for team training (about half the time, and random about the other)

* drhuffman12/add_team_utils_part_4 formatting

* drhuffman12/add_team_utils_part_4 revise estimate_better_delta

* drhuffman12/add_team_utils_part_4 TODO: (a) move to 'spec_bench' and (b) replace here with more 'always' tests

README.md

@@ -1,6 +1,8 @@
 # ai4cr
 
-CircleCI status: [![CircleCI](https://circleci.com/gh/drhuffman12/ai4cr.svg?style=svg)](https://circleci.com/gh/drhuffman12/ai4cr)
+[![CircleCI](https://circleci.com/gh/drhuffman12/ai4cr.svg?style=svg)](https://circleci.com/gh/drhuffman12/ai4cr)
+
+![example branch parameter](https://github.com/drhuffman12/ai4cr/workflows/Crystal%20CI/badge.svg?branch=master)
 
 [![GitHub release](https://img.shields.io/github/release/drhuffman12/ai4cr.svg)](https://GitHub.com/drhuffman12/ai4cr/releases/)
 
@@ -42,7 +44,7 @@ net2 = Ai4cr::NeuralNetwork::Backpropagation.from_json(json)
 assert_approximate_equality_of_nested_list net.weights, net2.weights, 0.000000001
 ```
 
-## Optimiaztions
+## Optimizations
 
 * Compiler
 
@@ -52,17 +54,36 @@ e.g.: `time crystal spec --release`
 
 * Multithreading
 
-Use the `-Dpreview_mt` (or `-D preview_mt`) flag for multithreading.
+Use the `-Dpreview_mt` (for `crystal build` or `-D preview_mt` for `crystal spec`) flag for multithreading.
+
+e.g.: `CRYSTAL_WORKERS=16 crystal spec spec/ai4cr/neural_network/rnn/rnn_simple_manager_spec.cr --release -D preview_mt`
 
 See also:
 * https://crystal-lang.org/2019/09/23/crystal-0.31.0-released.html
 * https://crystal-lang.org/2019/09/06/parallelism-in-crystal.html
 
 ## Comparison benchmarks
 
+### Clear Crystal cache
+
+* Shards:
+
+```
+rm -rf ~/.cache/shards/
+```
+
+* Compiler:
+
+```
+rm -rf ~/.cache/crystal/
+```
+
+### Run benchmarks
+
 REMINDER: Running Crystal in a Docker container (at least used to) runs slower than running Crystal on bare-metal.
 So, for more performance, run it outside of a Docker container.
 
+
 To build and run them:
 
 ```

docs/Ai4cr/NeuralNetwork/Backpropagation.html

@@ -107,7 +107,7 @@ <h2>Overview</h2>
 <p>Use class method get_parameters_info to obtain details on the algorithm
 parameters. Use set_parameters to set values for this parameters.</p>
 
-<ul><li>:disable_bias => If true, the algorithm will not use bias nodes.
+<ul><li>:bias_disabled => If true, the algorithm will not use bias nodes.
   False by default.</li><li>:initial_weight_function => f(n, i, j) must return the initial
   weight for the conection between the node i in layer n, and node j in
   layer n+1. By default a random number in [-1, 1) range.</li><li>:propagation_function => By default:
@@ -248,12 +248,12 @@ <h2>Instance Method Summary</h2>
       </li>
 
       <li class="entry-summary">
-        <a href="#disable_bias%3ABool-instance-method" class="signature"><strong>#disable_bias</strong> : Bool</a>
+        <a href="#bias_disabled%3ABool-instance-method" class="signature"><strong>#bias_disabled</strong> : Bool</a>
 
       </li>
 
       <li class="entry-summary">
-        <a href="#disable_bias%3D%28disable_bias%29-instance-method" class="signature"><strong>#disable_bias=</strong>(disable_bias)</a>
+        <a href="#bias_disabled%3D%28disable_bias%29-instance-method" class="signature"><strong>#bias_disabled=</strong>(bias_disabled)</a>
 
       </li>
 
@@ -647,12 +647,12 @@ <h2>Instance Method Detail</h2>
       </div>
     </div>
 
-    <div class="entry-detail" id="disable_bias:Bool-instance-method">
+    <div class="entry-detail" id="bias_disabled:Bool-instance-method">
       <div class="signature">
 
-        def <strong>disable_bias</strong> : Bool
+        def <strong>bias_disabled</strong> : Bool
 
-        <a class="method-permalink" href="#disable_bias%3ABool-instance-method">#</a>
+        <a class="method-permalink" href="#bias_disabled%3ABool-instance-method">#</a>
       </div>
 
       <br/>
@@ -663,12 +663,12 @@ <h2>Instance Method Detail</h2>
       </div>
     </div>
 
-    <div class="entry-detail" id="disable_bias&#61;&#40;disable_bias&#41;-instance-method">
+    <div class="entry-detail" id="bias_disabled&#61;&#40;bias_disabled&#41;-instance-method">
       <div class="signature">
 
-        def <strong>disable_bias=</strong>(disable_bias)
+        def <strong>bias_disabled=</strong>(bias_disabled)
 
-        <a class="method-permalink" href="#disable_bias%3D%28disable_bias%29-instance-method">#</a>
+        <a class="method-permalink" href="#bias_disabled%3D%28disable_bias%29-instance-method">#</a>
       </div>
 
       <br/>

docs/index.html

@@ -156,7 +156,7 @@ <h3>Example Spec Results (running on a Lenovo Ideapad y700 w/ i7-6700HQ)</h3>
   #dump
     when given a net with structure of [3, 2]
       @structure of the dumped net matches @structure of the loaded net
-      @disable_bias on the dumped net matches @disable_bias of the loaded net
+      @bias_disabled on the dumped net matches @bias_disabled of the loaded net
       @learning_rate of the dumped net approximately matches @learning_rate of the loaded net
       @momentum of the dumped net approximately matches @momentum of the loaded net
       @weights of the dumped net approximately matches @weights of the loaded net

shard.yml

@@ -1,5 +1,5 @@
 name: ai4cr
-version: 0.1.21
+version: 0.1.22
 
 authors:
   - Daniel Huffman <drhuffman12@yahoo.com>
@@ -13,8 +13,10 @@ dependencies:
 
   ascii_bar_charter:
     github: drhuffman12/ascii_bar_charter
-    # version: ~> 1.4.0
-    branch: master
+    # branch: master
+
+  counter_safe:
+    github: drhuffman12/counter_safe
 
 development_dependencies:
 
@@ -58,7 +60,7 @@ development_dependencies:
   faker:
     github: askn/faker
 
-  # # TODO: Fix ameba dependencies or wait for newer version of anykeyh/crystal-coverage
+  # # TODO: Fix ameba dependencies or wait for newer version of crystal-coverage (aka https://github.com/anykeyh/crystal-coverage)
   # # Error is:
   # # ```
   # # 12 | new(YAML::ParseContext.new, parse_yaml(string_or_io))

spec/ai4cr/breed/manager_spec.cr

@@ -20,11 +20,21 @@ class MyBreed
   ALLOWED_STRINGS      = (ALLOWED_STRING_FIRST..ALLOWED_STRING_LAST).to_a
   property some_string : String = (ALLOWED_STRINGS.sample) * 2
 
-  def initialize(@name, @some_value)
+  property history_size : Int32
+  property error_stats : Ai4cr::ErrorStats
+
+  def initialize(@name, @some_value, @history_size = 2)
+    @error_stats = Ai4cr::ErrorStats.new(history_size)
   end
 end
 
 class MyBreedManager < Ai4cr::Breed::Manager(MyBreed)
+  include JSON::Serializable
+  class_getter counter : CounterSafe::Exclusive = CounterSafe::Exclusive.new
+
+  def initialize
+  end
+
   def mix_parts(child : T, parent_a : T, parent_b : T, delta)
     some_value = mix_one_part_number(parent_a.some_value, parent_b.some_value, delta)
     child.some_value = some_value
@@ -39,10 +49,6 @@ class MyBreedManager < Ai4cr::Breed::Manager(MyBreed)
   end
 end
 
-def puts_debug(message = "")
-  puts message if ENV.has_key?("DEBUG") && ENV["DEBUG"] == "1"
-end
-
 Spectator.describe Ai4cr::Breed::Manager do
   let(my_breed_manager) { MyBreedManager.new }
 
@@ -107,7 +113,7 @@ Spectator.describe Ai4cr::Breed::Manager do
 
       # cain
       child_1 = my_breed_manager.breed(ancestor_adam, ancestor_eve, delta: delta_child_1)
-      child_1.name = "Cain, child of #{child_1.name} and #{ancestor_eve.name}"
+      child_1.name = "Cain, child of #{ancestor_adam.name} and #{ancestor_eve.name}"
 
       puts_debug "child_1: #{child_1.to_json}"
       expected_birth_counter += 1
@@ -119,7 +125,7 @@ Spectator.describe Ai4cr::Breed::Manager do
 
       # abel
       child_2 = my_breed_manager.breed(ancestor_adam, ancestor_eve, delta: delta_child_2)
-      child_2.name = "Abel, child of #{child_2.name} and #{ancestor_eve.name}"
+      child_2.name = "Abel, child of #{ancestor_adam.name} and #{ancestor_eve.name}"
 
       puts_debug "child_2: #{child_2.to_json}"
       expected_birth_counter += 1

spec/ai4cr/neural_network/backpropagation_spec.cr

@@ -91,7 +91,7 @@ describe Ai4cr::NeuralNetwork::Backpropagation do
       expected_weights_first_size = 2 # one less than prev example since bias is disabled here
       expected_weights_first_sub_size = 2
       net = Ai4cr::NeuralNetwork::Backpropagation.new(structure).init_network
-      net.disable_bias = true
+      net.bias_disabled = true
       net.init_network
 
       it "sets @activation_nodes to expected nested array" do
@@ -177,8 +177,8 @@ describe Ai4cr::NeuralNetwork::Backpropagation do
         assert_equality_of_nested_list net.structure, net2.structure
       end
 
-      it "@disable_bias on the dumped net matches @disable_bias of the loaded net" do
-        net.disable_bias.should eq(net2.disable_bias)
+      it "@bias_disabled on the dumped net matches @bias_disabled of the loaded net" do
+        net.bias_disabled.should eq(net2.bias_disabled)
       end
 
       it "@learning_rate of the dumped net approximately matches @learning_rate of the loaded net" do

spec/ai4cr/neural_network/cmn/chain_spec.cr

@@ -26,8 +26,8 @@ describe Ai4cr::NeuralNetwork::Cmn::Chain do
     context "#init_network" do
       it "the 'outputs_guessed' start as zeros" do
         # prep net vvv
-        net0 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 2, width: 4, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: false)
-        net1 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 4, width: 3, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: true)
+        net0 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 2, width: 4, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: false)
+        net1 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 4, width: 3, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: true)
 
         net0.init_network
         net0.learning_rate = 0.25
@@ -57,8 +57,8 @@ describe Ai4cr::NeuralNetwork::Cmn::Chain do
     context "#eval" do
       it "the 'outputs_guessed' are updated as expected" do
         # prep net vvv
-        net0 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 2, width: 4, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: false)
-        net1 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 4, width: 3, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: true)
+        net0 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 2, width: 4, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: false)
+        net1 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 4, width: 3, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: true)
 
         net0.init_network
         net0.learning_rate = 0.25
@@ -91,8 +91,8 @@ describe Ai4cr::NeuralNetwork::Cmn::Chain do
         # TODO: FIX!!!
 
         # prep net vvv
-        net0 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 2, width: 4, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: false)
-        net1 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 4, width: 3, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: true)
+        net0 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 2, width: 4, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: false)
+        net1 = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: 4, width: 3, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: true)
 
         net0.init_network
         net0.learning_rate = 0.25
@@ -136,10 +136,10 @@ describe Ai4cr::NeuralNetwork::Cmn::Chain do
     layer_3_size_without_bias = 6
     layer_4_size_without_bias = 7
 
-    nt = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_0_size_without_bias, width: layer_1_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_TANH, disable_bias: false)
-    nr = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_1_size_without_bias, width: layer_2_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_RELU, disable_bias: true)
-    np = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_2_size_without_bias, width: layer_3_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_PRELU, disable_bias: true)
-    ne = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_3_size_without_bias, width: layer_4_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, disable_bias: true)
+    nt = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_0_size_without_bias, width: layer_1_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_TANH, bias_disabled: false)
+    nr = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_1_size_without_bias, width: layer_2_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_RELU, bias_disabled: true)
+    np = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_2_size_without_bias, width: layer_3_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_PRELU, bias_disabled: true)
+    ne = Ai4cr::NeuralNetwork::Cmn::MiniNet.new(height: layer_3_size_without_bias, width: layer_4_size_without_bias, learning_style: Ai4cr::NeuralNetwork::LS_SIGMOID, bias_disabled: true)
 
     arr = Array(Ai4cr::NeuralNetwork::Cmn::MiniNet).new
     arr << nt