Merge pull request #73 from ArtLinkov/master

Evolution Strategies Optimizer [WIP]

README.md

@@ -5,7 +5,11 @@
 
 
 SHAInet - stands for Super Human Artificial Intelligence network
-a neural network in pure [Crystal](https://crystal-lang.org/)  
+a neural network in pure [Crystal](https://crystal-lang.org/)
+
+This is a free-time project, happily hosted by NeuraLegion that was created as part of some internal research. We started it with research in mind, rather than production, and just kept going, also thanks to members of the community.
+
+We wanted to try and implement some inspiration from the biological world into this project. In addition to that, we wanted to try an approach for NNs using object-oriented modeling instead of matrices. The main reason behind that was, to try new types of neurons aiming for more robust learning (if possible) or at least have more fine-tuned control over the manipulation of each neuron (which is difficult using a matrix-driven approach).
 
 At the [Roadmap](https://github.com/NeuraLegion/shainet#development) you can see what we plan to add to the network as the project will progress.  
 
@@ -194,26 +198,87 @@ puts "We managed #{correct_answers} out of #{test_data.data_pairs.size} total"
 puts "Cnn output: #{cnn.output}"
 ```
 
+Evolutionary optimizer example:
+```crystal
+label = {
+      "setosa"     => [0.to_f64, 0.to_f64, 1.to_f64],
+      "versicolor" => [0.to_f64, 1.to_f64, 0.to_f64],
+      "virginica"  => [1.to_f64, 0.to_f64, 0.to_f64],
+    }
+
+    iris = SHAInet::Network.new
+    iris.add_layer(:input, 4, :memory, SHAInet.sigmoid)
+    iris.add_layer(:hidden, 4, :memory, SHAInet.sigmoid)
+    iris.add_layer(:output, 3, :memory, SHAInet.sigmoid)
+    iris.fully_connect
+
+    # Get data from a local file
+    outputs = Array(Array(Float64)).new
+    inputs = Array(Array(Float64)).new
+    CSV.each_row(File.read(__DIR__ + "/test_data/iris.csv")) do |row|
+      row_arr = Array(Float64).new
+      row[0..-2].each do |num|
+        row_arr << num.to_f64
+      end
+      inputs << row_arr
+      outputs << label[row[-1]]
+    end
+    data = SHAInet::TrainingData.new(inputs, outputs)
+    data.normalize_min_max
+
+    training_data, test_data = data.split(0.9)
+
+    iris.train_es(
+      data: training_data,
+      pool_size: 50,
+      learning_rate: 0.5,
+      sigma: 0.1,
+      cost_function: :c_ent,
+      epochs: 500,
+      mini_batch_size: 15,
+      error_threshold: 0.00000001,
+      log_each: 100,
+      show_slice: true)
+
+    # Test the trained model
+    correct = 0
+    test_data.data.each do |data_point|
+      result = iris.run(data_point[0], stealth: true)
+      expected = data_point[1]
+      # puts "result: \t#{result.map { |x| x.round(5) }}"
+      # puts "expected: \t#{expected}"
+      error_sum = 0.0
+      result.size.times do |i|
+        error_sum += (result[i] - expected[i]).abs
+      end
+      correct += 1 if error_sum < 0.3
+    end
+    puts "Correct answers: (#{correct} / #{test_data.size})"
+    (correct > 10).should eq(true)
+```
+
+
 ## Development
 
 ### Basic Features  
   - [x] Add sgd,minibatch-update.  
   - [x] Add more activation functions.  
   - [x] Add more cost functions.  
   - [x] Add more gradient optimizers
-    - [x] Add iRprop+  
-    - [x] ADAM  
+    - [x] iRprop+  
+    - [x] ADAM
+    - [x] ES (evolutionary strategy, non-backprop)
 
 ### Advanced Features  
   - [x] Convolutional Neural Net.  
+  - [ ] Add support for multiple neuron types.  
   - [ ] Bind and use CUDA (GPU acceleration)  
   - [ ] graphic printout of network architecture.  
   - [ ] Add LSTM.  
     - [ ] RNN.  
   - [ ] GNG (growing neural gas).  
   - [ ] SOM (self organizing maps).  
   - [ ] DBM (deep belief network).  
-  - [ ] Add support for multiple neuron types.