2.0 planning

[robertleeplummerjr]

The work for 2.0 is really taking root. After getting recurrent neural networks in v1 (not yet released) I could see a great deal how we could make it simpler, but I also wanted to address the entire architecture as a whole. We can simplify the strategies of a recurrent neural network and a feedforward neural network, so that more advanced concepts, like convolution could easily be used. Also I really want to make the api's for both feedforward and recurrent to be as close as possible, so you could simply change out the underlying network type, and recurrent is what you'd have. After a lot of careful consideration, this is what I've got:

Nets will have at least two means of composition, imperative and functional. They can be mixed.

This is an imperative defined feedforward convolutional neural network:

new brain.FeedForward({
  inputLayer: () => input({ width: 24, height: 24 }),
  hiddenLayers: [
    (input) => convolution({ filterCount: 8, filterWidth: 5, filterHeight: 5, padding: 2, stride: 1 }, input),
    (input) => relu(input),
    (input) => pool({ padding: 2, stride: 2 }, input),
    (input) => convolution({ padding: 2, stride: 1, filterCount: 16, filterWidth: 5, filterHeight: 5 }, input),
    (input) => relu(input),
    (input) => pool({ width: 3, stride: 3 }, input),
    (input) => softMax({ width: 10 }, input)
  ],
  outputLayer: (input) => output({ width: 10 }, input)
});

This is a functional defined feedforward convolutional neural network:

new brain.FeedForward({
  inputLayer: () => input({ width: 24, height: 24 }),
  hiddenLayers: [
    (input) =>
      softMax({ width: 10 },
        pool({ width: 3, stride: 3 },
          relu(
            convolution({ padding: 2, stride: 1, filterCount: 16, filterWidth: 5, filterHeight: 5 },
              pool({ padding: 2, stride: 2 },
                relu(
                  convolution({ filterCount: 8, filterWidth: 5, filterHeight: 5, padding: 2, stride: 1 },
                    input
                  )
                )
              )
            )
          )
        )
      )
  ],
  outputLayer: (input) => output({ width: 10 }, input)
});

Both of these create the exact same type of network, the reason for the two strategies is that I really like to essentially not have to learn a language, and the functional approach matches up to math nearly one to one. The imperative version too, sometimes you just don't want to think in reverse, and lists are nice. For recurrent nets I think the functional will shine more as you start the mind-bending process of recursion. I'm open to suggestion!

Later we can apply the same composition to do:

const net = new brain.Recurrent({ inputLayer, hiddenLayers, outputLayer });

I'm sure there will be more, but this is a start.

[robertleeplummerjr]

Json input and output is currently tested and is:
toJSON():

net.toJSON();

note: no change, see next for json definition.

static fromJSON(json, function):

const net = FeedForward.fromJSON({
        layers: [
          {
            type: 'TestLayer',
            foo: true
          },
          {
            type: 'TestLayer',
            foo: true,
            inputLayerIndex: 0
          },
          {
            type: 'TestLayer',
            foo: true,
            inputLayerIndex: 1
          },
          {
            type: 'TestLayer',
            foo: true,
            inputLayerIndex: 2
          }
        ]
      }, (jsonLayer, input) => {
        switch (jsonLayer.type) {
          case 'TestLayer':
            return new TestLayer(jsonLayer, input);
          default:
            throw new Error(`unknown layer ${ jsonLayer.type }`);
        }
      });

Note: inputLayerIndex isn't terribly useful here, but will be when recurrent happens. The definition is FeedForward.fromJSON(json, (jsonLayer, input) => { return new ALayer(jsonLayer, input); });

The idea is that we can accept any layer, your own are fine, if you like.

[robertleeplummerjr]

Just a quick note. Recurrent neural networks proposal:

new brain.Recurrent({
  inputLayer:
  hiddenLayers: [],
  outputLayer: 
});

Because a LSTM neural network is essentially architecturally the same as an RNN, but with different hidden layers, we can have a layer be the definition for the LSTM. All the math is encapsulated. Easy to understand, and easy to extend. Picture the whole underlying method like this:

//this will be part of our lib
brain.LSTM = class LSTM extends brain.Recurrent {}
LSTM.defaults = {
  brain.Recurrent.defaults...,
  hiddenLayers: layer.lstm
};

//this will be outside of lib, in real world:
new brain.LSTM();

[robertleeplummerjr]

Also: the layer is in charge of how the layer learns, period. This sounds vague and probably a bit weird, but picture the forward propagate (predict) and back propagate (compare, and learn) as managed inside the layer so all the math that affects the layer is in one location. The order in which layers predict, compare, and learn, is only managed inside the lib. So we get this super nice and clean means of composing a layer:

import Base from 'brain/layer/base';
import makeKernel from 'brain/utilities/make-kernel';
class MyLayer extends Base {
  //attach to GPU
  createKernels() {
    this.predictKernel = makeKernel({ size: [this.width, this.height, this.depth] }, predict);
    this.compareKernel = makeKernel({ size: [this.width, this.height, this.depth] }, compare);
    this.learnKernel = makeKernel({ size: [this.width, this.height, this.depth] }, learn);
  }

  //attach GPU to our class, and special operations of class
  predict() {
    this.outputs = this.predictKernel(this.weights, this.inputLayer.outputs);
  }
  compare() {
    this.biases = this.compareKernel(...);
  }
  learn() {
    this.deltas = this.learnKernel(...);
  }
}

//math here
function predict(weights, inputs) {
  return weights[this.thread.z][this.thread.y][this.thread.x] * inputs[this.thread.z][this.thread.y][this.thread.x];
}
function compare() {}
function learn() {}

This runs on gpu, node, and js.

[robertleeplummerjr]

So here is the latest in trends for LSTM and recurrent behavior. We need to define a single set of programs that can be reused, and a simplified set of terminologies that describe them.

Here is the commit: d43b1ee#diff-b84ad68a55b45bd31698c5356097abf3R15

The idea behind it:

We need a way to define groups of layers that compose a single layer. LSTM, for example, is very simple in that it uses the same technology as RNN but it is a bit bigger. If we describe each step in the LSTM as a "layer" it becomes very hairy. However, if we describe each mathematical operation as part of the layer, the "layer" is actually composed of many "layers". But don't think about that. Just think of that if we want a complex layer, it can have multiple mathematical operations. To simplify (ha!) the understanding of this, we have the concept of "group". I'll try my best to describe it:

export default class SuperLayer extends Group {
  constructor(settings) {
    super(settings);
    //setup layer here
  }

  static kernel(settings) {
    return (layer, inputLayer, previousOutputs) => {
      //layer passed in above ^
      //tons of layers and math
      return result;
    };
  }
}

So the concept here is that the kernel gets created once, however the layer can be composed any number of times. The recurrent behavior stacks the layers, the layers can use add, multiply, sigmoid, relu, etc. inside the kernel, and then out comes the result. So we have one program to manage the operations.

Feedback welcomed!

[robertleeplummerjr]

Now that GPU.js is released as v1.0.0, we are full on! For reference: https://gist.github.com/robertleeplummerjr/4aaf8afb177c9c80f8452d5025117e26