Wide&Deep neural net: Multiple inputs and mixed neural net architecture using PyBrain

[franciscogmm]

I'm currently working with Pybrain. It's wonderful as it allows me to use a feedforward neural net without the backpropagation function. This enables me to evolve the neural network's weights using genetic algorithms.

Just so happens that I've hit a wall when it comes to the library's capabilities, and given that the devs have stopped working on it, I'm not sure how to move forward.

I'm trying to implement the wide and deep neural net (c/o Tensorflow) on PyBrain. I've read the tutorial and get the gist of it. However, PyBrain doesn't seem to allow this, despite having the modules and functions to build the architecture.

brain = FeedForwardNetwork()

inputLayer = LinearLayer(inputNodes, name = 'input')
hiddenLayer1 = ReluLayer(hiddenNodes[0], name='hidden1') #deep layer 1
hiddenLayer2 = ReluLayer(hiddenNodes[1], name='hidden2') #deep layer 2
hiddenLayer3 = TanhLayer(hiddenNodes[2], name='hidden3')
outLayer = SoftmaxLayer(outputNodes, name='output')

brain.addInputModule(inputLayer)
brain.addModule(hiddenLayer1)
brain.addModule(hiddenLayer2)
brain.addModule(hiddenLayer3)
brain.addOutputModule(outLayer)

''' connect layers 
wide_to_hidden3 connects crossed columns that I made to the tanh layer
deep1_to_hidden1 and deep2_to_hidden1 connects the base columns to the deep layers
'''
wide_to_hidden3 = Connection(inputLayer, hiddenLayer3, inSliceFrom=90, inSliceTo=714)
deep1_to_hidden1 = Connection(inputLayer, hiddenLayer1, inSliceFrom= 0, inSliceTo= 89)
deep2_to_hidden1 = Connection(inputLayer, hiddenLayer1, inSliceFrom=715)
hidden1_to_hidden2 = FullConnection(hiddenLayer1, hiddenLayer2)
hidden2_to_hidden3 = FullConnection(hiddenLayer2, hiddenLayer3)
hidden_to_out = FullConnection(hiddenLayer3, outLayer)

brain.addConnection(wide_to_hidden3)
brain.addConnection(deep1_to_hidden1)
brain.addConnection(deep2_to_hidden1)
brain.addConnection(hidden1_to_hidden2)
brain.addConnection(hidden2_to_hidden3)
brain.addConnection(hidden_to_out)

brain.sortModules()

Now, this is the error that I'm getting, which makes me think that this might not be possible on PyBrain as the developers have not anticipated a wide & deep neural net.

Traceback (most recent call last):
File "main.py", line 90, in <module>
actual_train, prediction_train, record, confidence_level, learning_rate, top_mode, int(sys.argv[2]))
File "/Volumes/200GB/Shoptaki/FX Prediction/v4/signalbot/bot/geneticalgorithm.py", line 159, in optimize
bot.update(slice_row, dates_train, actual_train, prediction_train, confidence_level, top_mode, verb)
File "/Volumes/200GB/Shoptaki/FX Prediction/v4/signalbot/bot/signalbot.py", line 337, in update
self.train(slice_row, dates_train, actual_train, prediction_train, 'train', top_mode, confidence_level, verb)
File "/Volumes/200GB/Shoptaki/FX Prediction/v4/signalbot/bot/signalbot.py", line 166, in train
action, signal_strength = self.think(step, actual_train, prediction_train, mode, top_mode)
File "/Volumes/200GB/Shoptaki/FX Prediction/v4/signalbot/bot/signalbot.py", line 114, in think
selectActions = self.brain.activate(stepInput)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/structure/networks/feedforward.py", line 20, in activate
return super(FeedForwardNetworkComponent, self).activate(inpt)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/structure/modules/module.py", line 106, in activate
self.forward()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/structure/modules/module.py", line 73, in forward
self.outputbuffer[self.offset])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/structure/networks/feedforward.py", line 33, in _forwardImplementation
c.forward()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/structure/connections/connection.py", line 77, in forward
self.outmod.inputbuffer[outmodOffset, self.outSliceFrom:self.outSliceTo])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/structure/connections/connection.py", line 96, in _forwardImplementation
abstractMethod()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pybrain/utilities.py", line 32, in abstractMethod
raise NotImplementedError('Method not implemented!')
NotImplementedError: Method not implemented!

However, the fact that they put the parameters inSliceFrom and inSliceTo in the Connection module suggests otherwise. Also, the tutorials hint at

possibilities of nesting networks within
others, using weight-sharing, and more exotic types of networks, connections
and modules

and

But it is possible to create very
sophisticated architectures with PyBrain, and it is also one of the library's
strength to do so.

I love PyBrain because of its modular nature, and I don't want to give up on this. I know Keras has the same modularity, but I don't think it comes with a feed forward net that doesn't have to be trained using back-propagation.

Any help re getting past this error would be helpful. Thanks!!

[Nitrooo]

Have you found a solution somehow? I'm looking for a library for evolving deep neural networks with genetic algorithms and PyBrain would have been a good choice, but it's a dead project, so I'm looking for a valid alternative. Any suggestion?