A bit puzzled about the results of using this FFNN

[vittoriom]

Hi,
I finally tried your FFNN (since until now I just lurked reading the code and the future developments), and I stumbled across something I don't fully understand:

I'm trying to train a FFNN with 784 inputs and 10 outputs, with 100 hidden nodes. Momentum is 0.5 and learning rate 0.7, error function Default and activation Sigmoid.

This is the output of the first iteration of the training (first input/output), I have 50000 items in my training set but all the items give the same result (so imagine this output pasted 50000 times).
The training ends immediately since the error is "0" and every test I make with a validation set "succeeds" with output [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] (that of course is wrong).

Hidden output cache [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
Output errors cache [-0.0, -0.0, -0.0, -0.0, -0.0, 0.0, -0.0, -0.0, -0.0, -0.0]
Hidden errors cache [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Error 0.0
Iteration 0 got [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] with error 0.0 (expected output [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0]

Do you have an idea of why this happens?

[vittoriom]

Forget about it, just noob initialization of the weights.

[collinhundley]

Glad you got it figured out. Initializing the weights as nil will automatically assign random weights, although it currently only works well for Sigmoid activation.

It sounds like you're using the MNIST training set. I've had good results training FFNN with that set, and I'm preparing an example for it that will be finished soon. Check out the iOS example project - I update it daily and it includes some neat features. The handwriting recognition part is almost done.

[vittoriom]

Ah, that's kind of good to know (that means we're working on the same thing at the same time, so not so good :D ), interested to see what will come out! Will check the iOS example project in a few days

[collinhundley]

I'm only writing it for iOS right now. If you're writing an OS X version, that will be great to have. I'll move the MNIST data to the Examples directory so it's accessible for both apps (so we don't package two copies).

I'm adding Softmax activation to FFNN right now. It will be useful for handwriting recognition, since it helps with classification problems.