Skip dropout rate error

[jvwilliams23]

Hi,

I have noticed a potential issue in the following code:

keras2cpp/keras_model.cc

Lines 431 to 433 in ce407cc

	} else if(layer_type == "Dropout") {
	continue; // we dont need dropout layer in prediciton mode
	}

Are you sure that we do not need to include dropout layer in prediction mode? In Figure 2 of Srivastava et al. (2014), they say that in training, the weights are randomly set to 0 with probably equal to the dropout rate. In prediction mode, the dropout rate is still there but is simply multiplied to all weights in the layer - which disagrees with the code.

Additionally, I have noticed major differences in my python keras models vs keras2cpp models with dropout when using the default keras_model.cc. Then, when the weights are multiplied by dropout rate, the error goes away.

Reference
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), pp.1929-1958.

[pplonski]

@jvwilliams23 when I was testing the code the predictions were exactly the same as in keras. I dont think that dropout is used in prediction.

[jvwilliams23]

@pplonski Interesting, I will look more into my code. Were you testing using the mnist example?

[pplonski]

Yes, with mnist data.

[jvwilliams23]

Hi @pplonski
I just got around to looking into this further. It seems keras do not use dropout in prediction (https://github.com/keras-team/keras/blob/dc95ceca57cbfada596a10a72f0cb30e1f2ed53b/keras/layers/core.py#L116). I guess this is consistent then, but strange that it goes against the original paper. I will close this issue.