How to use batch-normalization? #30
Comments
The easiest way to use them is to provide the There is a lot more win with batch normalization in convolutional layers than fully connected, but you may get win there if your batch size is big enough. Also, it is important to note that your train and test networks will be different when using batch normalization, so you will need to construct one with Also, I don't mind answering questions here, especially if you close them when you are satisfied. |
I've looked at the function Anyway, when I search for its definition using the PyCharm editor, I end up in the file
There's not much documentation in this file so it isn't clear how to use this, what the parameters mean and which values might be appropriate. Could I ask you to modify the code I wrote above to give an example of how to do this? Thanks. |
The parameters map directly to the batch_normalization method. It is norm = pt.,BatchNormalizationArguments(scale_after_normalization=True)
with pt.defaults_scope(activation_fn=tf.nn.relu, phase=pt.Phase.train):
y_pred, loss = x_pretty.\
conv2d(kernel=5, depth=64, name='layer_conv1', batch_normalize=norm).\
max_pool(kernel=2, stride=2).\
conv2d(kernel=5, depth=64, name='layer_conv2', batch_normalize=norm).\
max_pool(kernel=2, stride=2).\
flatten().\
fully_connected(size=256, name='layer_fc1', batch_normalize=norm). \
fully_connected(size=128, name='layer_fc2').\
softmax_classifier(class_count=10, labels=y_true) On Wed, Aug 17, 2016 at 1:10 AM, Hvass-Labs notifications@github.com
|
Thanks for the example. However, I get an error using your sample code. First I write the following which you suggest:
And in the code for defining the network I have the following line for creating the first conv-layer, as you suggest:
But this causes the following exception:
which is raised in line 1981 in
What I do instead is that I use I've read the following doc which you suggested, but it really doesn't explain much: https://github.com/google/prettytensor/blob/master/docs/PrettyTensor.md#batch_normalize The docs also don't make clear what is the difference in using different phases in Pretty Tensor. When I look at the docs for e.g. Once again I'd like to encourage you to significantly improve the documentation because it is frustrating to try and learn how to use Pretty Tensor from reading the current docs. Scikit-learn has very good documentation which could serve as inspiration. But I read in the TensorFlow forum that the dev-team is currently consolidating the builder-API's, so perhaps you have different plans going forward? |
Thanks for the bug, I found the problem and will provide a fix. Pretty Tensor is supported and alive, but it is a rather small effort now and so fixes take time. The larger effort is tf.contrib.learn and it has some tutorials here: https://www.tensorflow.org/versions/r0.9/tutorials/index.html These can all be mixed and matches and any functions that you like and are missing can be added (the simplest way would be by doing |
Once again I hope it's OK that I ask this question here instead of on StackOverflow.
I don't know if batch-normalization is really useful, there seems to be different opinions on the matter. But I'd like to try it. I can see that it's implemented in Pretty Tensor:
https://github.com/google/prettytensor/blob/master/docs/PrettyTensor.md#batch_normalize
But I can't figure out how to use it for the following Convolutional Neural Network:
Any help would be appreciated.
The text was updated successfully, but these errors were encountered: