Maybe wrong when computing batch_normalization?

[lfwin]

Hi,
in function batch_normalization, there has a line which update the mean and var or not according to in training or evaluating mode, but seems like both return the same variables.
... mean, var = tf.python.control_flow_ops.cond( is_training, update_mean_var, lambda: (ema_mean, ema_var)) ...

[aymericdamien]

Hi,

By returning same variables, do you mean same value? or is it another issue?
This operation returns a single Tensor, whatever "is_training" value. But during computation, the calculation graph will be altered according to "is_training" value, so that if is_training is True, the first operation (mean/variance update) will be performed, or the second one otherwise.

If you are using that operation without TFLearn trainer class, you need to specify a training mode:

# To set training mode:
tflearn.is_training(True)
# To remove training mode:
tflearn.is_training(False)

(There is little typo in the doc hat can be confusing; this operation actually 'set' the training mode, and not 'check' it. I will update it)

[lfwin]

Hi,
During training, function update_mean_var seems just not update the mean/variance and just return a identity of it.
` def update_mean_var():
with tf.control_dependencies([ema_apply_op]):
return tf.identity(ema_mean), tf.identity(ema_var)

    is_training = tflearn.get_training_mode()
    mean, var = tf.python.control_flow_ops.cond(
        is_training, update_mean_var, lambda: (ema_mean, ema_var))`

[aymericdamien]

tf.control_dependencies is an operation used to ensure that ema_apply_op is calculated before return tf.identity(ema_mean), tf.identity(ema_var), so the ema_apply_op is actually computed (updating the mean/variance).

[lfwin]

Hi,
sorry, I am not correct, confused by the use of the class tf.train.ExponentialMovingAverage.
Another question, where does this op 'ema_apply_op' runs during training?

[aymericdamien]

ema_apply_op runs at training time only, everytime a data batch 'enter' a batch_normalization layer, it will be applied (So the mean/variance is updated at every batch).

[lfwin]

I find a example code use the very similar code in tensorflow/tensorflow#1724 as in batch_normalization:
`import tensorflow as tf
import numpy as np

x = tf.Variable(1.)
update_x = tf.assign_add(x, 1.0)

do_update = tf.placeholder(tf.bool)

ema = tf.train.ExponentialMovingAverage(.9)
ema_assign = ema.apply([x])

avg_without_update = ema.average(x)

with tf.control_dependencies([ema_assign]):
avg_with_update = tf.identity(avg_without_update)

avg = tf.python.control_flow_ops.cond(do_update, lambda: avg_with_update, lambda: avg_without_update)

with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(sess.run([update_x, avg], {do_update: False}))
print(sess.run([update_x, avg], {do_update: False}))
print(sess.run([update_x, avg], {do_update: False}))
print(sess.run([update_x, avg], {do_update: False}))
print(sess.run([update_x, avg], {do_update: False}))`

here do_update is false, but the average is updated, could you tell me where goes wrong?

[aymericdamien]

It seems there is a bug here. Referring to the issue you sent, it seems that conditions (fn1 & fn2) dependencies are always executed, no matter the condition of tf.cond(), thus the result... I can try to see if there is a workaround, or wait for TensorFlow to patch it.