Problem of BatchNorm in Fluid.

[qingqing01]

Now the moving mean and variance in batch_norm are created as parameters and set trainable False:

https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/fluid/layers/nn.py#L1512

   mean = helper.create_parameter(
        attr=ParamAttr(
            name=moving_mean_name, initializer=Constant(0.0), trainable=False),
        shape=param_shape,
        dtype=input.dtype)
    mean.stop_gradient = True

    variance = helper.create_parameter(
        attr=ParamAttr(
            name=moving_variance_name,
            initializer=Constant(1.0),
            trainable=False),
        shape=param_shape,
        dtype=input.dtype)
    variance.stop_gradient = True

But when see the ProgrameDesc proto string, there is still some calculation operators related to the moving mean and variance. For example, I print the ProgrameDesc in MobileNet-SSD of one GPU, the following proto string is about the moving variance(batch_norm_x.w_2 is the moving variance):

  ops {
    inputs {
      parameter: "X"
      arguments: "batch_norm_2.w_2"
    }
    outputs {
      parameter: "Out"
      arguments: "_generated_var_52"
    }
    type: "scale"
    attrs {
      name: "scale"
      type: FLOAT
      f: 4.99999987369e-05
    }
  }
  ops {
    inputs {
      parameter: "X"
      arguments: "batch_norm_2.w_2@GRAD"
    }
    inputs {
      parameter: "Y"
      arguments: "_generated_var_52"
    }
    outputs {
      parameter: "Out"
      arguments: "batch_norm_2.w_2@GRAD"
    }
    type: "elementwise_add"
    attrs {
      name: "axis"
      type: INT
      i: -1
    }
  }

[jacquesqiao]

can you paste the desc info of batch_norm_op as well as these elementwise_add ops?

[qingqing01]

The above ProgrameDesc is about the regularization since I set L2 regularization 0.00005. Although, the un-trainable parameters skip the parameter updating process in https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/fluid/optimizer.py#L202 . But there is no need to append these ops in ProgrameDesc.

[qingqing01]

@jacquesqiao the desc for batch_norm_op:

 ops {
    inputs {
      parameter: "Bias"
      arguments: "batch_norm_2.b_0"
    }
    inputs {
      parameter: "Mean"
      arguments: "batch_norm_2.w_1"
    }
    inputs {
      parameter: "Scale"
      arguments: "batch_norm_2.w_0"
    }
    inputs {
      parameter: "Variance"
      arguments: "batch_norm_2.w_2"
    }
    inputs {
      parameter: "X"
      arguments: "conv2d_1.tmp_0"
    }
    outputs {
      parameter: "MeanOut"
      arguments: "batch_norm_2.w_1"
    }
    outputs {
      parameter: "SavedMean"
      arguments: "batch_norm_2.tmp_0"
    }
    outputs {
      parameter: "SavedVariance"
      arguments: "batch_norm_2.tmp_1"
    }
    outputs {
      parameter: "VarianceOut"
      arguments: "batch_norm_2.w_2"
    }
    outputs {
      parameter: "Y"
      arguments: "batch_norm_2.tmp_2"
    }
    type: "batch_norm"
    attrs {
      name: "data_layout"
      type: STRING
      s: "NCHW"
    }
    attrs {
      name: "epsilon"
      type: FLOAT
      f: 9.99999974738e-06
    }
    attrs {
      name: "momentum"
      type: FLOAT
      f: 0.899999976158
    }
    attrs {
      name: "is_test"
      type: BOOLEAN
      b: false
    }
  }

The above descs for scale_op and elementwise_add_op I pasted are added by transpiler, not in the model config.