The hyper parameters of paddle.optimizer does not work in v2 API.

[qingqing01]

The hyper parameters in the paddle.optimizer does not work in v2 API. For example, using momentum optimizer in the sentiment demo as follows,

    optimizer = paddle.optimizer.Momentum(
        learning_rate=2e-3,
        momentum=0.9,
        gradient_clipping_threshold=25.0,
        regularization=paddle.optimizer.L2Regularization(rate=8e-4),
        model_average=paddle.optimizer.ModelAverage(average_window=0.5))

Then print the proto-string of config before this line in python/paddle/v2/trainer.py, it can be found that the proto-string of parameters does not contain the hyper parameters, such as L2 regularization and momentum. The momentum is 0 if you print it before this line in paddle/parameter/FirstOrderOptimizer.h. The proto-string of parameters are as follows,

parameters {
  name: "___embedding_layer_0__.w0"
  size: 658816
  initial_mean: 0.0
  initial_std: 0.0139387206988
  dims: 5147
  dims: 128
  initial_strategy: 0
  initial_smart: true
}
parameters {
  name: "___sequence_conv_pool_0___conv_fc.w0"
  size: 49152
  initial_mean: 0.0
  initial_std: 0.051031036308
  dims: 384
  dims: 128
  initial_strategy: 0
  initial_smart: true
}
....

But the correct proto-string of parameters should contain decay_rate and momentum, as follows,

parameters {
  name: "___embedding_0__.w0"
  size: 3840000
  momentum: 0.9
  initial_mean: 0.0
  initial_std: 0.0057735026919
  decay_rate: 0.0008
  dims: 30000
  dims: 128
  initial_strategy: 0
  initial_smart: true
  gradient_clipping_threshold: 25.0
}
parameters {
  name: "___fc_layer_0__.w0"
  size: 65536
  momentum: 0.9
  initial_mean: 0.0
  initial_std: 0.0883883476483
  decay_rate: 0.0008
  dims: 128
  dims: 512
  initial_strategy: 0
  initial_smart: true
  gradient_clipping_threshold: 25.0
}
...

[reyoung]

This bug is hard to fix. Because we split model configuration into two parts, the topology configuration, and the optimizer settings. When we config and parse topology, there is no optimizer information we set. However, the weight_decay belongs to topology now.

Here is a step by step solution to this issue.

1. Disable weight_decay in optimizer settings. If a user wants a global weight_decay, he can maintain a ParamAttr by himself.

2. Make weight_decay out of Parameter Configuration or set the global weight_decay in topology configuration(e.g. ModelConfig) itself.

The second step is a little bit hard to implement, may change the C++ Core of Paddle.

[qingqing01]

不仅仅是weight_decay的问题，还包括momentum, gradient_clipping_threshold。

[reyoung]

The solution might be as following.

The global weight_decay, momentum, gradient_clipping_threshold should be saved into proto::OptimizationConfig which saved inside TrainerConfig.proto, just like learning_rate in OptimizationConfig.

And then we can get the global weight_decay in all optimizers.

[lcy-seso]

This problem has been fixed by this PR #2288.
Thank you for the issue.

[lcy-seso]

This problem is not solved yet, so I reopen it.

[qingqing01]

close since the v2 API has fixed this issue.