Step size for parameter updater #3459
Conversation
There was a problem hiding this comment.
The step size for upload / download is fixed as the same value. We will ultimately need to configure different step size for upload and download. But I think this is fine for this PR.
Could you create a new card in https://github.com/PaddlePaddle/Paddle/projects/18 to track this feature request?
| @@ -167,7 +173,8 @@ def train(self, reader, num_passes=1, event_handler=None, feeding=None): | |||
| batch_id=batch_id, | |||
| cost=cost, | |||
| evaluator=batch_evaluator)) | |||
| self.__parameter_updater__.finishBatch(cost) | |||
| if batch_id >= step_size and batch_id % step_size == 0: | |||
| self.__parameter_updater__.finishBatch(cost) | |||
There was a problem hiding this comment.
This is assuming the gradient is not reset to 0 and always accumulating new gradients across different steps, unless para->getBuf(PARAMETER_GRADIENT)->zeroMem(); is called.
I have checked the implementation inside NeuralNetwork.cpp, which does not reset the gradient across different steps, but each layer have access to the parameters and could potentially reset the gradient.
Just double check, did you verify with @jacquesqiao or @hedaoyuan about the correctness of this implementation?
|
感谢您给PaddlePaddle贡献代码。由于Paddle V1/V2版本已不再维护，相关代码也已从develop分支上删除，因此关闭您的PR，欢迎您向Paddle最新版-Fluid贡献代码。 |
Fixed #3369