Skip to content

niangxiu/linGrad

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

51 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build Status License: GPL v3

linGrad

Initial stepsize (learning rate) in gradient descent selected by linear range.

Linear range is the range of parameter perturbations which lead to approximately linear perturbations in the states of a network. Linear range is computed from the difference between actual perturbations in states and the tangent solution. In linGrad, the optimal initial stepsize is such that parameter changes on all minibatches are within linear range.

For detailed explanations check the accompanying paper: https://arxiv.org/abs/1905.04561.

About

gradient descent by effective range method

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages