the grad of lars should be scaled in lbsgd

[starimpact]

 776     def _get_lars(self, weight, g, wd):
 777         """Returns a scaling factor for the learning rate for this layer
 778         default is 1
 779         """
 780         weight2 = self._l2norm(weight)
 781         grad2 = self._l2norm(g)

 782         grad2 = grad2*(self.rescale_grad**2)

 783         lars = math.sqrt(weight2 / (grad2 + wd * weight2 + 1e-18))
 784         if lars < 0.01:
 785             lars = 0.01
 786         elif lars > 100:
 787             lars = 100
 788         return lars

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Bug

[frankfliu]

@mxnet-label-bot add [operator, bug]

[abhinavs95]

Hi @starimpact Could you provide some more info like a brief description of the problem with a minimum reproducible example?

[abhinavs95]

@mxnet-label-bot add [Pending Requester Info]

[lanking520]

The user point to a valid location:

python/mxnet/optimizer/optimizer.py

Please track this file for further investigation.

@starimpact could you please bring more information about why this change is necessary?

[anirudhacharya]

@starimpact please try this optimizer https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/optimizer/optimizer.py#L788 and close this issue if your concern is addressed. lbsgd is likely to be deprecated.

[starimpact]

_l2norm is time consuming for a large parameter, so I suggest it should be:

    def _l2norm(self, v):
        "inner product implementation"
        #for big local parameter
        v = v.reshape(-1)
        if len(v) > 100000:
            step = len(v)/100000+1
            v = v[::step]
        norm = multiply(v, v).asnumpy().sum()
        #norm = (multiply(v, v).sum()).asnumpy()
        norm = math.sqrt(norm)
        return norm