SINGA -475 add Sign operator to singa

[pinpom]

No description provided.

[joddiy]

ready for merge

[chrishkchris]

I have run the test in docker container using cloud K80. It passes the tests.
I have left four comments above which are minor issue. It will be ready for merge after these minor changes.

[chrishkchris]

delete the comment line at 625:
#dx = [x/|x|]'

[chrishkchris]

delete the comment line at 644:
#dx = [x/|x|]'

[pinpom]

fix the code

[chrishkchris]

Add the default initializer:

def __init__(self):
    super(Sign, self).__init__() 

[pinpom]

fix the code

[chrishkchris]

Thanks for the modification. The below is the reason why this is needed.

This function passes the name "Sign" to the superclass (Operation) initializer.
The superclass Operation initializer is as follows:

class Operation(object):
    op_count = 0
    def __init__(self, name=None):
        if name is None:
            self.name = "{}#{}".format(
                self.__class__.__name__, Operation.op_count
            )
            Operation.op_count += 1
        else:
            self.name = name

If there is no initialzer in the subclass, the superclass initializer will be used instead with the "name=None"

[chrishkchris]

Thanks! Ready for merge

[nudles]

the dx is always 0?

[chrishkchris]

In my opinion:
For the function y=sign(x) , y = 1 when x > 0, y = -1 when x < 0. Therefore, the derivative is 0 except x = 0.
The function y=sign(x) is discontinuous at x=0 where the derivative is undefined in the mathematical logic. However, an undefined derivative leads to a nan number if we use it for backpropagation, so instead we used a derivative of zero also at x=0.

As a result, dx is always 0 while the output shape (size of array) is the same as input shape.

[chrishkchris]

This is what tensorflow uses: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/cc/gradients/math_grad.cc#L268

Status SignGrad(const Scope& scope, const Operation& op,
                const std::vector<Output>& grad_inputs,
                std::vector<Output>* grad_outputs) {
  auto shape = Shape(scope, op.input(0));
  auto zero = Cast(scope, Const(scope, 0.0), op.input(0).type());
  auto dx = Fill(scope, shape, zero);
  grad_outputs->push_back(dx);
  return scope.status();
}
REGISTER_GRADIENT_OP("Sign", SignGrad);

seems to fill the dx with all zero using the dy shape

[chrishkchris]

there are conflicts and there are two solutions (same as what happened in log operator):

1. merge this branch with the latest master
2. checkout a new branch from the latest master. Then copy the added code into the autograd.py and test_operations.py; send a new PR.

(I guess this maybe because different PRs put the new functions at the same line, so git detected it as different ways of changing code and hence conflicts)