New style relu #3175

delta2323 · 2017-08-18T04:31:08Z

This PR implements new-style version of F.relu.

This PR depends on #3096, and is a part of #3147.

beam2d

Added some comments.

beam2d · 2017-08-18T04:34:28Z

chainer/functions/activation/relu.py

        self.retain_outputs((0,))
+        self._use_cudnn = False


I think it's simpler to make _use_cudnn = False a class attribute and only sets explicitly on cuDNN path.
I mean:

class ReLU(function_node.FunctionNode): _use_cudnn = False ... def forward_gpu(self, x): if ...: self._use_cudnn = True ... ...

beam2d · 2017-08-18T04:36:34Z

chainer/functions/activation/relu.py

+            return ReLUGrad2().apply((y, gy[0]))
+
+
+class Zero(function_node.FunctionNode):


Is it needed?

beam2d · 2017-08-18T04:37:30Z

chainer/functions/activation/relu.py

+        return Zero().apply(gy)
+
+
+class Heaviside(function_node.FunctionNode):


The following code simplifies it.

def heaviside(x): return utils.force_array((x.data > 0).astype(x.dtype))

beam2d

Thank you. I added some more comments.

beam2d · 2017-08-18T04:57:37Z

chainer/functions/activation/relu.py

+    def backward(self, indexes, gy):
+        ret = []
+        if 0 in indexes:
+            ret.append(None)


Could you remove the first argument from inputs and instead pass it as an argument of __init__? (It will simplifies the backprop, which is good for performance)

beam2d · 2017-08-18T04:57:50Z

chainer/functions/activation/relu.py

+    def backward(self, indexes, gy):
+        ret = []
+        if 0 in indexes:
+            ret.append(None)


beam2d · 2017-08-18T04:57:55Z

chainer/functions/activation/relu.py

+        if 0 in indexes:
+            ret.append(None)
+        if 1 in indexes:
+            ret.append(None)


beam2d · 2017-08-18T04:58:19Z

chainer/functions/activation/relu.py

@@ -12,48 +12,103 @@
    _mode = cudnn.cudnn.CUDNN_ACTIVATION_RELU


-class ReLU(function.Function):
+class ReLU(function_node.FunctionNode):

    """Rectified Linear Unit."""
    # TODO(beam2d): Implement in-place version.


It is not directly releated to this PR, but I found this TODO comment is obsolete. Can you remove it?

delta2323 · 2017-08-18T07:35:01Z

As #3096 is merged to the master branch, I rebased the PR.

…relu-2

delta2323 · 2017-08-21T00:54:05Z

Thank you for your comments. I updated the PR. Note that although I wrote the docstrings of ReLUGrad2 and ReLUGrad3, I do not intend to make them official APIs.

beam2d

I added some more comments.

Note: I noted that "you do not need to check if indexes is empty", but I found that Variable.backward() does not check it correctly. I'll fix this point in another PR, so it's ok to proceed with removing the check.

beam2d · 2017-08-21T01:16:15Z

chainer/functions/activation/relu.py

        return gx,

+    def backward(self, indexes, gy):
+        if 0 in indexes:


You do not need to check this; indexes is always non-empty (otherwise backward is not called).

OK. I have removed it.

beam2d · 2017-08-21T01:16:21Z

chainer/functions/activation/relu.py

+        return cudnn.activation_backward(a, b, inputs[0], _mode),
+
+    def backward(self, indexes, gy):
+        if 0 in indexes:


Remove the check (see the above comment)

beam2d · 2017-08-21T01:22:07Z

chainer/functions/activation/relu.py

+
+
+def _heaviside(x):
+    return utils.force_array((x > 0).astype(x.dtype))


Is this force_array needed?

beam2d · 2017-08-21T01:22:18Z

chainer/functions/activation/relu.py

+    def backward(self, indexes, gy):
+        if 0 in indexes:
+            xp = cuda.get_array_module(gy[0])
+            b = xp.asarray(self.b)


Is this asarray needed?

delta2323 · 2017-08-21T01:59:37Z

Thank you. Updated.

beam2d · 2017-08-23T01:39:13Z

Please resolve the flake8 errors.

beam2d · 2017-08-23T04:44:47Z

Jenkins, test this please

beam2d

One more comment

beam2d · 2017-08-23T08:09:47Z

chainer/functions/activation/relu.py

+    """
+
+    def __init__(self, b):
+        super(ReLUGrad2).__init__()


, self is missing (or remove this line; it is allowed to not call super init in FunctionNode)

beam2d · 2017-08-24T00:05:09Z

LGTM

beam2d · 2017-08-24T00:05:20Z

Thank you!

delta2323 added 5 commits August 17, 2017 14:52

Apply new-style API to ReLU

f777e1e

Pass existing unit tests

b5e20bc

Pass existing GPU tests

6fcdfad

Fix relu to pass unit tests

31f9acc

Add more unit tests for F.relu

7666ff6

beam2d requested changes Aug 18, 2017

View reviewed changes

delta2323 added 3 commits August 18, 2017 13:40

Refactor

fcd551f

make _use_cudnn class variable

06b805d

Simplify backward for efficient backprop

184570f

beam2d requested changes Aug 18, 2017

View reviewed changes

Merge branch 'master' into new-style-relu-2

1cdf7e8

delta2323 added 4 commits August 21, 2017 09:22

Remove an obsolete TOOD comemnt

2d7b6e2

Trim unneeded backprops for efficient computation

d24079f

Add documents to ReLUGrad2 and ReLUGrad3

45e79b8

Merge remote-tracking branch 'delta/new-style-relu-2' into new-style-…

570264f

…relu-2

beam2d requested changes Aug 21, 2017

View reviewed changes

delta2323 added 3 commits August 21, 2017 10:55

Assume that indexes are always non-empty

c0c2b5b

force_array is not needed

22549b1

No need of the conversion of arrays with asarray

5e161fe

beam2d mentioned this pull request Aug 21, 2017

Double backward support for v3 #3147

Closed

27 tasks

niboshi added the cat:feature Implementation that introduces new interfaces. label Aug 22, 2017

Fix flake8

5ee9fd5

beam2d requested changes Aug 23, 2017

View reviewed changes

Fix ReLUGrad2 and ReLUGrad3

0b55b0d

beam2d approved these changes Aug 24, 2017

View reviewed changes

beam2d merged commit c8c9aec into chainer:master Aug 24, 2017

beam2d added this to the v3.0.0rc1 milestone Aug 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New style relu #3175

New style relu #3175

delta2323 commented Aug 18, 2017

beam2d left a comment

beam2d Aug 18, 2017

beam2d Aug 18, 2017

beam2d Aug 18, 2017

beam2d left a comment

beam2d Aug 18, 2017

beam2d Aug 18, 2017

beam2d Aug 18, 2017

beam2d Aug 18, 2017

delta2323 commented Aug 18, 2017 •

edited

Loading

delta2323 commented Aug 21, 2017

beam2d left a comment

beam2d Aug 21, 2017

delta2323 Aug 21, 2017

beam2d Aug 21, 2017

beam2d Aug 21, 2017

beam2d Aug 21, 2017

delta2323 commented Aug 21, 2017

beam2d commented Aug 23, 2017

beam2d commented Aug 23, 2017

beam2d left a comment

beam2d Aug 23, 2017

beam2d commented Aug 24, 2017

beam2d commented Aug 24, 2017

		return ReLUGrad2().apply((y, gy[0]))


		class Zero(function_node.FunctionNode):

		return Zero().apply(gy)


		class Heaviside(function_node.FunctionNode):



		def _heaviside(x):
		return utils.force_array((x > 0).astype(x.dtype))

New style relu #3175

New style relu #3175

Conversation

delta2323 commented Aug 18, 2017

beam2d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beam2d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 commented Aug 18, 2017 • edited Loading

delta2323 commented Aug 21, 2017

beam2d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 commented Aug 21, 2017

beam2d commented Aug 23, 2017

beam2d commented Aug 23, 2017

beam2d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beam2d commented Aug 24, 2017

beam2d commented Aug 24, 2017

delta2323 commented Aug 18, 2017 •

edited

Loading