Add ResNet #321

yuyu2172 · 2017-07-06T06:43:21Z

Merge after #265 (DONE)

EDIT:
merge after #427 (DONE)

…o HEAD

yuyu2172 · 2017-09-22T09:22:35Z

I did a quick survey on various architectures that are called ResNet.
There seems to be three.

Original architecture (https://arxiv.org/pdf/1512.03385.pdf)
Facebook's ResNet.

The difference between the original architecture is with "strided convolution" (link).
TensorFlow calls this ResNet v1. They have no implementation for the original architecture in the official repository (?) link

ResNet v2 (also called Pre-ResNet in Torch)

Introduced in https://arxiv.org/pdf/1603.05027.pdf. This architecture uses BNActivConv2D instead of Conv2DBNActiv as a basic building block.
I am not sure if the name "ResNet v2" is common, but it is used by people using Tensorflow and MXNet.
The performance is not always better than the counterparts. https://github.com/tornadomeet/ResNet#resnet-v2-vs-resnet-v1

Names

I think naming the ResNet class as ResNet*** (e.g. ResNet50) is good.

We do not need to support ResNet v2 because it seems to be unpopular compared to the other variants.
We can support both the original architecture and FB ResNet in one class by switching between the two using a variable. The logic is relatively simple.

Hakuyume · 2017-09-23T11:00:58Z

The third model is called Pre-ResNet in torch (https://github.com/facebook/fb.resnet.torch/blob/master/models/preresnet.lua).

yuyu2172 · 2017-09-23T22:23:24Z

Thanks. I added that information to the summary.

yuyu2172 · 2017-10-08T08:12:43Z

@Hakuyume

Can you briefly take a look at resnet.py?
https://github.com/yuyu2172/chainercv/blob/9288eb869a64f6f17aaf8dbf90f4206faf6cfc42/chainercv/links/model/resnet/resnet.py

Note that the uploaded pretrained weight would not work with the current organization of weights.

yuyu2172 · 2018-02-09T06:24:24Z

mode sounds like switching between inference and training.
arch is better.

yuyu2172 · 2018-02-27T03:59:54Z

@Hakuyume
Could you check this?

Hakuyume · 2018-02-27T04:49:24Z

@yuyu2172 OK, I'll check. Please fix the coding style first.

yuyu2172 · 2018-02-27T05:16:53Z

Oh. Sorry about that.

Hakuyume

First comments

Hakuyume · 2018-02-27T05:30:01Z

chainercv/links/model/resnet/resblock.py

+                                       nobias=True)
+            self.conv3 = Conv2DBNActiv(mid_channels, out_channels, 1, 1, 0,
+                                       initialW=initialW, nobias=True,
+                                       activ=lambda x: x)


Calling Caonv2DBActiv without activ looks tricky. How about adding Conv2DBN? (It is ok to make it a private class)

Hmm...
I think the current version is fine, but alternatively I can explicitly use conv and bn.

I do not like the idea of adding a private class. This usage comes up a lot.
I am less hesitant on the idea of adding Conv2DBN, but personally it looks redundant given that this is just a Conv2DBNActiv with no activation....

If you don't like adding a private class, using Conv2D + BatchNormalization is better.
Another solution is adding active='no' option. to Conv2DBNActiv.

OK. I like the second idea.
How about setting the name of the special string to identity?

Since the default value of active is chainer.functions.relu (https://github.com/chainer/chainercv/blob/master/chainercv/links/connection/conv_2d_bn_activ.py#L65), we can use active=None for no activation.

Hakuyume · 2018-02-27T05:39:07Z

chainercv/links/model/resnet/resblock.py

+        stride (int or tuple of ints): Stride of filter application.
+        initialW (4-D array): Initial weight value used in
+            the convolutional layers.
+        conv_shortcut (bool): If :obj:`True`, apply a 1x1 convolution


How about residual_conv? I prefer residual to shortcut. Using both residual and shortcut is confusing and residual is more common.

Hakuyume · 2018-02-27T05:51:11Z

chainercv/links/model/resnet/resnet.py

+
+class ResNet(PickableSequentialChain):
+
+    """Base class for ResNet Network.


ResNet architecture is better because ResNet Network is Residual Network Network.

Hakuyume · 2018-02-27T05:52:19Z

chainercv/links/model/resnet/resnet.py

+
+    """Base class for ResNet Network.
+
+    This is a feature extraction link.


pickable sequential link? We have not defined feature extraction link officially.

Hakuyume · 2018-02-27T05:54:42Z

chainercv/links/model/resnet/resnet.py

+        This is only supported when :obj:`arch=='he'`.
+
+    Args:
+        model_name (str): Name of the resnet model to instantiate.


How about n_layer? It takes one of 50, 101, 152 as integer. model_name is difficult to understand.

Hakuyume · 2018-02-27T05:56:00Z

chainercv/links/model/resnet/resnet.py

+            the mean value used to train the pretrained model is used.
+            Otherwise, the mean value calculated from ILSVRC 2012 dataset
+            is used.
+        initialW (callable): Initializer for the weights.


weights for convolution kernels?

Hakuyume · 2018-02-27T05:57:19Z

docs/source/reference/links.rst

@@ -20,6 +20,7 @@ Feature extraction links extract feature(s) from given images.
 .. toctree::

   links/vgg
+   links/resnet


alphabetical order

yuyu2172 · 2018-03-05T11:18:28Z

@Hakuyume

Hakuyume · 2018-03-05T12:48:11Z

chainercv/links/model/resnet/resblock.py

+    """A bottleneck layer.
+
+    Args:
+        in_channels (int): The number of channels of input arrays.


input arrays -> (the) input array? From my understanding, this link takes only one array.

Hakuyume · 2018-03-05T12:48:28Z

chainercv/links/model/resnet/resblock.py

+    Args:
+        in_channels (int): The number of channels of input arrays.
+        mid_channels (int): The number of channels of intermediate arrays.
+        out_channels (int): The number of channels of output arrays.


Hakuyume · 2018-03-05T12:49:57Z

examples/classification/README.md

+| VGG16 | 27.1 % |   |
+| ResNet50 | 23.0 % | 22.9 % [2] |
+| ResNet101 |21.8 % | 21.8 % [2] |
+| ResNet152 |21.4 % | 21.4 % [2] |


| 21.4 % (space after |).

yuyu2172 · 2018-03-05T13:09:12Z

Thanks.

Hakuyume · 2018-03-05T13:33:11Z

chainercv/links/model/resnet/resblock.py

+        for name in self._forward:
+            l = getattr(self, name)
+            x = l(x)
+        return x


I prefer using PickableSequentialChain to managing _forward manually. Although the pickable feature is not used, it will be more simple.

Hakuyume

LGTM

knorth55 · 2018-05-26T05:17:49Z

@yuyu2172 I have a question. Some repositories like mxnet and https://github.com/KaimingHe/deep-residual-networks, he resnet is nobias=True on conv1.
Which one is correct?

https://github.com/KaimingHe/deep-residual-networks/blob/master/prototxt/ResNet-101-deploy.prototxt#L18
https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/symbols/resnet-v1.py#L123-L124

yuyu2172 force-pushed the resnet-link branch from 8feeb87 to 61cc896 Compare July 6, 2017 06:44

yuyu2172 force-pushed the resnet-link branch from 7283cb9 to 6265260 Compare August 21, 2017 07:48

add resnet

20d85a3

yuyu2172 force-pushed the resnet-link branch from 6265260 to 20d85a3 Compare August 21, 2017 08:08

yuyu2172 added 2 commits September 22, 2017 17:36

merge master

2d36cd6

Merge remote-tracking branch 'yuyu2172/pickable-sequential-chain' int…

ff9ddf0

…o HEAD

update resnet

7b86cea

yuyu2172 force-pushed the resnet-link branch from ed0f51c to 7b86cea Compare September 22, 2017 10:29

yuyu2172 changed the title ~~[WIP] Add ResNet~~ Add ResNet Sep 22, 2017

yuyu2172 added the feature label Sep 22, 2017

yuyu2172 added this to the v0.8 milestone Sep 22, 2017

update doc

8bfca83

yuyu2172 mentioned this pull request Sep 29, 2017

Add ResNet training code #436

Merged

6 tasks

yuyu2172 added 3 commits October 4, 2017 17:32

Merge two bottlenecks

42577c5

update initializer

b55b8bb

Merge remote-tracking branch 'origin/master' into resnet-link

37a1453

yuyu2172 force-pushed the resnet-link branch from 61899eb to 0105440 Compare October 4, 2017 09:07

update doc

d30d37c

yuyu2172 force-pushed the resnet-link branch from 0105440 to d30d37c Compare October 4, 2017 09:10

yuyu2172 added 5 commits October 4, 2017 18:31

use Conv2DBNActiv in the beggining

2858a4b

update convert_resnet

d7b3c35

use fb_resnet=True by default

e2d440b

use conv_shortcut option

54105e6

flake8

9288eb8

change building_block

3be1872

yuyu2172 force-pushed the resnet-link branch from 048e10d to 3be1872 Compare October 8, 2017 09:43

fix doc

c7b10b6

yuyu2172 assigned Hakuyume Dec 19, 2017

yuyu2172 modified the milestones: v0.8, v0.9 Dec 19, 2017

yuyu2172 mentioned this pull request Dec 23, 2017

Relevant changes in Chainer v4 #506

Closed

mode --> arch

a0780ac

dict -> {}

87598a2

Hakuyume reviewed Feb 27, 2018

View reviewed changes

reflect comment

464815e

yuyu2172 mentioned this pull request Mar 1, 2018

Support None option for activ #529

Merged

yuyu2172 added 2 commits March 1, 2018 11:43

Merge branch 'activ-none' into resnet-link

18863d8

use None for activ

ec66cc3

Hakuyume reviewed Mar 5, 2018

View reviewed changes

reflect comments

2b729d7

yuyu2172 added 3 commits March 5, 2018 22:09

Merge remote-tracking branch 'origin/master' into resnet-link

ba9be47

fix doc

1882f5e

fix doc

5c9b8f1

Hakuyume reviewed Mar 5, 2018

View reviewed changes

use PickableSequentialChain for ResBlock

b7d4931

Hakuyume approved these changes Mar 6, 2018

View reviewed changes

Hakuyume merged commit fbe0331 into chainer:master Mar 6, 2018

yuyu2172 deleted the resnet-link branch March 6, 2018 02:11

yuyu2172 mentioned this pull request May 26, 2018

Use bias only for He's ResNet50 #621

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ResNet #321

Add ResNet #321

yuyu2172 commented Jul 6, 2017 •

edited

yuyu2172 commented Sep 22, 2017 •

edited

Hakuyume commented Sep 23, 2017

yuyu2172 commented Sep 23, 2017

yuyu2172 commented Oct 8, 2017

yuyu2172 commented Feb 9, 2018

yuyu2172 commented Feb 27, 2018

Hakuyume commented Feb 27, 2018 •

edited

yuyu2172 commented Feb 27, 2018

Hakuyume left a comment

Hakuyume Feb 27, 2018

yuyu2172 Feb 28, 2018 •

edited

Hakuyume Feb 28, 2018

yuyu2172 Feb 28, 2018

Hakuyume Mar 1, 2018

yuyu2172 Mar 1, 2018

yuyu2172 Mar 1, 2018

Hakuyume Feb 27, 2018

Hakuyume Feb 27, 2018

Hakuyume Feb 27, 2018

Hakuyume Feb 27, 2018

Hakuyume Feb 27, 2018

Hakuyume Feb 27, 2018

yuyu2172 commented Mar 5, 2018

Hakuyume Mar 5, 2018

Hakuyume Mar 5, 2018

Hakuyume Mar 5, 2018

yuyu2172 commented Mar 5, 2018

Hakuyume Mar 5, 2018

Hakuyume left a comment

knorth55 commented May 26, 2018


		class ResNet(PickableSequentialChain):

		"""Base class for ResNet Network.


		"""Base class for ResNet Network.

		This is a feature extraction link.

Add ResNet #321

Add ResNet #321

Conversation

yuyu2172 commented Jul 6, 2017 • edited

yuyu2172 commented Sep 22, 2017 • edited

Names

Hakuyume commented Sep 23, 2017

yuyu2172 commented Sep 23, 2017

yuyu2172 commented Oct 8, 2017

yuyu2172 commented Feb 9, 2018

yuyu2172 commented Feb 27, 2018

Hakuyume commented Feb 27, 2018 • edited

yuyu2172 commented Feb 27, 2018

Hakuyume left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuyu2172 Feb 28, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuyu2172 commented Mar 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuyu2172 commented Mar 5, 2018

Choose a reason for hiding this comment

Hakuyume left a comment

Choose a reason for hiding this comment

knorth55 commented May 26, 2018

yuyu2172 commented Jul 6, 2017 •

edited

yuyu2172 commented Sep 22, 2017 •

edited

Hakuyume commented Feb 27, 2018 •

edited

yuyu2172 Feb 28, 2018 •

edited