Build R-CNN with ResNet #27

JihongJu · 2017-06-09T22:54:24Z

Build an RCNN model with keras_resnet.ResNet50.
Implement heads described in Mask R-CNN.

My current implementation is:

        y = keras_resnet.ResNet50(inputs)
        features = y.layers[-2].output

        rpn_classification = keras.layers.Conv2D(
            9 * 1, (1, 1), activation="sigmoid")(features)
        rpn_regression = keras.layers.Conv2D(9 * 4, (1, 1))(features)

        rpn_prediction = keras.layers.concatenate(
            [rpn_classification, rpn_regression])

        # proposals of shape (1, None, 4)
        proposals = keras_rcnn.layers.object_detection.ObjectProposal(
            rois)([rpn_regression, rpn_classification])

        # slices of shape (1, None, 7, 7, 3)
        slices = keras_rcnn.layers.ROI((7, 7), rois)([inputs, proposals])
        
        # Implement the R-CNN heads shown in the figure below
        [score, boxes] = keras_rcnn.heads.ResHead(classes)(slices)

And the R-CNN heads from Mask R-CNN looks like

That becomes:

        y = keras.layers.TimeDistributed(
            keras.layers.Conv2D(1024, (1, 1)))(x)

        # conv5 block as in Deep Residual Networks with first conv operates
        # on a 7x7 RoI with stride 1 (instead of 14x14 / stride 2)
        for i in range(3):
            y = _bottleneck(512, (1, 1))(y)

        y = keras.layers.TimeDistributed(
            keras.layers.BatchNormalization(axis=channel_axis))(y)
        y = keras.layers.TimeDistributed(
            keras.layers.Activation("relu"))(y)

        # class and box branches
        y = keras.layers.TimeDistributed(
            keras.layers.AveragePooling2D((7, 7)))(y)

        score = keras.layers.TimeDistributed(
            keras.layers.Dense(classes, activation="softmax"))(y)

        boxes = keras.layers.TimeDistributed(
            keras.layers.Dense(4 * classes))(y)

in keras_rcnn.heads.ResHead.

What do you think? @0x00b1

P.S. The API design should be discussed in #28

codecov-io · 2017-06-09T23:00:45Z

Codecov Report

Merging #27 into master will increase coverage by 4.81%.
The diff coverage is 97.14%.

@@            Coverage Diff             @@
##           master      #27      +/-   ##
==========================================
+ Coverage   48.44%   53.25%   +4.81%     
==========================================
  Files          15       17       +2     
  Lines         545      569      +24     
==========================================
+ Hits          264      303      +39     
+ Misses        281      266      -15

Impacted Files	Coverage Δ
keras_rcnn/heads/__init__.py	`100% <100%> (ø)`
keras_rcnn/models.py	`81.48% <100%> (+81.48%)`	⬆️
keras_rcnn/heads/resnet.py	`94.11% <94.11%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3e6f4c...9aadff0. Read the comment docs.

0x00b1 · 2017-06-10T21:42:50Z

@JihongJu Whoa! You’ve been busy! I’m excited to comment. Would you mind giving me a day or two? It’ll take some time to consider and respond.

0x00b1 · 2017-06-10T21:43:05Z

cc: @mcquin

0x00b1 · 2017-06-14T16:38:21Z

keras_rcnn/heads/resnet.py

+        else:
+            channel_axis = 1
+
+        y = keras.layers.TimeDistributed(


In my opinion, this is the trickiest part of the implementation.

I foresee two problems with using TimeDistributed.

I suspect that in nearly every situation TimeDistributed is used to step across a temporal dimension. Unfortunately, we’re the rare exception. It’ll be a challenge to explain this in our documentation (especially if we expect users to re-implement the CNN that use for feature extraction to use TimeDistributed). I’d really like Keras to modify the name to better emphasize its utility.

TimeDistributed requires users to implement their feature extractor layer by layer instead of calling on a model instance directly. I’d really prefer something like:

x = keras.layers.Input((224, 224, 3)) y = keras_resnet.ResNet50(x) y = keras.layers.TimeDistributed(y)

If it worked like this, we could hide the y = keras.layers.TimeDistributed(y) statement inside the RCNN constructor. We could probably implement our own wrapper that iterates across a model’s layers and appends a TimeDistributed layer.

@0x00b1 I agree with you here. But I can hardly see how to implement such a wrapper.
For example, we have ResHead that takes feature x as input and returns some outputs y. We need to apply TimeDistributed to each of the layers in between.
Maybe you have good ideas how to do that?

Good question. 😆

A quick method might be iterating over model.layers then pop each layer and append TimeDistributed.

Hi, @JihongJu, I added a temporal module to keras-resnet. You should be able to use keras_resnet.block.temporal.basic, keras_resnet.block.temporal.bottleneck, and keras_resnet.block.temporal._shortcut.

@0x00b1 That would be great! I am a bit busy these days. I will get back to this asap.

@0x00b1 I sent a PR to keras-resnet to fix a bug for the temporal shortcut. And could you make a release (0.0.3?) afterwards?

Released (0.0.4 because I made a dumb release mistake!)

https://pypi.python.org/pypi/keras-resnet/0.0.4

@0x00b1 Great! Thanks!

0x00b1 · 2017-06-14T16:38:43Z

keras_rcnn/layers/pooling.py

-        y2 = regions[:, 1] + regions[:, 3]
+        boxes = keras.backend.cast(boxes, keras.backend.floatx())
+        boxes = boxes / self.stride
+        x1 = boxes[..., 0]


0x00b1 · 2017-06-14T16:40:17Z

keras_rcnn/models.py

 import keras_rcnn.layers
+import keras_rcnn.heads


 class RCNN(keras.models.Model):


This looks great!

We should also consider adding a features argument that users would use to pass their feature extractor (e.g. ResNet50). What do you think?

@0x00b1 Actually, I think we should go for two arguments: features and heads (maybe rename these two to keep consistency).

@JihongJu I like it! 👍

0x00b1 · 2017-06-15T18:24:23Z

@JihongJu I merged the backend and pooling changes.

JihongJu · 2017-06-15T22:14:53Z

Great. I will continue the work on weekends.

JihongJu · 2017-07-04T11:24:19Z

@0x00b1 If you are okay with this design, I will continue with the Mask branch as in TODO with a new PR.

0x00b1 · 2017-07-10T16:14:05Z

@JihongJu Awesome! Merged! (I might do a little cleanup today or tomorrow.)

JihongJu added 3 commits June 10, 2017 00:51

Prototype mask loss

40fbffb

Add Faster R-CNN with Resnet.

3efbb2b

Add keras-resnet to setup.

b891948

JihongJu mentioned this pull request Jun 9, 2017

Region proposal network (RPN) layer #7

Closed

JihongJu added 4 commits June 10, 2017 01:19

Minor change to comment

d360886

Add test_models

8f74958

Crop and resize take 3D tensor for boxes.

15e04ea

Fix bugs in pooling layer; Structrue R-CNN head;

b540e05

JihongJu changed the title ~~[Do not merge] Structure R-CNN with ResNet~~ Build R-CNN with ResNet Jun 10, 2017

JihongJu added 5 commits June 10, 2017 15:28

Remove redundant code in models

1d0259f

Add TODO in the head for Mask R-CNN

78f3b49

Leave out the mask loss from PR#27.

bf5555e

Fix typo in comment

f0d4799

Remove regions from the ROI arguments

210c389

0x00b1 reviewed Jun 14, 2017

View reviewed changes

Merge branch 'master' into rcnn_losses

d075efc

JihongJu and others added 4 commits June 19, 2017 21:11

use keras-resnet temporal blocks

d8645ec

Merge branch 'master' into rcnn_losses

beb9324

Merge remote-tracking branch 'upstream/master' into rcnn_losses

d5a0c9b

Abstract the RCNN model and add ResNet50RCNN

9aadff0

JihongJu mentioned this pull request Jul 5, 2017

Fix TimeDistributed layer wrapper #38

Closed

0x00b1 merged commit eafbfcc into broadinstitute:master Jul 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build R-CNN with ResNet #27

Build R-CNN with ResNet #27

JihongJu commented Jun 9, 2017 •

edited

Loading

codecov-io commented Jun 9, 2017 •

edited

Loading

0x00b1 commented Jun 10, 2017

0x00b1 commented Jun 10, 2017

0x00b1 Jun 14, 2017

JihongJu Jun 14, 2017

0x00b1 Jun 15, 2017

0x00b1 Jun 19, 2017

JihongJu Jun 19, 2017

JihongJu Jun 19, 2017

0x00b1 Jun 19, 2017

0x00b1 Jun 19, 2017

JihongJu Jun 19, 2017

0x00b1 Jun 14, 2017

0x00b1 Jun 14, 2017

JihongJu Jun 14, 2017

0x00b1 Jun 19, 2017

0x00b1 commented Jun 15, 2017

JihongJu commented Jun 15, 2017

JihongJu commented Jul 4, 2017

0x00b1 commented Jul 10, 2017

Build R-CNN with ResNet #27

Build R-CNN with ResNet #27

Conversation

JihongJu commented Jun 9, 2017 • edited Loading

codecov-io commented Jun 9, 2017 • edited Loading

Codecov Report

0x00b1 commented Jun 10, 2017

0x00b1 commented Jun 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x00b1 commented Jun 15, 2017

JihongJu commented Jun 15, 2017

JihongJu commented Jul 4, 2017

0x00b1 commented Jul 10, 2017

JihongJu commented Jun 9, 2017 •

edited

Loading

codecov-io commented Jun 9, 2017 •

edited

Loading