[MXNET-108] Adding BilinearResize2D and AdaptiveAvgPool2d operators #9688

zhanghang1989 · 2018-02-03T02:08:14Z

Description

Add operators:

BilinearResize2D
AdaptiveAvgPooling2d

link to JIRA issue 108

Checklist

Essentials

Passed code style checking (make lint)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

BilinearResize2D
AdaptiveAvgPool2D
docs
unit test

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

piiswrong · 2018-02-03T05:38:15Z

src/operator/bilinear_upsample.cu

+template<typename Dtype, typename Acctype>
+__global__ void caffe_gpu_interp2_kernel(const int n,
+    const Acctype rheight, const Acctype rwidth,
+    const DeviceTensor<Dtype, 4> data1, DeviceTensor<Dtype, 4> data2) {


Use mshadow Tensor instead of device tensor

piiswrong · 2018-02-03T05:39:57Z

src/operator/bilinear_upsample-inl.h

+  CHECK_EQ(outputs.size(), 1U);
+  mshadow::Stream<xpu> *s = ctx.get_stream<xpu>();
+  MSHADOW_REAL_TYPE_SWITCH_EX(inputs[0].type_flag_, DType, AccReal, {
+    SpatialUpSamplingBilinearUpdateGradInput<xpu, DType, AccReal>(s, inputs, outputs);


check for req

piiswrong · 2018-02-03T05:42:12Z

src/operator/bilinear_upsample-inl.h

+namespace op {
+
+struct BilinearSampleParam : public dmlc::Parameter<BilinearSampleParam> {
+  int out_height;


use tuple scale

is data channel first or channel last?

This operator mainly supports fractional scale ratio (input and output sizes can be arbitrary), so it is more convenient to use output size instead of scale.

piiswrong · 2018-02-03T05:45:43Z

We prefer a rewrite based on mxnet utilities instead of copy paste if possible.

The current code doesn't use openmp for cpu parallelization. If you use kernel launch it will be handled automatically.

piiswrong · 2018-02-03T05:50:28Z

src/operator/bilinear_upsample.cc

+  DType *data2 = gradOutput.data_ptr();
+  channels = nbatch * channels;
+
+  // special case: same-size matching grids


This should be handled at the top level by passing through to identity compute function

piiswrong · 2018-02-03T05:51:46Z

src/operator/bilinear_upsample.cc

+
+DMLC_REGISTER_PARAMETER(BilinearSampleParam);
+
+NNVM_REGISTER_OP(BilinearUpsample2D)


Does it handle upsampling only or downsample too?

It also supports downsample, since the output size can be arbitrary integer.

For these bilinear-sampling related ops, could we implement by directly calling mx.sym.BilinearSampler? I feel that the implementations are mostly the same.

chinakook · 2018-02-06T11:14:21Z

How can we define the receptive field of this operator?

zhanghang1989 · 2018-02-06T18:50:48Z

@chinakook

import mxnet as mx
x1 = mx.nd.ones(shape=(2,3,4,4))
y1 = mx.nd.contrib.BilinearResize2D(x1, height=5, width=5)

marcoabreu · 2018-02-06T23:46:51Z

Quick note: You can also run "make lint" to test the linting locally.

piiswrong · 2018-02-14T20:24:34Z

I think its more appropriate to call the BilinearResize

zhanghang1989 · 2018-02-16T23:17:40Z

Agree. I will make the changes.

ascust · 2018-02-18T00:20:35Z

I think it would be good that we can pass a reference symbol as a target shape. For example like mx.sym.Crop, where you can crop according to the shape of the second symbol. This would be useful for a variable input size.

szha · 2018-03-22T19:00:12Z

src/operator/contrib/bilinear_resize-inl.h

+    DMLC_DECLARE_FIELD(out_height).set_range(1, 1000)
+    .describe("output height");
+    DMLC_DECLARE_FIELD(out_width).set_range(1, 1000)
+    .describe("output width");


Simply use height and width as argument names.

Addressed :)

szha · 2018-03-22T19:02:00Z

src/operator/contrib/bilinear_resize.cc

+Resize the 2D input, using bilinear interpolation.
+
+  Expected input is a 4 dimensional array (batch x channel x height x width) and the output
+  with the shape of (batch x channel x out_height x out_width). 


We consistently use NCHW to describe such layout elsewhere, so you can consider using it here too to make less verbose doc.

zhanghang1989 · 2018-03-22T20:21:24Z

Hi @cjolivier01 , the CI server seems to stop working.

zhanghang1989 · 2018-03-23T17:07:21Z

Hi @szha @piiswrong @cjolivier01 , I have made the changes following the suggestions. Let me know if you have further comments. Thx

cjolivier01 · 2018-03-23T17:11:50Z

LGTM

zhanghang1989 · 2018-03-23T17:15:03Z

Thanks @cjolivier01 ! Could you approve the request?

piiswrong · 2018-03-23T23:54:19Z

src/operator/contrib/adaptive_avg_pooling.cc

+.describe(R"code(
+Applies a 2D adaptive average pooling over an input signal composed of several input planes.
+
+    The output size is (N x C x output_size x output_size), for any input (NCHW).


In pytorch output_size can be either a single int or a tuple of 2 ints.

Also why do you indent here? How would it render?

An int or a tuple of 2 ints are allowed now.

piiswrong · 2018-03-23T23:55:19Z

src/operator/contrib/adaptive_avg_pooling-inl.h

+  int output_size;
+  DMLC_DECLARE_PARAMETER(AdaptiveAvgPoolParam) {
+    DMLC_DECLARE_FIELD(output_size).set_range(1, 1000)
+    .describe("output size");


expand the doc

piiswrong · 2018-03-23T23:55:34Z

src/operator/contrib/bilinear_resize-inl.h

+  int width;
+  DMLC_DECLARE_PARAMETER(BilinearSampleParam) {
+    DMLC_DECLARE_FIELD(height).set_range(1, 1000)
+    .describe("output height");


anirudh2290 · 2018-04-09T00:23:48Z

@zhanghang1989 any updates on this ?

* two shapes input * docs * typo

crazyleg · 2018-05-15T18:26:33Z

@zhanghang1989, can you please provide a code example of using BilinearResize2D in a Symbol-based network fully convolutional network to do ratio based upscaling, when shape is unknown/dynamic? (x2, x3)

zhanghang1989 · 2018-05-15T18:29:06Z

@crazyleg

import mxnet as mx
x1 = mx.nd.ones(shape=(2,3,4,4))
y1 = mx.nd.contrib.BilinearResize2D(x1, height=5, width=5)

crazyleg · 2018-05-16T05:08:59Z

@zhanghang1989 That's a NDArray way of doing it. I was asking about Symbol-ish way and shape inference.

zhanghang1989 · 2018-05-16T05:10:43Z

I am not familiar with Symbol API. Sorry about that.

…pache#9688) * bilinear upsample from PYTorch * fix cpu backward * fix indent, add req * fix lint * fix lint * lint * handle req * add adaptive avg pooling operator * rename to bilinear resize * fix name * change assertion * rm unused var * refactor using mshadow tensor * rm devicetensor, only using mshadow * add docs * naming * merge * Revert "merge" This reverts commit a2a809a. * add unit test for BilinearResize2D and AdaptiveAvgPool2D * for test in python2, cast into float * mv function inside * link docs * address the comments * lint * add back private () * correct lint * decleare var * link params docs * fix bug * mv to contrib and upodate docs * contrib header * change include path for contrib * lint * register to contrib * lint * rename width, height, docs * rename param * Patch1 (#1) * two shapes input * docs * typo * lint * lint

bilinear upsample from PYTorch

99e824f

zhanghang1989 mentioned this pull request Feb 3, 2018

I cannot make mxnet.ndarray.UpSampling work with bilinear interpolation #9138

Closed

piiswrong reviewed Feb 3, 2018

View reviewed changes

zhanghang1989 added 4 commits February 6, 2018 11:16

fix cpu backward

3453fd9

fix indent, add req

6085571

fix lint

9dd48a9

fix lint

4bffb27

zhanghang1989 added 4 commits February 6, 2018 16:00

lint

94c1954

Merge github.com:apache/incubator-mxnet into bilinear

52d1905

handle req

ecbc23f

add adaptive avg pooling operator

df3ffe9

zhanghang1989 changed the title ~~bilinear upsample from PyTorch~~ Adapt operators from PyTorch, will keep adding Feb 7, 2018

Roshrini mentioned this pull request Feb 23, 2018

Update Upsampling operator for "bilinear" mode onnx/onnx-mxnet#33

Open

sxjscience mentioned this pull request Mar 3, 2018

Incomplete UpSamping doc #9970

Closed

rename to bilinear resize

86a544b

zhanghang1989 requested a review from cjolivier01 as a code owner March 3, 2018 00:41

zhanghang1989 added 3 commits March 2, 2018 16:45

fix name

aaf19c6

change assertion

39c1e14

rm unused var

4960f19

szha reviewed Mar 22, 2018

View reviewed changes

rename width, height, docs

033a675

zhanghang1989 added 2 commits March 22, 2018 13:21

Merge https://github.com/apache/incubator-mxnet into bilinear

eb59dc3

rename param

916ec0d

sxjscience approved these changes Mar 23, 2018

View reviewed changes

piiswrong reviewed Mar 23, 2018

View reviewed changes

zhanghang1989 and others added 4 commits April 8, 2018 21:09

Patch1 (#1)

2ce32ab

* two shapes input * docs * typo

Merge https://github.com/apache/incubator-mxnet into bilinear

89aa321

lint

3baea36

lint

dea3295

piiswrong merged commit 09b880a into apache:master Apr 9, 2018

This was referenced Apr 9, 2018

Custom interpolation layer. Gluon. #10249

Closed

how to use mx.sym.UpSampling for bilinear upsampling #4134

Closed

Upsampling: lack of arguments crashes the kernel #8560

Closed

zhanghang1989 deleted the bilinear branch April 10, 2018 19:56

wang-yijie mentioned this pull request Apr 16, 2018

Patch BilinearResize2D TuSimple/mxnet#28

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-108] Adding BilinearResize2D and AdaptiveAvgPool2d operators #9688

[MXNET-108] Adding BilinearResize2D and AdaptiveAvgPool2d operators #9688

zhanghang1989 commented Feb 3, 2018 •

edited

piiswrong Feb 3, 2018

piiswrong Feb 3, 2018

piiswrong Feb 3, 2018

piiswrong Feb 3, 2018

zhanghang1989 Feb 6, 2018

piiswrong commented Feb 3, 2018

piiswrong Feb 3, 2018 •

edited

piiswrong Feb 3, 2018

zhanghang1989 Feb 6, 2018

sxjscience Feb 13, 2018

chinakook commented Feb 6, 2018

zhanghang1989 commented Feb 6, 2018 •

edited

marcoabreu commented Feb 6, 2018

piiswrong commented Feb 14, 2018

zhanghang1989 commented Feb 16, 2018

ascust commented Feb 18, 2018

szha Mar 22, 2018 •

edited

zhanghang1989 Mar 22, 2018

szha Mar 22, 2018 •

edited

zhanghang1989 commented Mar 22, 2018

zhanghang1989 commented Mar 23, 2018

cjolivier01 commented Mar 23, 2018 •

edited

zhanghang1989 commented Mar 23, 2018

piiswrong Mar 23, 2018

piiswrong Mar 23, 2018

zhanghang1989 Apr 9, 2018

piiswrong Mar 23, 2018

piiswrong Mar 23, 2018

anirudh2290 commented Apr 9, 2018

crazyleg commented May 15, 2018

zhanghang1989 commented May 15, 2018

crazyleg commented May 16, 2018

zhanghang1989 commented May 16, 2018


		DMLC_REGISTER_PARAMETER(BilinearSampleParam);

		NNVM_REGISTER_OP(BilinearUpsample2D)

[MXNET-108] Adding BilinearResize2D and AdaptiveAvgPool2d operators #9688

[MXNET-108] Adding BilinearResize2D and AdaptiveAvgPool2d operators #9688

Conversation

zhanghang1989 commented Feb 3, 2018 • edited

Description

Checklist

Essentials

Changes

Comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piiswrong commented Feb 3, 2018

piiswrong Feb 3, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chinakook commented Feb 6, 2018

zhanghang1989 commented Feb 6, 2018 • edited

marcoabreu commented Feb 6, 2018

piiswrong commented Feb 14, 2018

zhanghang1989 commented Feb 16, 2018

ascust commented Feb 18, 2018

szha Mar 22, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szha Mar 22, 2018 • edited

Choose a reason for hiding this comment

zhanghang1989 commented Mar 22, 2018

zhanghang1989 commented Mar 23, 2018

cjolivier01 commented Mar 23, 2018 • edited

zhanghang1989 commented Mar 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudh2290 commented Apr 9, 2018

crazyleg commented May 15, 2018

zhanghang1989 commented May 15, 2018

crazyleg commented May 16, 2018

zhanghang1989 commented May 16, 2018

zhanghang1989 commented Feb 3, 2018 •

edited

piiswrong Feb 3, 2018 •

edited

zhanghang1989 commented Feb 6, 2018 •

edited

szha Mar 22, 2018 •

edited

szha Mar 22, 2018 •

edited

cjolivier01 commented Mar 23, 2018 •

edited