Subpixel convolution(state of art) implementation rather than using Deconvolution.

[chowkamlee81]

Is there is any mxnet implementation of subpixel CNN rather than using Deconvolution which is the state of art according to Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network.

There could be any developement from amazon mxnet framework?

[eldercrow]

As far as I know, mxnet does not support subpixel upsampling layer yet. Still, you can emulate it using a series of transpose and reshape layers. I am currently using the followings, messy but works anyway.

def subpixel_upsample(data, ch, c, r, name=None):
    '''
    Transform input data shape of (n, ch*r*c, h, w) to (n, ch, h*r, c*w).

    ch: number of channels after upsample
    r: row scale factor
    c: column scale factor
    '''
    if r == 1 and c == 1:
        return data
    X = mx.sym.reshape(data=data, shape=(-3, 0, 0))  # (n*ch*r*c, h, w)
    X = mx.sym.reshape(data=X, shape=(-4, -1, r * c, 0, 0))  # (n*ch, r*c, h, w)
    X = mx.sym.transpose(data=X, axes=(0, 3, 2, 1))  # (n*ch, w, h, r*c)
    X = mx.sym.reshape(data=X, shape=(0, 0, -1, c))  # (n*ch, w, h*r, c)
    X = mx.sym.transpose(data=X, axes=(0, 2, 1, 3))  # (n*ch, h*r, w, c)
    X = mx.sym.reshape(data=X, name=name, shape=(-4, -1, ch, 0, -3))  # (n, ch, h*r, w*c)
    return X

[chowkamlee81]

Thanks eldercrow. Yeah it is the same sub-pixel convolutions...Thanks for suggestions

[chowkamlee81]

Hai Eldercrow, ihave incorporated your suggestions for sub-pixel dense upsampling operation rather than deconvolution.
In jupyter notebook, iam able to visualize the dimension of network graphically. It is working perfectly but when i given this symbol for training throws an exception below.
Kindly suggest
" src/operator/./crop-inl.h:139: Check failed: param_.offset[1] <= data_shape[3]-out_shape[3] (8 vs. 0) offset[1] should be less than the residual space of width"

[chowkamlee81]

@eldercrow , kindly suggest for this dense upsampling operation.

My i/p is 1024x48x64
My o/p required is 197681024.

Hence i did conv to produce (191616)4864. Here upsampling factor is 16, num_classes=19
Through your technique i got 197681024 which is desired. But iam getting error during training.
Kindly suggest

[eldercrow]

Seems that the error is from a concat layer, but the function above has no concat layer. I guess you did something wrong in cropping your feature map.

[chowkamlee81]

iam not using any concat layer, below is the code along with jupyter notebook producing dimensions.
Kindly pls suggest...Im getting correct result in jupyter notebook but when the symbol is given for training, exception thrown
Code
num_classes = 19
upscale_factor = 16 #Input (1024,48,64)
interim_featmap = mx.symbol.Convolution(name='interim_featmap', data=relu_fc6, num_filter=upscale_factorupscale_factornum_classes, pad=(0,0), kernel=(1, 1), stride=(1, 1),no_bias=True) #(N,Cr^2,H,W)
splitted = mx.symbol.reshape(name='splitted', data=interim_featmap, shape=(0, -4, -1, upscale_factor**2, 0, 0)) #(N, C, r^2, H, W)
unflatten = mx.symbol.reshape(name='unflatten', data=splitted, shape=(0, 0, -4, upscale_factor, upscale_factor, 0, 0)) #(N, C, r, r, H, W)
swapped = mx.symbol.transpose(name='swapped', data=unflatten, axes=(0, 1, 2, 4, 3, 5)) # (N, C, H, r, W, r)
upsampling_featmap = mx.symbol.reshape(name='upsampling_featmap', data=swapped, shape=(0, 0, -3, -3)) # (N, C, Hr, W*r)
featmap_score = mx.symbol.Convolution(data=upsampling_featmap, kernel=(1, 1), pad=(0, 0), stride = (1, 1),num_filter=num_classes, name="featmap_score",no_bias = True) #(1,19,768,1024)

croped_score = mx.symbol.Crop(*[featmap_score, data], offset=(8, 8), name='croped_score') #(1,19,768,1024)
softmax = mx.symbol.SoftmaxOutput(data=croped_score, label=seg_cls_gt, normalization='valid', multi_output=True,use_ignore=True, ignore_label=255,

• name="softmax")

[eldercrow]

Sorry, I was meaning crop not concat. I suggest you to check the feature dim after crop layer.

[chowkamlee81]

Thanks for your reply ..
featmap_score = mx.symbol.Convolution(data=upsampling_featmap, kernel=(1, 1), pad=(0, 0), stride = (1, 1),num_filter=num_classes, name="featmap_score",no_bias = True) # Dimensions (1,19,768,1024)
croped_score = mx.symbol.Crop(*[featmap_score, data], offset=(8, 8), name='croped_score') # Dimensions (1,19,768,1024)
Crop is done so that for other datasets it should be compatible with i/p dimensions. In this scenario, before and after crop dimensions remains same since network is adjusted to work for this dataset i.e multiples of 2,4,8etc.
Hence kindly suggest if you have any ideas since iam trying hard

[eldercrow]

From your error message, it seems that cropping is unnecessary since data_shape[3] - out_shape[3] is 0.

src/operator/./crop-inl.h:139: Check failed: param_.offset[1] <= data_shape[3]-out_shape[3] (8 vs. 0) offset[1] should be less than the residual space of width

[chowkamlee81]

Dear Eldercrow,

I tried according to you and i removed crop function as you suggested. After that module started training. Even after several epochs training log loss measure remains same after several epochs.

Below is report
Epoch[0] Batch [10] Speed: 3.88 samples/sec Train-FCNLogLoss= 2.944441
.............................................

Epoch[0] Batch [730] Speed: 3.843 samples/sec Train-FCNLogLoss= 2.944441.

Kindly suggest how to go ahead and solve this issue

[chowkamlee81]

Dear Eldercrow, I tried according to you and i removed crop function as you suggested. After that module started training. Even after several epochs training log loss measure remains same after several epochs. Below is report Epoch[0] Batch [10] Speed: 3.88 samples/sec Train-FCNLogLoss= 2.944441 ............................................. Epoch[0] Batch [730] Speed: 3.843 samples/sec Train-FCNLogLoss= 2.944441. Kindly suggest how to go ahead and solve this issue

…

On 16 September 2017 at 08:38, eldercrow ***@***.***> wrote: From your error message, it seems that cropping is unnecessary since data_shape[3] - out_shape[3] is 0. src/operator/./crop-inl.h:139: Check failed: param_.offset[1] <= data_shape[3]-out_shape[3] (8 vs. 0) offset[1] should be less than the residual space of width — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#7717 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AeIabTFBaVXZGQyucnecWh-G0Nt-8NPrks5sizuugaJpZM4PLPjX> .

[szha]

@apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage.

For general "how-to" questions, our user forum (and Chinese version) is a good place to get help.

[leleamol]

Recommended Labels: "Question", "Discussion"

[ThomasDelteil]

@chowkamlee81 are you still working on that? If not please consider closing.
Can you provide your a full reproducible code sample and I'll try to investigate why your loss is not changing. One possibility is that you haven't set your learning rate properly, or your gradients are not flowing back from your loss.

You can have a look at the GluonCV FCN implementation as a reference:
https://gluon-cv.mxnet.io/build/examples_segmentation/train_fcn.html#sphx-glr-build-examples-segmentation-train-fcn-py

@sandeep-krishnamurthy can you add "Pending Requester Info" tag? And remove the Example tag

[kalyc]

@chowkamlee81 could you please close the issue if it has been fixed? Thanks