New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BilinearUpsamplingFiller used with DeconvolutionLayer for upsampling #2213

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
7 participants
@tnarihi
Contributor

tnarihi commented Mar 27, 2015

This is intended to be used in DeconvolutionLayer acting like UpsamplingLayer which is not implemented explicitly. You can upsample a feature map by any integer factor using the following proto.

layer {
  name: "upsample", type: "Deconvolution"
  bottom: "{{bottom_name}}" top: "{{top_name}}"
  convolution_param {
    kernel_size: {{2 * factor - factor % 2}} stride: {{factor}}
    num_output: {{channels_in}} group: {{channels_in}}
    pad: {{ceil((factor - 1) / 2.)}}
    weight_filler: { type: "bilinear_upsampling" } bias_term: false
  }
  param { lr_mult: 0 decay_mult: 0 }
}

Replace {{}} with your values. Note that the learning rate and the weight decay are set to 0 in order to keep coefficient values of bilinear interpolation unchanged during training. If you apply this to an image, this operation is equivalent to the following call in Python with Scikit.Image.

out = skimage.transform.rescale(img, factor, mode='constant', cval=0)
@tnarihi

This comment has been minimized.

Contributor

tnarihi commented Mar 27, 2015

weiliu89 added a commit to weiliu89/caffe that referenced this pull request Apr 4, 2015

Merge pull request BVLC#2213 from tnarihi/bilinear-upsampling-filler
BilinearUpsamplingFiller used with DeconvolutionLayer for upsampling
@weiliu89

This comment has been minimized.

weiliu89 commented Apr 6, 2015

I believe this is wrong. According to https://gist.githubusercontent.com/shelhamer/80667189b218ad570e82/raw/938f470ad91072929cbb9f6c739dc34488ca7d03/solve.py, it uses net.params[l][0].data[range(m), range(k), :, :] = filt. This doesn't fill filt in each channel in each kernel. It only fills the ith channel in the ith kernel, all others are 0.

Bilinear interpolation is doing channels-wise. It could be easy to fix it.

@tnarihi

This comment has been minimized.

Contributor

tnarihi commented Apr 6, 2015

Thanks for commenting @weiliu89, but this is not wrong. Please note that I assume you specify group: {{num_in}} in convolution_param. That results channel-wise convolution, and it is more efficient for both memory and computation. We have conovolunal weights with size of (num_in, 1, kernel_size, kernel_size) instead of (num_in, num_in, kernel_size, kernel_size). I don't do diagonal convolution explicitly. I updated my IPython notebook example to add a RGB image example other than gray one. This example shows you this implementation gives you result images that are equivalent to results from a well-known imaging library in Python (Scikits.Image).

EDIT: (num_in, 1, kernel_size, kernel_size) is equivalent to (num_out, 1, kernel_size, kernel_size) here. Actually, this num_out/num_in channels for weights are still redundant for channel-wise interpolation. If we implement Convolution/Deconvolution with weight shading over channels, this redundancy could be removed, but I think this will make implmentation more complicated and unmaintenanciable.

@weiliu89

This comment has been minimized.

weiliu89 commented Apr 6, 2015

@tnarihi Thanks for pointing out my mistake. Yours is indeed more efficient than the original python code. I thought group is not important in the setting and thus ignored it. Probably it is better to emphasize this more in the code, otherwise the filler is doing completely the wrong thing.

@shelhamer

This comment has been minimized.

Member

shelhamer commented Apr 6, 2015

@tnarihi @weiliu89 setting group is more obvious and makes use of less memory. The posted semantic segmentation FCNs will likely be switched to group deconvolution at some point for this reason. However the current Python filler is correct:

[range(m), range(k), :, :]

indexes each (idx, idx) pair for channelwise interpolation while all "off-diagonal" filter coefficients are left to their zero initialization.

@shelhamer

This comment has been minimized.

Member

shelhamer commented Apr 6, 2015

@tnarihi thanks for the C++ filler so interpolation can be defined instead of needing initialization by net surgery. I might call this bilinear instead of bilinear_upsampling since its the stride that results in upsampling while the bilinear kernel alone is interpolation -- although to be fair the main point of this kernel is for upsampling.

@longjon thoughts?

@shelhamer shelhamer added the JL label Apr 6, 2015

@tnarihi tnarihi force-pushed the tnarihi:bilinear-upsampling-filler branch from 15d784f to 4f249a0 Apr 6, 2015

@tnarihi

This comment has been minimized.

Contributor

tnarihi commented Apr 6, 2015

@weiliu89 Your point is right. I updated doc.

@shelhamer Thanks for your attention to this PR. Yeah, I once called it bilinear, but as you pointed, it is mainly dedicated for upsampling and I couldn't find other purpose of this filler besides upsamling, so I now call it bilinear_upsampling for clarity. Anyway, I don't have any strong preference about the name. Either is fine for me.

@longjon

This comment has been minimized.

Contributor

longjon commented Apr 6, 2015

@shelhamer @tnarihi, I agree that "upsampling" is a bit overly specific, e.g., you can use this to initialize a convolution layer and do _down_sampling. Either name is okay with me though. Also consider bilinear_filter.

@tnarihi

This comment has been minimized.

Contributor

tnarihi commented Apr 8, 2015

@longjon I am still unclear how we could create a downsampling layer which is equivalent to common image downsampling implementation by using this filler though, bilinear_filter also sounds good to me.
@shelhamer @longjon Please choose what you prefer. Is bilinear simpler? I don't have any preference among three now. Once naming is fixed, I think this PR is ready to merge.

naibaf7 added a commit to naibaf7/caffe that referenced this pull request Jun 22, 2015

shelhamer added a commit that referenced this pull request Jul 1, 2015

Merge pull request #2213 from tnarihi/bilinear-upsampling-filler
add bilinear interpolation filler
@shelhamer

This comment has been minimized.

Member

shelhamer commented Jul 1, 2015

Merged in 805a995. @tnarihi I changed the filler name to bilinear but kept your commit details and authorship. Thanks for the filler!

@shelhamer shelhamer closed this Jul 1, 2015

@tnarihi

This comment has been minimized.

Contributor

tnarihi commented Jul 1, 2015

Thanks for merging, @shelhamer!

@xyy19920105

This comment has been minimized.

xyy19920105 commented Jul 7, 2016

how to use this to downsampling with bilinear?

@thigi

This comment has been minimized.

thigi commented Apr 23, 2017

This does not work for 5D blobs. Does anyone know a workaround to use the Deconvolution-Layer with bilinear filler for 5D blobs? 5056

@weiliu89 weiliu89 referenced this pull request May 3, 2017

Closed

Questions about DSSD #563

@zmlmanly

This comment has been minimized.

zmlmanly commented Jul 5, 2017

Hi, I have a problem that the result of tensorflow.image.resize_bilinear() is different from Deconvolution-Layer with bilinear in Caffe, could you give me a help? Thank you very much.

@sfzhang15 sfzhang15 referenced this pull request May 9, 2018

Closed

about odm #53

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment