Feature Request: Locally Connected Layer

[ronrest]

The Lua version of Torch implements a Locally Connected Layer (Convolution without shared weights). It would be good to have an implementation of this available on the python API version as well.

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345

[1zb]

I came up with the same issue and found torch.nn.backends.thnn.backend.SpatialConvolutionLocal.

For anyone who want to use locally connected layer in PyTorch,

inputs=Variable(torch.randn(1,3,32,32)) # N x inC x inH x inW
weight=Variable(torch.randn(30,30,64,3,3,3)) # outH x outW x outC x inC x kH x kW
bias=Variable(torch.randn(64,30,30)) # outC x outH x outW

## arguments: kH, kW, strideH, strideW, padH, padW, inH, inW, outH, outW
local=torch.nn.backends.thnn.backend.SpatialConvolutionLocal(3, 3, 1, 1, 0, 0, 32, 32, 30, 30)
outputs=local(inputs, weight, bias) # N x outC x outH x outW

[apaszke]

This should be easy to add once #1048 is implemented

[apaszke]

Oh, if the auto-generated function works then we can expose it. @1zb can you send a PR that creates a module to do it? You can base the implementation on other Conv* modules, and use the function you found in forward

[1zb]

Sure, I am happy to do this.

[ibulu]

I would like to request the same for 1D data, Conv1DLocal?

[bwesen]

Beware that the torch.nn.backends.thnn.backend.SpatialConvolutionLocal function mentioned above is very slow, feeding it batches you get no parallelisation (minibatch of 10 is 10x the time of a 1 batch). At least in pyTorch 0.3.1. Other high-level methods using Unfold seem to work better.

[kausic94]

@1zb
I try this, but it say that required arguement bias is missing despite it being present. Do you know what the problem could be?

## arguments: kH, kW, strideH, strideW, padH, padW, inH, inW, outH, outW
local=torch.nn.backends.thnn.backend.SpatialConvolutionLocal(3, 3, 1, 1, 0, 0, 32, 32, 30, 30)
outputs=local(inputs, weight, bias) # N x outC x outH x outW

[sorenchiron]

Saddly, neither of torch.nn.backends.thnn.backend.SpatialConvolutionLocal nor torch.nn._functions.thnn.SpatialConvolutionLocal works for pyTorch 0.4.0.

directly calling local(x,w,b) yields:

lib\site-packages\torch\nn\_functions\thnn\auto.py in forward(ctx, input, *params)
    173                 args += (None,)
    174             else:
--> 175                 raise ValueError("missing required argument '%s'" % param.name)
    176
    177         args += tuple(additional_args)

ValueError: missing required argument 'bias'

That's because, as mentioned in This doc, invoking local.forward(ctx,inputs, *params) requires a ctx (SpatialConvolutionLocalBackward object here) , which by the documentation is sth not supposed to be handled by users creating extensions.

On the other hand, the correct way of invoking forward is explained in Doc: Anomaly detection example, which is by local.apply(x,w,b)

Saddly again, the outcomes of calling local.apply(x,w,b) is :

\lib\site-packages\torch\nn\_functions\thnn\auto.py in forward(ctx, input, *params)
    193             del ctx.buffers
    194
--> 195         getattr(ctx._backend, update_output.name)(ctx._backend.library_state, input, output, *args)
    196         return output
    197

TypeError: FloatSpatialConvolutionLocal_updateOutput received an invalid combination of arguments - got (int, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor), but expected (int state, torch.FloatTensor input, torch.FloatTensor output, torch.FloatTensor weight, torch.FloatTensor bias, torch.FloatTensor finput, torch.FloatTensor fgradInput, int kW, int kH, int dW, int dH, int padW, int padH, int inputWidth, int inputHeight, int outputWidth, int outputHeight)

By hacking around auto.py, I found that it is not the problem of the input (x,w,b) or args kW,dW,dH... which are of the exact required type torch.FloatTensor. The problem lays in some hidden builtin functions that were designed to generate (int, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor) params.

Hope that there will be a more general ConvBase class that provides ways to define cuda accelerated spatial iterations. Providing this ConvBase might satisfy most of the Conv-related feature-requirement issues. Since end users know math well. Rather than asking busy developers to implement well-known LocalConv, maybe users only want simple access to performance, which is what they cannot achieve by themselves except by experting cuda nvcc.

Most obviously, given ConvBase, I myself can contribute almost all kinds of special Conv layers.
Perhaps, users may feel good if pyTorch ask them to do the math, providing high-performance friendly API.
But it's possible that users may feel bad if pyTorch is slow, and they have to wait until the all-known math is implemented by overworked/tired developers using the parallel APIs that the user actually need.

[bwesen]

I was also hit by this breakage, and ended up re-learning CUDA and wrote the kernels I needed myself and added them as pyTorch extensions. This turned out to be far less work than I spent trying to figure out how to mash the other tensor functions together to do the same thing and it was 10-20x faster as well than those functions even while they worked (the im2col-based unfold things). Mainly due to the massive memory usage of the latter (it has to keep an intermediary gigantonourmous array in RAM).

But for a general case, building up functions to work in a library with multiple frontends and backends, it's much more work than for the very specific fixed kernels I needed. I guess my point being, if you have a specific case that doesn't need flexibility, adding a cuda extension is not as hard as it might seem.