Add Input Layer to Replace inputs

[shelhamer]

Approaches a fix for #1245.

The input fields are oddball members of net definitions that require special case code to initialize and manipulate since they are not layers. They cannot be tied to a phase / stage / level nor can they be defined in NetSpec.

InputLayer has tops that are assignable like inputs. The layer is essentially a no-op that holds the data assigned to its tops.

# one data input (for ILSVRC deploy for instance)
layer { 
  name: 'data' 
  type: 'Input' 
  top: 'data' 
  input_param { 
    shape: { dim: 10 dim: 3 dim: 227 dim: 227 } 
  } 
} 
# two inputs with the same shape (for RGB + HHA images for instance)
layer { 
  name: 'data-rgb-hha' 
  type: 'Input' 
  top: 'rgb' 
  top: 'hha'
  input_param { 
    shape: { dim: 10 dim: 3 dim: 500 dim: 500 } 
  } 
}
# two inputs, two shapes (for image and vector for instance)
layer { 
  name: 'data-im-vec 
  type: 'Input' 
  top: 'im' 
  top: 'vec'
  input_param { 
    shape: { dim: 1 dim: 3 dim: 500 dim: 500 }  # im shape
    shape: { dim: 1 dim: 10 }  # vec shape
  } 
}

or in net spec

import caffe
from caffe import layers as L, params as P

n = caffe.NetSpec()
#1 x 3 x 500 x 500 top
n.data = L.Input(shape=[dict(dim=[1, 3, 500, 500])])

The input fields are translated to an input layer automagically.
As a next, more radical step the Forward + bottoms ForwardPrefilled() interface is done away with.
Net inputs are preserved by mapping to Input layers instead.

Although this looks like modestly less code, ~100 lines are trimmed from Net.

Note that one can hack this behavior with a DummyDataLayer by beginning forward after the dummy but that was an unnecessary hassle.

TODO

• decide whether to preserve Net inputs: yes (at least for now)
• decide whether to preserve Forward(bottoms): no deprecated
• adapt pycaffe + matcaffe: no need since keeping Net inputs
• update docs + examples

[ronghanghu]

Some thoughts here: how to coordinate this with MemoryDataLayer? When should one use InputDataLayer and when to use MemoryDataLayer?

For me, it seems we can set a guideline as follows: if you want to have a large input array that is going to be splitted into different batches, use MemoryDataLayer. Otherwise, use InputDataLayer.

A side issue: we should also allow MemoryDataLayer to be compatible with ND-array (now they only handle 4D).

I can handle the Matcaffe part of InputDataLayer interface (and MemoryDataLayer interface).

[jeffdonahue]

Note that one can hack this behavior with a DummyDataLayer by beginning forward after the dummy but that was an unnecessary hassle.

Not that this should be an argument against an InputDataLayer, but I'll note that technically you'd be fine doing Forward through the whole net as long as you didn't specify a non-constant filler type, since DummyDataLayer does constant filling only once (in LayerSetUp).

[ronghanghu]

Quote from #1245

the Python and MATLAB interfaces duplicate a lot of code and lead to brittle transitions between train / val / data layers and the input fields for deployment.

IMO, I hope to have a InputDataLayer and allow data transform in it, in order to avoid code duplication (triplication) in C++ code, Python and Matlab. Hope to have a discuss on this.

[lukeyeager]

Thanks for this! 👍

I wouldn't say this fixes #1245 quite yet. You still need a third DEPLOY phase in order to specify all the data layers in one network, right? Hopefully that will be a simpler PR that can come after this one.

As a next, more radical step the Net inputs are done away with entirely in 87cdf87.

Are you saying that the old input-style deploy networks won't work anymore? That is pretty radical. I suggest thinking through a strategy for alerting people to the change (plug for versioned release notes! - #486).

[shelhamer]

I wouldn't say this fixes #1245 quite yet

Fair enough; I've only done part but I hope this can scaffold further work.

You still need a third DEPLOY phase in order to specify all the data layers in one network, right?

Instead of overloading phase I'd make use of the stage of the Net, which is a little looked at part of the state perhaps due to the name (I'm to blame) or it's incomplete interface (it can't be set from pycaffe for instance).

Are you saying that the old input-style deploy networks won't work anymore? That is pretty radical. I suggest thinking through a strategy for alerting people to the change (plug for versioned release notes! - #486).

The old style input fields are upgraded automatically in 9698cc8 so those old defs will still work. "Net inputs are done away with entirely" means that there is no longer any notion of Net input blob in the code. There used to be special cases but now a Net is layers all the way down.

[lukeyeager]

The old style input fields are upgraded automatically in 9698cc8 so those old defs will still work.

Cool.

[shelhamer]

@jeffdonahue @longjon please check but this looks ready to do away with input fields so that Nets are composed of layers alone. I decided to leave Net.inputs intact for the pycaffe and matcaffe interfaces but if we ever decide the inputs list is unnecessary there's even more code to be dropped.

[lukeyeager]

Ping? This combined with a PR to allow pycaffe to set the stage would finally allow all-in-one nets and let me avoid horrible hacks like prefixing layer names with train_ or deploy_:

https://github.com/NVIDIA/DIGITS/blob/v3.2.0/digits/model/tasks/caffe_train.py#L579-L584

[shelhamer]

@jeffdonahue @longjon this should finally be wrapped up if you can review -- I can squash 0d9a78f into 51f79a8 if desired because of the backtracking.

[jeffdonahue]

IMO at least ForwardPrefilled should be marked as deprecated and rewritten to just wrap around the new Forward -- just removing it breaks Caffe for probably most anyone using it for inference in production. Not sure if something similar would be reasonable for the other Forward methods.

[shelhamer]

I kept ForwardPrefilled() and Forward(bottom, loss) for now with deprecation.

[jeffdonahue]

Thanks @shelhamer, this simplifies a lot of code and LGTM besides the removal vs. deprecation issue of existing Forward methods as commented above. Since these methods have been in Caffe from the start, I think it'd be a bit to jarring to simply remove them all right off the bat (especially ForwardPrefilled which is likely used by most production Caffe users).

[shelhamer]

@jeffdonahue agreed, dropping the methods all at once is too jarring. I'll relax to a deprecation warning, although I do look forward to the day we can clear the old code. Thanks for the review!

[lukeyeager]

Tested and it's working for me 👍

For the curious, here's an example of how to read the input dimensions for data from a deploy file regardless of how it's specified:

    network = caffe_pb2.NetParameter()
    with open(deploy_file) as infile:
        text_format.Merge(infile.read(), network)

    dims = None
    # Method 1 - Input layer
    for layer in network.layer:
        if layer.type == 'Input':
            for top_index, top_name in enumerate(layer.top):
                if top_name == 'data':
                    dims = layer.input_param.shape[top_index].dim
                    break
            if dims is not None:
                break

    if dims is None:
        # Method 2: network.input
        for input_index, input_name in enumerate(network.input):
            if input_name == 'data':
                # Method 2a - input_shape.dim
                if network.input_shape:
                    dims = network.input_shape[input_index].dim
                # Method 2b - input_dim
                else:
                    dims = network.input_dim[(4*input_index):(4*(input_index+1))]

    print 'dims are:', ','.join(str(d) for d in dims)

[shelhamer]

@lukeyeager or through pycaffe

net = caffe.Net(deploy_file, caffe.TEST)
print list(net.blobs['data'].shape)

which will be robust to the Caffe format version and input field vs. layer although of course this requires the computation + memory of instantiating the net.

[lukeyeager]

Oh good call, thanks. I can let reshape do the work for me.