Added support for composing layers.

[eaplatanios]

@rxwei not sure if we want to merge this following our earlier discussion but I open this to gather feedback and consider it.

Quoting the reason I brought this up from our earlier discussion:

The reason I brought this up is that currently there is no simple way to compose layers. In order to create a layer that is defined as the composition of two other layers one needs to create an entirely new layer struct. For example, a multi-layer perceptron could be simply defined as something like this if we allowed for a composition operator:

let mlp = Dense<Float>(inputSize: 784, outputSize: 128) >>> ReLU<Float>() >>>
  Dense<Float>(inputSize: 128, outputSize: 64) >>> ReLU<Float>() >>>
  Dense<Float>(inputSize: 64, outputSize: 32) >>> ReLU<Float>() >>>
  Dense<Float>(inputSize: 32, outputSize: 10) >>> Softmax<Float>()

This is useful when defining layer "wrappers", which I currently do for the contextual parameter generation methods that I propose in my research. Such wrappers could also be more generally useful. Currently one needs to define a new layer struct per such composition (I cannot use the sequenced API directly because I'm not applying the composition to an input).

[Shashi456]

This is very cool stuff :o

[jon-tow]

@eaplatanios I really like the idea of composable layers for describing these little mini architectures, but I find the operator >>> a bit noisy. Have you experimented with any other designs? For example, a Composable layer structure with a variadic initializer that consumes layers and chains them together in some fashion.

Overall, I look forward to having this in the API!

[rxwei]

Combinators and operators like >>> are usually not quote idiomatic in Swift, and we'd prefer exploring a Sequential(...) solution. However, this is not yet possible because Swift does not yet support variadic generic parameters.

For now, I'd strongly suggest going a different route: Define a Sequential type and overload its initializer to a certain arity like today's sequenced(through:...) method. Then you'll be able to use Sequential(layer1, layer2, ...) for all common cases. When variadic generics are possible (within the next year or two), we will be able to switch to that directly.

Note: This is not yet possible today because the differentiation feature does not yet support differentiating closure contexts, and other ways of defining it may be limited to the fact that Swift doesn't yet support parameterized extensions.

Another (much nicer) possibility is to sugar away combinator types with function builders:

public struct Sequential<Layer1: Layer, Layer2: Layer>: Layer where Layer1.Output == Layer2.Input {
    internal var layer1: Layer1
    internal var layer2: Layer2

    public typealias Input = Layer1.Input
    public typealias Output = Layer2.Output

    internal init(_ layer1: Layer1, _ layer2: Layer2) {
        self.layer1 = layer1
        self.layer2 = layer2
    }

    @differentiable
    public func callAsFunction(_ input: Input) -> Output {
        layer2(layer1(input))
    }

    public init(@LayerBuilder layers: () -> Self) {
        self = layers()
    }
}

@_functionBuilder
public struct LayerBuilder {
    public static func buildBlock<L1: Layer, L2: Layer>(_ l1: L1, _ l2: L2) -> Sequential<L1, L2>
        where L1.Output == L2.Input {
        Sequential(l1, l2)
    }

    public static func buildBlock<L1: Layer, L2: Layer, L3: Layer>(_ l1: L1, _ l2: L2, _ l3: L3) -> Sequential<L1, Sequential<L2, L3>>
        where L1.Output == L2.Input, L2.Output == L3.Input {
        Sequential(l1, Sequential(l2, l3))
    }

    public static func buildBlock<L1: Layer, L2: Layer, L3: Layer, L4: Layer>(_ l1: L1, _ l2: L2, _ l3: L3, _ l4: L4) -> Sequential<L1, Sequential<L2, Sequential<L3, L4>>>
        where L1.Output == L2.Input, L2.Output == L3.Input {
        Sequential(l1, Sequential(l2, Sequential(l3, l4)))
    }
}

let layers = Sequential {
    Dense<Float>(inputSize: 3, outputSize: 3)
    Conv2D<Float>(filterShape: (1, 1, 1))
}

This unfortunately won't compile today because type inference isn't so flexible with function builders, and I'm getting quite a few compiler crashers because the feature is very much experimental.

[eaplatanios]

@rxwei I like the function builder approach you proposed. I implemented it and it compiles fine for me. Could you please check the changes I just pushed?

Also, I'm not sure what to do to make the generated file conform to our style guidelines without adding a bunch of heuristics to the gyb file. Should I just remove the gyb file and reformat the generated file manually?

[rxwei]

That’s awesome! Yeah it somehow didn’t compile for me when I wrote the example, but I’m glad it worked.

[rxwei]

It's fine to leave the gyb in. Don't worry about formatting for generated files.

[Shashi456]

@eaplatanios, a small doubt here. why do we have several functions here and in the layer.swift file explicitly mentioning the number of layers ? Like layer1, layer2...

A very weird question but can't recursive functions or loops work in this case?
And what if a sequential has 10 layers will that be out of support ?

[rxwei]

This is because Swift does not support variadic generic parameters. When that feature becomes possible, only one builder function needs to be defined.