Komputation

Komputation is a neural network framework for the JVM written in the Kotlin programming language.

Layers

• Entry points:

  • Input
  • Lookup

• Standard feed-forward networks:

  • Projection
  • Dense

• Convolutional neural networks (CNNs):

  • Convolution
  • Max-pooling

• Recurrent neural networks (RNNs):

  • Encoder
    • single output
    • multi-output
  • Decoder
    • single input
    • multi-input
    • attentive

• RNN units:

  • Simple recurrent unit
  • Minimal Gated Unit

• Dropout

• Highway layer

• Activation functions:

  • Identity
  • Rectified Linear Units (ReLUs)
  • Sigmoid
  • Softmax
  • Tanh

• Other layers:

  • Column repetition
  • Concatenation
  • Counter-probability
  • Exponentiation
  • Normalization
  • Transposition

Demos

• Boolean functions:

  • AND
  • Negation
  • XOR

• Running total:

  • Projection
  • Multi-input decoder

• Computer vision toy problem

• Word embedding toy problem:

  • CNN with one filter height
  • CNN with two filter heights

• Addition problem:

  • Simple recurrent unit
  • Minimal Gated Unit

• Reverse function:

  • Unidirectional RNN
  • Bidirectional RNN
  • RNN with attention

• MNIST:

  • Dropout and Nesterov's Accelerated Gradient
  • Highway layers

• TREC question classification

Sample code

The following code instantiates a convolutional neural network for sentence classification:

val network = Network(
   lookupLayer(embeddings, embeddingDimension, batchSize, maximumLength, optimization),
   concatenation(
       maximumLength * embeddingDimension,
       *filterWidths
           .map { filterWidth ->
               arrayOf(
                   convolutionalLayer(numberFilters, filterWidth, filterHeight, initialization, initialization, optimization),
                   maxPoolingLayer(numberFilters),
                   reluLayer(numberFilters),
                   dropoutLayer(numberFilters, random, keepProbability)
               )
           }
           .toTypedArray()
   ),
   projectionLayer(numberFilterWidths * numberFilters, numberCategories, initialization, initialization, optimization),
   softmaxLayer(numberCategories)
)

See the TREC demo for more details.

Initialization

• Provided
• Constant
• Gaussian
• He
• Identity
• Uniform
• Zero

Loss functions

• Logistic loss
• Squared loss

Optimization

• Stochastic Gradient Descent
• Historical:
  • Momentum
  • Nesterov's Accelerated Gradient
• Adaptive:
  • Adagrad
  • Adadelta
  • RMSProp
  • Adam