31. Neural Network Specifications

Neural Network Specifications

Revision: 0.27.0

Neural Network Specifications

1. Purpose

The configuration defines a reinforcement learning Agent and its associated neural network.

The network is built as a graph composed of inputs, layers, vertices, and blocks.

2. Agent Configuration

The Agent requires:

• alphas: positive numerical parameters;
• beta: positive parameter;
• numEpochs: number of training epochs, at least 1;
• trajectorySize: trajectory size, at least 1;
• batchSize: batch size, at least 1;
• path: configuration/data path;
• network: neural network configuration.

Optional parameters:

• gamma: value between 0 and 1;
• avgReward: average reward value.

3. Network

The network contains:

• graphBuilder: required;
• activation: default activation function;
• dropOut: dropout value between 0 and 1;
• optimizationAlgo: optimization algorithm;
• updater: weight update method;
• weightInit: weight initialization method;
• l1, l2: regularization parameters.

4. Inputs

Two input types are supported.

FeedForward

A vector input defined by its size.

Input shape:

[size]

Output shape:

[size]

Convolutional

A two-dimensional input defined by:

• width;
• height;
• depth;
• format: NCHW or NHWC.

The input shape depends on the selected format:

NCHW:

[batch, depth, height, width]

NHWC:

[batch, height, width, depth]

5. Available Layers

The network supports the following layers.

ActivationLayer

Applies a non-linear activation function to its input.

Main properties:

• inputs: input layer or node;
• activation: activation function.

Output dimensions:

The output dimensions are unchanged.

Input:  [N, C, H, W]
Output: [N, C, H, W]

For a feed-forward input:

Input:  [N, size]
Output: [N, size]

The ActivationLayer is typically used after a convolution or another feature-processing layer to introduce non-linearity into the network.

BatchNormalizationLayer

Normalizes the activations produced by the previous layer.

Main property:

• inputs: input layer or node.

Output dimensions:

The output dimensions are unchanged.

Input:  [N, C, H, W]
Output: [N, C, H, W]

Batch normalization can improve training stability and convergence, particularly in deeper networks.

ConvolutionLayer

Applies a two-dimensional convolution to the input.

Main properties:

• inputs: input layer or node;
• kernelSize: convolution kernel dimensions;
• stride: convolution step;
• nOut: number of output feature maps.

For example:

• kernelSize: [5, 5] uses a 5×5 convolution kernel;
• stride: [1, 1] moves the kernel one position at a time;
• nOut: 32 produces 32 output feature maps.

Output dimensions:

For an input:

[batch, channels, height, width]

the output has:

[batch, nOut, outputHeight, outputWidth]

The spatial dimensions are calculated as:

outputHeight = floor((height - kernelHeight) / strideHeight) + 1
outputWidth  = floor((width  - kernelWidth)  / strideWidth)  + 1

The exact padding behavior is determined by the underlying network implementation.

The schema indicates [5, 5] as the default kernel size and [1, 1] as the default stride, although kernelSize is formally required by the schema.

SubsamplingLayer

Reduces the spatial dimensions of the feature maps using a pooling operation.

Main properties:

• inputs: input layer or node;
• kernelSize: pooling window dimensions;
• stride: pooling step;
• poolingType: pooling operation.

Supported pooling types:

• Avg: average pooling;
• Max: maximum pooling;
• PNorm: p-norm pooling;
• Sum: sum pooling.

Output dimensions:

For an input:

[batch, channels, height, width]

the output has:

[batch, channels, outputHeight, outputWidth]

The spatial dimensions are calculated from the kernel size, stride, and padding:

outputHeight = floor((height - kernelHeight) / strideHeight) + 1
outputWidth  = floor((width  - kernelWidth)  / strideWidth) + 1

The number of channels remains unchanged.

For example:

type: SubsamplingLayer
inputs:
  - activation1
kernelSize: [2, 2]
stride: [2, 2]
poolingType: Max

With an input of:

[batch, 32, 64, 64]

and no padding, the output is:

[batch, 32, 32, 32]

ZeroPaddingLayer

Adds zero-valued padding around the input.

Main properties:

• inputs: input layer or node;
• padding: padding dimensions.

Output dimensions:

Padding increases the spatial dimensions of the input.

For example, with one pixel of padding on each side:

Input:  [batch, channels, 32, 32]
Output: [batch, channels, 34, 34]

The exact output dimensions depend on the padding configuration.

OutputLayer

Produces the final output of the neural network and defines the training objective.

Main properties:

• inputs: input layer or node;
• nOut: number of output units;
• activation: output activation function;
• lossFunction: loss function;
• dropOut: dropout value.

Output dimensions:

For a feed-forward output:

[batch, nOut]

The value of nOut therefore determines the size of the final output.

For example:

type: OutputLayer
inputs:
  - pool1
nOut: 4
activation: SoftMax
lossFunction: Mse

produces:

[batch, 4]

For a reinforcement learning agent, these four outputs can, for example, represent four action values or probabilities, depending on the implementation.

6. Activation Functions

The supported activation functions are:

• Identity: returns the input without modification;
• Relu: applies the Rectified Linear Unit activation;
• SoftMax: converts outputs into normalized values, typically used for probabilities;
• Tanh: maps values to the range [-1, 1].

Activation functions do not normally change the dimensions of their input.

7. Pooling

The SubsamplingLayer supports:

• Avg: average pooling;
• Max: maximum pooling;
• PNorm: p-norm pooling;
• Sum: sum pooling.

Pooling normally preserves the number of channels while reducing the spatial dimensions.

8. Output and Loss Functions

The OutputLayer requires:

• inputs;
• nOut.

It can also specify:

• activation function;
• loss function;
• dropout.

Supported loss functions include:

• CosineProximity;
• Mcxent;
• Hinge;
• KLDivergence;
• L1;
• L2;
• MeanAbsoluteError;
• MeanAbsolutePercentageError;
• MeanSquaredLogarithmicError;
• Mse;
• NegativeLogLikelihood;
• Poisson;
• ReconstructionCrossentropy;
• SparseMcxent;
• SquaredLoss;
• SquaredHinge;
• Wasserstein;
• Xent.

The appropriate loss function depends on the type of output and the training objective.

9. Connections and Vertices

The network is represented as a graph. Each component has a name, and connections are specified using the inputs property.

ElementWiseVertex

An ElementWiseVertex combines multiple inputs using an element-wise operation.

Properties:

• type: ElementWiseVertex;
• inputs: list of input nodes.

Output dimensions:

The output normally has the same dimensions as the inputs.

For example:

Input A: [batch, 32, 32, 32]
Input B: [batch, 32, 32, 32]

Output:  [batch, 32, 32, 32]

The inputs must therefore be compatible for the selected element-wise operation.

10. ResNet Blocks

Two residual block types are supported.

ConvolutionResNetBlock

A residual block that includes convolutional processing.

Properties:

• type: ConvolutionResNetBlock;
• inputs: input nodes;
• kernelSize: convolution kernel dimensions;
• stride: convolution stride;
• filters: filter configuration.

Output dimensions:

The spatial dimensions depend on:

• input dimensions;
• kernelSize;
• stride;
• filters.

The number of output channels is determined by the block's filter configuration (last value).

The spatial dimensions are calculated from the kernel size, stride:

outputHeight = floor((height - kernelHeight) / strideHeight) + 1
outputWidth  = floor((width  - kernelWidth)  / strideWidth) + 1

This block can be used when the residual block needs to perform a convolutional transformation.

IdentityResNetBlock

A residual block that preserves the identity path.

Properties:

• type: IdentityResNetBlock;
• inputs: input nodes;
• kernelSize: convolution kernel dimensions;
• filters: filter configuration.

Output dimensions:

The output is normally compatible with the input dimensions so that the residual connection can be combined with the processed branch.

The number of output channels is determined by the block's filter configuration (last value).

The spatial dimensions are calculated from the kernel size, stride:

outputHeight = floor((height - kernelHeight) / strideHeight) + 1
outputWidth  = floor((width  - kernelWidth)  / strideWidth) + 1

11. Optimization

The following updater methods are supported:

• Adam;
• Sgd;
• RmsProp.

Adam

Adaptive optimization method with configurable learning rate and related parameters.

Sgd

Stochastic Gradient Descent updater.

RmsProp

Adaptive updater based on a moving average of squared gradients.

12. Weight Initialization

The following weight initialization methods are available:

• Xavier;
• TruncatedNormalDistribution.

Xavier

Initializes weights using the Xavier initialization strategy, commonly used to maintain a suitable variance of activations across layers.

TruncatedNormalDistribution

Initializes weights using a truncated normal distribution.

Parameters:

• mean;
• stddev.

13. Layer Dimension Summary

Component	Input Dimensions	Output Dimensions
FeedForward	—	[batch, size]
Convolutional	—	[batch, C, H, W] or [batch, H, W, C]
ActivationLayer	[N, ...]	[N, ...]
BatchNormalizationLayer	[N, ...]	[N, ...]
ConvolutionLayer	[N, C, H, W]	[N, nOut, H', W']
SubsamplingLayer	[N, C, H, W]	[N, C, H', W']
ZeroPaddingLayer	[N, C, H, W]	[N, C, H', W']
ElementWiseVertex	Compatible inputs	Same shape as inputs
OutputLayer	[N, ...]	[N, nOut]
ConvolutionResNetBlock	[N, C, H, W]	Depends on filters/stride
IdentityResNetBlock	[N, C, H, W]	Normally compatible with input

Where:

• N = batch size;
• C = number of channels;
• H = height;
• W = width;
• H', W' = resulting spatial dimensions.

For NHWC, the channel dimension is placed at the end:

[N, H, W, C]

14. Simplified Structure

The configuration is organized into three main areas:

1. Agent — training parameters and network configuration.
2. Network — general neural network settings such as activation, optimization, initialization, and regularization.
3. GraphBuilder — the actual neural network structure, including inputs, layers, vertices, blocks, and outputs.

The GraphBuilder uses named components. Connections between components are defined through the inputs property.

15. YAML Example

The following example shows a simple convolutional network with an input, convolution layer, activation layer, pooling layer, and output layer.

$schema: https://mmarini.org/wheelly/dl-agent-builder-schema-0.1
class: org.mmarini.rl.agents.DLAgentBuilder

alphas:
  default: 0.001

beta: 0.01
gamma: 0.99
numEpochs: 10
trajectorySize: 128
batchSize: 32
path: ./models/agent

network:
  activation: Relu
  dropOut: 0.2

  updater:
    type: Adam
    learningRate: 0.001

  weightInit:
    type: Xavier

  l1: 0.0
  l2: 0.0001

  graphBuilder:

    inputs:
      image:
        type: Convolutional
        width: 64
        height: 64
        depth: 3
        format: NCHW

    hiddens:

      conv1:
        type: ConvolutionLayer
        inputs:
          - image
        kernelSize: [5, 5]
        stride: [1, 1]
        nOut: 32

      activation1:
        type: ActivationLayer
        inputs:
          - conv1
        activation: Relu

      pool1:
        type: SubsamplingLayer
        inputs:
          - activation1
        kernelSize: [2, 2]
        stride: [2, 2]
        poolingType: Max

    outputs:

      output:
        type: OutputLayer
        inputs:
          - pool1
        nOut: 4
        activation: SoftMax
        lossFunction: Mse

For an input of:

[batch, 3, 64, 64]

the example produces approximately:

Input
  [batch, 3, 64, 64]
          │
          ▼
ConvolutionLayer
  [batch, 32, 60, 60]
          │
          ▼
ActivationLayer
  [batch, 32, 60, 60]
          │
          ▼
SubsamplingLayer
  [batch, 32, 30, 30]
          │
          ▼
OutputLayer
  [batch, 4]

This calculation assumes:

• convolution kernel: 5 × 5;
• convolution stride: 1 × 1;
• no convolution padding;
• pooling kernel: 2 × 2;
• pooling stride: 2 × 2;
• no pooling padding.

The exact dimensions depend on the padding behavior of the underlying network implementation.

16. Summary

The configuration defines a graph-based neural network for a reinforcement learning Agent.

The network can combine:

• vector or convolutional inputs;
• convolutional layers;
• activation functions;
• batch normalization;
• pooling;
• zero padding;
• element-wise connections;
• residual blocks;
• output layers.

Each layer transforms the dimensions of its input according to its configuration. Convolution and pooling layers can change the spatial dimensions, while activation and batch normalization normally preserve them.

Training behavior can be configured through the optimization algorithm, updater, weight initialization, dropout, and regularization parameters.