v0.9.0 - Add Regularization Layers in Neural Network

Release Note for RustyML v0.9.0

Significant Update and Explanation

Added: Add Regularization Layers in Neural Network

"Regularization layers" mainly refer to those layers that can directly control or suppress overfitting during the training process. They help the model generalize by imposing constraints, noise or normalization on parameters or outputs.

In this update, RustyML has added two types of regularization layers (a total of 6)

• Dropout class:

  • Dropout
  • Spatial Dropout 1D
  • Spatial Dropout 2D
  • Spatial Dropout 3D

• Normalization class:

  • Batch Normalization
  • Layer Normalization

Just write it normally like other layers and it can be used.

The addition of the regularization layer has led to some underlying changes: the Layer trait now adds the set_training_if_mode_dependent method for setting the training mode if the layer is mode-dependent.

Added: The activation function implements the Layer trait and can be called as a layer

Previously, when you created a new layer, you wrote it as follows:

// Build a neural network
let mut model = Sequential::new();
model
    .add(Dense::new(784, 128, Activation::ReLU))
    .add(Dense::new(128, 64, Activation::ReLU))
    .add(Dense::new(64, 10, Activation::Softmax))

The third parameters accepted by layers are of the 'Activation' enumeration type.

Now, the activation functions are implemented as layers (implementing the 'Layer' trait), and by implementing the 'ActivationLayer' marker trait, the code confirms that this is an activation layer that can (and should) run along with the layer. The current writing style is as follows:

// Build a neural network
let mut model = Sequential::new();
model
    .add(Dense::new(784, 128, ReLU::new()))
    .add(Dense::new(128, 64, ReLU::new()))
    .add(Dense::new(64, 10, Softmax::new()))

The third parameter saccepted by layers are the activation layers (Layer' + 'ActivationLayer').

This change also indicates that users can now call up the activation function separately as a separate layer. If users want to define activation functions themselves, they can achieve this by implementing two traits, 'Layer' and 'ActivationLayer'.

Changed: Change the parameters that require arrays in reference to use a more general solution

ndarray has multiple ways to write functions that take arrays (a problem captured well in issue rust-ndarray/ndarray#1059). For example:

fn sum(a: ArrayView1<f64>) -> f64;
fn sum(a: &ArrayView1<f64>) -> f64;
fn sum(a: &Array1<f64>) -> f64;

All of these work, but having several equivalent forms causes confusion. The most general solution, writing generically over storage types:

fn sum<S>(a: &ArrayBase<S, Ix1>) -> f64
where S: Data<Elem = f64>;

RustyML has fully shifted to this more general writing style. For those using code with a version lower than v0.9.0, adding '&' can solve the problem.

It is worth noting that ndarray promises to add new references type in the next version to address the issue that the above code is written complexly and is difficult to understand. At that time, RustyML will switch to using the new reference types.

Update

Added

• Add regularization layers in neural network
• Introduce AdaGrad (Adaptive Gradient Algorithm) optimizer
• The activation function implements the Layer trait and can be called as a layer
• Add support for GRU layers in the neural network module
• Introduce adaptive parallel processing thresholds in layers of neural_network module

Changed

• Update default feature set to include machine_learning and neural_network
• Enhance error handling and input validation across ML models
• Enhance error handling and input validation in utility module
• Refactor gradient merging in pooling layers by introducing macros for 1D, 2D, and 3D operations
• Organize the pooling layers, recurrent layers, and the convolutional layers under their respective modules
• Streamline tensor handling and improve training stability in recurrent and dense layers
• Change optimizer computation to adaptive parallel thresholds
• Change the parameters that require arrays in reference to use a more general solution for math module
• Change the parameters that require arrays in reference to use a more general solution for machine_learning module
• Change the parameters that require arrays in reference to use a more general solution for utility module

Removed

• Remove Result type from numerical functions