RustyML v0.16.0

RustyML 0.16.0 is an architecture release for the neural network. Release 0.15 added layers. This release changes how the layers connect, how they get their shapes, and where they keep their state. A model can now be a directed graph, so residual blocks, shared encoders, and models with several inputs or outputs become expressible. Every layer infers its shapes at build time, and a layer keeps no state between passes. Checkpoints address every array by name. The classical machine learning side does not change. This release breaks model construction code and every saved model file, so read the migration section before you upgrade.

Graph models

GraphBuilder builds a Graph, a model whose layers form a directed graph rather than a chain. input adds an input, add calls a new layer on 1 or more nodes, and build(&outputs) is the only call that can fail. The build refuses an empty graph, a node that reads a later node, and a node that reaches no output. Each refusal names the node and the layer.

let mut builder = GraphBuilder::new();
let x = builder.input(Shape::known(&[4, 8]));
let hidden = builder.add(Dense::new(8, Activation::ReLU)?, &[x]);
let sum = builder.add(Add::new(), &[x, hidden]);
let head = builder.add(Dense::new(2, Activation::Linear)?, &[sum]);
let mut model = builder.build(&[head])?;

A node is 1 call of 1 layer. layer puts a layer in the graph without calling it, and apply calls it on a node. Several nodes can call the same layer, which gives weight sharing. The shared layer holds 1 set of arrays and gets 1 update per step, and its gradient is the sum over the nodes that call it. compile gives every output the same loss, and compile_many gives each output its own loss. with_loss_weights sets the weight of each loss in the total. Training, inference, summary, and persistence work as they do on Sequential. A graph that holds 1 chain gives the same result as the equivalent Sequential, bit for bit.

7 merge layers join the branches of a graph: Add, Subtract, Multiply, Average, Maximum, Minimum, and Concatenate. The 6 elementwise layers broadcast every axis after the batch axis. Concatenate::new(axis) accepts a negative axis, which counts back from the last axis. The metrics module already has an Average, so the prelude root keeps metrics::Average. Import the layer as prelude::neural_network::Average.

Shape inference and SequentialBuilder

A layer constructor no longer takes an input shape. The shape reaches the layer through Layer::build, and Layer::compute_output_shape gives the output shape from the configuration alone. The new Shape type holds 1 entry per axis, and each entry is a fixed extent or a free one. The batch axis is usually free, so 1 layer serves every batch size.

SequentialBuilder is now the only way to get a Sequential. build(&input_shape) walks the stack once and gives each layer the shape that reaches it. A shape mismatch fails at build time, and the error names the position and the type of the layer. Before this release, the same mismatch failed in the middle of a forward pass, with no layer named. The training and inference methods exist only on the built model, so the compiler rejects code that trains a model that was never built.

let mut model = SequentialBuilder::new()
    .add(Dense::new(128, Activation::ReLU)?)
    .add(Dense::new(10, Activation::Softmax { axis: -1 })?)
    .build(&Shape::known(&[32, 784]))?;

After the build, a layer refuses an input whose rank or fixed axes disagree with its build shape. The output shapes that summary prints are therefore the shapes the model produces. They no longer depend on the last input the model saw.

A layer keeps no state between passes

A new Ctx value holds everything that exists only for 1 pass: the training flag, the caches, the parameter gradients, and the state changes that a training pass proposes. forward and backward take &self, and the model applies the proposed state after the pass. Ctx::training() and Ctx::inference() select the mode, so 1 forward method serves both modes. A compiled Sequential or Graph is Send + Sync, so several threads can run inference on 1 model.

The layer trait is now 3 traits. LayerBase holds the name, the arrays, and the build shape. UnaryLayer holds the computation of a layer with 1 input. Layer is the general interface for 1 or more inputs, and every UnaryLayer gets it through a blanket implementation. The optimizers key their per-parameter state on a ParamId, which is the layer position and the array name. A reordered or resized layer therefore no longer gives its optimizer state to a different array.

Named checkpoints

A checkpoint now stores each array under a path such as 0.kernel. The path is the layer position and the array name. Each array also records whether it is trainable, and each layer records the shape it was built for. load_from_path stays strict. It refuses any disagreement and names the path, and a refused load does not change the model. The new load_partial_from_path loads what the file and the model agree on and returns a LoadReport of applied, missing, and unused paths. weight_paths() lists every path, and weight(path) gives a read-only view of 1 array.

This design removes the LayerWeight enum, its 20 container structs, and the type checks that wrote each container into its layer. About 1000 lines go. A new layer now needs only its type name and the names of its arrays.

New layers and options

Rescaling computes x * scale + offset, so input normalization can be part of the model. Reverse reverses the order of 1 axis, which a bidirectional recurrent model needs. SimpleRNN, LSTM, and GRU gain return_sequences and go_backwards, verified against Keras. A stack of recurrent layers therefore no longer needs RepeatVector between its layers. Guide section 3.7.9 shows a complete bidirectional model.

Each of the 10 convolution layers gains dilation_rate. Conv1D also gains causal padding through the new ConvPadding enum. Dense and the convolution layers gain use_bias, and the 4 normalization layers gain center and scale, with the same meanings as in Keras. Dropout gains noise_shape. Dense now accepts an input of rank 2 or more and transforms the last axis, so 1 Dense layer can process every timestep of a sequence.

Fixes

A layer that gave 2 arrays the same name had their gradients and optimizer state merged, and no error occurred. A gradient stored under a path that no parameter reads was also lost. The model build and the training step now refuse both cases and name the path.

SimpleRNN::set_weights kept an array in the memory order it received. A transposed kernel therefore caused a panic on the first training step. The layer now converts the array to standard order, as LSTM and GRU already did.

The parallel and serial paths of BatchNormalization gave different results in the last bit, so a change to the parallel gate changed the result. The 2 paths now agree bit for bit. Some performance problems are also fixed. The parallel activation forward passes no longer copy the tensor first, which makes them 1.9 to 3.9 times faster.

Breaking changes and migration

• Model construction. 32 layer constructors no longer take an input shape. For example, Dense::new(units, activation) replaces Dense::new(input_dim, units, activation). Sequential::new, Sequential::new_with_seed, and Sequential::add are removed. Use SequentialBuilder, and give the input shape to build.
• Saved models. MODEL_FORMAT_VERSION is now 3. A file from an earlier release does not load, and the error names both versions. Build the model with this release, train it again, and save it again.
• Array names. The array names now follow Keras: kernel, depthwise_kernel, pointwise_kernel, and moving_mean and moving_variance on BatchNormalization. The arguments of set_weights use the same names.
• Custom layers and optimizers. Layer::predict, set_training, get_weights, and ParamGrad are removed. A custom layer implements LayerBase and UnaryLayer, and it reads the mode and stores its caches and gradients through Ctx. Optimizer::update now takes the gradient store and a &mut dyn LayerBase.
• Random seeds. A layer with a seed now records the seed and draws its weights at build time. A seeded layer that you add to a stack therefore no longer changes the weights of the layers after it.
• Kernel size check. The rule that an input must be at least as large as the kernel now applies only under Valid padding. It runs at build time, as in Keras.

The full list of changes is in CHANGELOG.md. The API reference is at docs.rs/rustyml, and the user guide covers graph models and shape inference in chapter 3.