Skip to content

Module_DeepLearning

Jump to bottom
cyclone_dll edited this page Jun 13, 2026 · 10 revisions

Location: Home / Deep learning module

Deep Learning Module

Starting with 0.9, this module provides a complete differentiable training stack: tensor and layer definitions, automatic differentiation (forward/backward), loss functions, optimizers, training loops, and v3 model serialization. Aside from the early legacy static Layers utility class, the recommended APIs live under the Core / Modules / Training / Serialization namespaces.

Subsystem Namespace Description
Core Vorcyc.Mathematics.DeepLearning BatchTensor, Sequential, BatchSequential, CnnMlpModel, BatchParallelConcatModel, Parameter
Layers Vorcyc.Mathematics.DeepLearning.Modules Single-sample ILayer and batch IBatchLayer implementations
Training Vorcyc.Mathematics.DeepLearning.Training Trainer, BatchLabelSample, BatchClassLabelSample
Losses Vorcyc.Mathematics.DeepLearning.Losses ILoss, IBatchLoss, BatchMeanSquaredErrorLoss, ISparseBatchLoss
Optimizers Vorcyc.Mathematics.DeepLearning.Optimizers SgdOptimizer, AdamOptimizer
Serialization Vorcyc.Mathematics.DeepLearning.Serialization ModelSerializer (format v3), ModelArchitectureCatalog
Integration Vorcyc.Mathematics.DeepLearning.Integration Signal ↔ BatchTensor bridge, FeatureBatchBuilder, AudioTrainingSamples
DSP frontends Vorcyc.Mathematics.DeepLearning.Integration.Frontends BatchPreEmphasisLayer, BatchStftMagnitudeLayer, BatchMelFilterbankLayer
Legacy Vorcyc.Mathematics.DeepLearning.Layers Non-differentiable static Layers forward operators (parallel CPU)

Data layout

  • Single-sample tensor Tensor<T> (LinearAlgebra): width × height × channels (W×H×C), used by Sequential<T> / MLP heads.
  • Batch tensor BatchTensor<T>: N×H×W×C (NHWC) contiguous memory, used for CNN batch training.
  • Bridge: CnnMlpModel<T> converts backbone output N×1×1×F to head input 1×N×F.

📒Contents

Module architecture and training flow

Typical CNN classification pipeline:

  1. Assemble BatchSequential<float> from BatchConvolution2DLayer, BatchBatchNormLayer, pooling/activation layers, and so on.
  2. (Optional) BatchFlattenLayer + BatchFullyConnectedLayer + BatchSoftmaxLayer.
  3. Build a batch list with BatchLabelSample<float>(input, target).
  4. new Trainer<float>().FitBatchSequential(model, loss, optimizer, batches, epochs).
  5. ModelSerializer.SaveToFile(model, path) to persist weights and architecture metadata (v3).

Hybrid model: CnnMlpModel<T> = BatchSequential backbone + Sequential head, trained with Trainer.FitCnnMlp.

Dual branch: BatchParallelConcatModel<T> runs left and right branches on the same input in parallel and concatenates along the channel dimension, trained with Trainer.FitBatchParallelConcat.

Core: BatchTensor and model containers

BatchTensor<T>

Four-dimensional NHWC batch tensor; Values is a Span<T>. Main members:

  • Constructor: BatchTensor(batch, height, width, channels)
  • Indexer: this[n, h, w, c]
  • Shape → BatchShape; GetSample(int) exports a single-sample Tensor<T>

IBatchLayer<T> / BatchSequential<T>

Differentiable layer interface: Forward(BatchTensor, training), Backward(grad), Parameters, GetOutputShape.

BatchSequential<T> stacks multiple IBatchLayer<T> instances in order and provides ZeroGradients().

Sequential<T> / ILayer<T>

Single-sample layer stack; input/output are Tensor<T> (1×N×F feature vectors or W×H×C image tensors).

CnnMlpModel<T>

  • Backbone: BatchSequential<T> (NHWC)
  • Head: Sequential<T> (MLP)
  • End-to-end Forward / Backward / ZeroGradients

BatchParallelConcatModel<T>

  • Left / Right: two BatchSequential<T> branches
  • Forward: same input, channel-dimension Concat

Parameter<T>

Trainable weight/bias wrapper with Value, Gradient, ZeroGradient().

Modules: batch layer overview

The following types all implement IBatchLayer<T> (T is float, double, or another IBinaryFloatingPointIeee754):

Type Description
BatchConvolution2DLayer 2D convolution (im2col + GEMM)
BatchDepthwiseConvolution2DLayer Depthwise separable convolution
BatchTransposedConvolution2DLayer Transposed convolution / upsampling
BatchBatchNormLayer Batch normalization (NHWC)
BatchLayerNormLayer Layer normalization
BatchFullyConnectedLayer Fully connected (use with Flatten)
BatchFlattenLayer N×H×W×C → N×1×1×(HWC)
BatchMaxPool2DLayer / BatchAvgPool2DLayer Pooling
BatchGlobalAveragePool2DLayer Global average pooling
BatchUpsample2DLayer Nearest-neighbor 2× upsampling
BatchDropoutLayer Dropout (controlled by training flag)
BatchReLUActivation / BatchLeakyReLUActivation / BatchSigmoidActivation / BatchTanhActivation Activations
BatchSoftmaxLayer Softmax
BatchConcatenateLayer Channel-dimension concatenation (use BatchParallelConcatModel when dual inputs need external wiring)
BatchResidualBlockLayer Residual block
BatchSqueezeExciteLayer SE attention

Single-sample layers (ILayer<T>): FullyConnectedLayer, Convolution2DLayer, BatchNormLayer, ReLUActivation, SigmoidActivation, SoftmaxLayer, FlattenLayer, MaxPool2DLayer, and others, used by Sequential<T> and MLP heads.

Training: Trainer

Trainer<T> provides high-level training loops:

Method Model Data
Fit Sequential<T> TrainingSample<T> list
FitBatched Sequential<T> Batches packed as 1×N×F
FitCnnMlp CnnMlpModel<T> NHWC batch + labels
FitBatchSequential BatchSequential<T> BatchLabelSample<T> or BatchClassLabelSample<T> (sparse labels)
FitBatchParallelConcat BatchParallelConcatModel<T> BatchLabelSample<T>

Step APIs: TrainBatchSequentialStep, TrainBatchParallelConcatStep, and others for custom loops.

Learning-rate scheduling: ILearningRateScheduler<T> — optional callback at epoch start (also used by MlpTrainingOptions<T>).

Scheduler Behavior
ConstantLearningRateScheduler<T> Fixed learning rate
StepDecayScheduler<T> Multiply by DecayFactor every StepEpochs
ExponentialDecayScheduler<T> initial * exp(-decayRate * (epoch-1))
CosineAnnealingScheduler<T> Cosine decay from initial to minimum LR

BatchTrainingSample<T> — record struct for FitCnnMlp: NHWC Input + legacy 1×N×F Target tensor.

MlpTrainingOptions<T> — curve-fitting MLP training: OptimizerKind (Sgd/Adam), LR, momentum, Adam betas, optional LearningRateScheduler, RandomSeed, ComputingContext; CreateOptimizer() builds IOptimizer<T>.

MlpRegressor — static helpers to build/train small MLPs for CurveFitter<T> (see Experimental).

Fit, FitBatched, FitBatchSequential, FitBatchParallelConcat, and FitCnnMlp all accept an optional computingContext parameter (or use an outer ComputingScope). Batch layer forwards read the execution policy; see the computing context guide for details.

Losses and Optimizers

Losses

Type Interface Use case
MeanSquaredErrorLoss<T> ILoss<T> Regression (single sample)
CategoricalCrossEntropyLoss<T> ILoss<T> Classification (one-hot targets)
BatchCategoricalCrossEntropyLoss<T> ISparseBatchLoss<T> NHWC batch classification (integer class labels or one-hot)

IBatchLoss<T>: Compute(prediction, target) + Backward(...), gradient shape matches BatchTensor.

Optimizers

  • SgdOptimizer<T>: stochastic gradient descent (optional momentum)
  • AdamOptimizer<T>: Adam

Both implement IOptimizer<T>: ZeroGrad, Step(parameters).

Serialization: ModelSerializer v3

File magic VMDL, FormatVersion = 3. In addition to tensor weights, writes layer type IDs and hyperparameters (LayerDescriptor); on load, ModelArchitectureCatalog.VerifyMatches can validate structure.

Supported model kinds (ModelKind):

Kind Type
Sequential Sequential<T>
CnnMlp CnnMlpModel<T>
BatchSequential BatchSequential<T>
BatchParallelConcat BatchParallelConcatModel<T>

API example:

ModelSerializer.SaveToFile(model, "model.vmdl");
ModelSerializer.LoadFromFile(model, "model.vmdl"); // Model structure must be constructed with the same architecture first

ModelArchitectureCatalog.Describe(model) exports a layer descriptor list for debugging or unit tests.

ModelArchitectureBuilder — rebuild trainable models from LayerDescriptor lists:

Method Output
BuildBatchSequential<T>(layers) BatchSequential<T>
BuildCnnMlp<T>(layers) CnnMlpModel<T> (splits backbone.* / head.* prefixed names)
BuildBatchLayer<T>(descriptor) Single IBatchLayer<T>

LayerBase<T> / BatchLayerBase<T> — optional base classes for custom layers.

BatchTensorUtilities / TensorUtilities — shape checks, conversions, debug helpers.

Batch training code example

using Vorcyc.Mathematics.DeepLearning;
using Vorcyc.Mathematics.DeepLearning.Losses;
using Vorcyc.Mathematics.DeepLearning.Modules;
using Vorcyc.Mathematics.DeepLearning.Optimizers;
using Vorcyc.Mathematics.DeepLearning.Training;
using Vorcyc.Mathematics.DeepLearning.Serialization;

// Simple CNN: Conv → BN → ReLU → GAP → FC → Softmax
var model = new BatchSequential<float>(
    new BatchConvolution2DLayer<float>(3, 16, kernelSize: 3, stride: 1, name: "conv1"),
    new BatchBatchNormLayer<float>(16, name: "bn1"),
    new BatchReLUActivation<float>("relu1"),
    new BatchGlobalAveragePool2DLayer<float>("gap"),
    new BatchFullyConnectedLayer<float>(16, 10, name: "fc"),
    new BatchSoftmaxLayer<float>("softmax"));

var batches = new List<BatchLabelSample<float>>(); // Fill with NHWC input / target
var trainer = new Trainer<float>();
var loss = new BatchCategoricalCrossEntropyLoss<float>();
var optimizer = new AdamOptimizer<float>(learningRate: 1e-3f);

trainer.FitBatchSequential(model, loss, optimizer, batches, epochs: 10,
    onEpochEnd: (epoch, avgLoss) => Console.WriteLine($"epoch {epoch}: {avgLoss}"));

ModelSerializer.SaveToFile(model, "cnn.vmdl");

Signal processing integration

DeepLearning.Integration converts DSP Signal values to BatchTensor<float> for direct use with Trainer.FitBatchSequential. Dependency direction is DeepLearning → SignalProcessing (the signal module does not reference deep learning).

Type Layout Use case
Waveform vector batch N×1×1×L Raw sample sequences; length L on the channel dimension
Framed batch 1×F×L×1 Sliding-window framing; F is the frame count
Spectrogram batch N×T×F×1 STFT magnitude spectrogram; T is time frames, F is frequency bins 
using Vorcyc.Mathematics.DeepLearning.Integration;
using Vorcyc.Mathematics.DeepLearning.Training;
using Vorcyc.Mathematics.SignalProcessing.Signals;
using Vorcyc.Mathematics.SignalProcessing.Transforms;

var low = new Signal(256, 8000f);
low.GenerateWave(WaveShape.Sine, 200f);

var stft = new Stft(windowSize: 64, hopSize: 32);
var spec = SpectrogramBatchBuilder.FromSignal(low, stft);            // 1×T×F×1
var vectorBatch = BatchTensorSignalExtensions.FromSignal(low);       // 1×1×1×L
var restored = vectorBatch.ToSignal(0, 8000f);                     // Round-trip waveform

var sample = AudioTrainingSamples.SpectrogramClassification(
    signals: [low, highTone],
    classIndices: [0, 1],
    numClasses: 2,
    stft);

trainer.FitBatchSequential(model, loss, optimizer, [sample], epochs: 100);

Feature batching (Phase B)

FeatureBatchBuilder standardizes DSP feature extraction results as DL tensors:

DSP source Layout API
Stft.AveragePeriodogram N×1×1×F FromPeriodogram / FromPeriodograms
FeatureExtractor frame sequence N×T×F×1 FromExtractor / FromExtractors
Temporal mean of frame features N×1×1×F FromExtractorMean / FromExtractorMeans
Arbitrary equal-length vectors N×1×1×F FromVectors

AudioTrainingSamples provides training sample factories such as PeriodogramClassification, FeatureMeanClassification, and MfccSequenceClassification.

var mfcc = AudioTrainingSamples.CreateDefaultMfccExtractor(8000);
var sample = AudioTrainingSamples.FeatureMeanClassification(
    signals, labels, numClasses: 2, extractor: mfcc);

DSP frontend layers (Phase C)

Integration.Frontends wraps fixed DSP operations as IBatchLayer<float> layers that can be composed into BatchSequential:

Layer Input → Output Backpropagation
BatchPreEmphasisLayer N×1×1×L → N×1×1×L Differentiable (linear)
BatchStftMagnitudeLayer N×1×1×L → N×T×F×1 stop-gradient
BatchMelFilterbankLayer N×T×F×1 → N×T×M×1 Differentiable (fixed matrix) 
using Vorcyc.Mathematics.DeepLearning.Integration.Frontends;

var frontend = AudioFrontendLayers.CreateMelSpectrogramStack(
    samplingRate: 8000, windowSize: 64, hopSize: 32, melBands: 24);

var model = new BatchSequential<float>(
    frontend[0], frontend[1], frontend[2],
    new BatchConvolution2DLayer<float>(1, 16, kernelSize: 3),
    new BatchReLUActivation<float>(),
    new BatchGlobalAveragePool2DLayer<float>(),
    new BatchFullyConnectedLayer<float>(16, numClasses));

var waves = BatchTensorSignalExtensions.FromSignalVectors(signals);
trainer.FitBatchSequential(model, loss, optimizer,
    [new BatchLabelSample<float>(waves, targets)], epochs: 100);

Closed-loop applications (Phase D)

  • Experimental regression: CurveFitter<T>.NeuralNetwork(IReadOnlyList<Signal>, …) → SignalFitResult<T> (uses Integration + MlpRegressor internally).
  • Batch regression loss: BatchMeanSquaredErrorLoss<T>, paired with AudioTrainingSamples.WaveformRegression.
  • Documentation and examples: see Module_DeepLearning_Audio.md.

Legacy: Layers static class

The following are early non-autodiff parallel forward operators. New projects should prefer Modules + Trainer. Retained for compatibility and reference.

Vorcyc.Mathematics.DeepLearning.Layers.Layers class

Vorcyc.Mathematics.DeepLearning.Layers.Layers is a static class that implements neural network layers.

Methods

1. BatchNorm method

  • public static Tensor<T> BatchNorm<T>(Tensor<T> input, Tensor<T> mean, Tensor<T> variance, Tensor<T> shift, Tensor<T> scale) where T : IBinaryFloatingPointIeee754<T>

    • Applies batch normalization to the input tensor.
    • Parameters:
      • input: Input tensor.
      • mean: Mean tensor.
      • variance: Variance tensor.
      • shift: Shift tensor.
      • scale: Scale tensor.
    • Returns: Normalized tensor.

  • public static Tensor BatchNorm(Tensor input, Tensor mean, Tensor variance, Tensor shift, Tensor scale)

    • Applies batch normalization to the input tensor.
    • Parameters:
      • input: Input tensor.
      • mean: Mean tensor.
      • variance: Variance tensor.
      • shift: Shift tensor.
      • scale: Scale tensor.
    • Returns: Normalized tensor.

2. Conv2D method

  • public static Tensor<T> Conv2D<T>(Tensor<T> input, Tensor<T>[] filters, Tensor<T> biases, int stride = 1, int dilation = 1) where T : IBinaryFloatingPointIeee754<T>

    • Performs convolution on a 2D tensor.
    • Parameters:
      • input: Input 2D tensor.
      • filters: Array of convolution kernels.
      • biases: Bias tensor.
      • stride: Convolution stride; default is 1.
      • dilation: Convolution dilation rate; default is 1.
    • Returns: Convolution result tensor.

  • public static Tensor Conv2D(Tensor input, Tensor[] filters, Tensor biases, int stride = 1, int dilation = 1)

    • Performs convolution on a 2D tensor.
    • Parameters:
      • input: Input 2D tensor.
      • filters: Array of convolution kernels.
      • biases: Bias tensor.
      • stride: Convolution stride; default is 1.
      • dilation: Convolution dilation rate; default is 1.
    • Returns: Convolution result tensor.

3. Dense method

  • public static Tensor<T> Dense<T>(Tensor<T> input, Tensor<T>[] weights, Tensor<T> biases) where T : IBinaryFloatingPointIeee754<T>

    • Applies a dense (fully connected) layer to the input tensor.
    • Parameters:
      • input: Input tensor.
      • weights: Array of weight tensors.
      • biases: Bias tensor.
    • Returns: Tensor after the dense operation.

  • public static Tensor Dense(Tensor input, Tensor[] weights, Tensor biases)

    • Applies a dense (fully connected) layer to the input tensor.
    • Parameters:
      • input: Input tensor.
      • weights: Array of weight tensors.
      • biases: Bias tensor.
    • Returns: Tensor after the dense operation.

4. Fusion method

  • public static Tensor<T> Fusion<T>(Tensor<T> input, Tensor<T> joint) where T : IBinaryFloatingPointIeee754<T>

    • Merges two tensors.
    • Parameters:
      • input: Input tensor.
      • joint: Tensor to merge.
    • Returns: Merged tensor.

  • public static Tensor Fusion(Tensor input, Tensor joint)

    • Merges two tensors.
    • Parameters:
      • input: Input tensor.
      • joint: Tensor to merge.
    • Returns: Merged tensor.

5. JoinLayer method

  • public static Tensor<T> JoinLayer<T>(Tensor<T> input, Tensor<T> joint) where T : IBinaryFloatingPointIeee754<T>

    • Joins two tensor layers.
    • Parameters:
      • input: Input tensor.
      • joint: Tensor to join.
    • Returns: Joined tensor.

  • public static Tensor JoinLayer(Tensor input, Tensor joint)

    • Joins two tensor layers.
    • Parameters:
      • input: Input tensor.
      • joint: Tensor to join.
    • Returns: Joined tensor.

6. LinearLayer method

  • public static Tensor<T> LinearLayer<T>(Tensor<T> input, Tensor<T>[] weights, Tensor<T> biases) where T : IBinaryFloatingPointIeee754<T>

    • Applies a linear (fully connected) layer to the input tensor.
    • Parameters:
      • input: Input tensor.
      • weights: Array of weight tensors.
      • biases: Bias tensor.
    • Returns: Tensor after the linear operation.

  • public static Tensor LinearLayer(Tensor input, Tensor[] weights, Tensor biases)

    • Applies a linear (fully connected) layer to the input tensor.
    • Parameters:
      • input: Input tensor.
      • weights: Array of weight tensors.
      • biases: Bias tensor.
    • Returns: Tensor after the linear operation.

7. MaxPool2D method

  • public static Tensor<T> MaxPool2D<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>, IMinMaxValue<T>

    • Applies 2D max pooling to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Tensor after max pooling.

  • public static Tensor MaxPool2D(Tensor input)

    • Applies 2D max pooling to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Tensor after max pooling.

8. ReLU method

  • public static Tensor<T> ReLU<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>

    • Applies the ReLU activation function to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Tensor after ReLU.

  • public static Tensor ReLU(Tensor input)

    • Applies the ReLU activation function to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Tensor after ReLU.

9. Sigmoid method

  • public static Tensor<T> Sigmoid<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>

    • Applies the Sigmoid activation function to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Tensor after Sigmoid.

  • public static Tensor Sigmoid(Tensor input)

    • Applies the Sigmoid activation function to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Tensor after Sigmoid.

10. Upsample2D method

  • public static Tensor<T> Upsample2D<T>(Tensor<T> input) where T : IBinaryFloatingPointIeee754<T>

    • Applies 2D upsampling to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Upsampled tensor.

  • public static Tensor Upsample2D(Tensor input)

    • Applies 2D upsampling to the input tensor.
    • Parameters:
      • input: Input tensor.
    • Returns: Upsampled tensor.

Code example

The following example uses several methods from the Layers class, with comments in the sample:

using System;
using Vorcyc.Mathematics.DeepLearning.Layers;
using Vorcyc.Mathematics.LinearAlgebra;

public class LayersExample
{
    public static void Main()
    {
        // Define input tensor
        var input = new Tensor<float>(new float[,,] 
        { 
            { 
                { 1.0f, 2.0f }, 
                { 3.0f, 4.0f } 
            }, 
            {
                { 5.0f, 6.0f }, 
                { 7.0f, 8.0f }
            } 
        });
        // Define mean, variance, shift, and scale tensors
        var mean = new Tensor<float>(new float[] { 1.0f, 2.0f });
        var variance = new Tensor<float>(new float[] { 1.0f, 1.0f });
        var shift = new Tensor<float>(new float[] { 0.0f, 0.0f });
        var scale = new Tensor<float>(new float[] { 1.0f, 1.0f });

        // Apply batch normalization
        var normalized = Layers.BatchNorm(input, mean, variance, shift, scale);
        Console.WriteLine("BatchNorm Result:");
        PrintTensor(normalized);

        // Define convolution kernels and bias tensor
        var filters = new Tensor<float>[]
        {
        new Tensor<float>(new float[,,] { { { 1.0f, 0.0f }, { 0.0f, 1.0f } } }),
        new Tensor<float>(new float[,,] { { { 0.0f, 1.0f }, { 1.0f, 0.0f } } })
        };
        var biases = new Tensor<float>(new float[] { 0.0f, 0.0f });

        // Apply convolution
        var convResult = Layers.Conv2D(input, filters, biases);
        Console.WriteLine("Conv2D Result:");
        PrintTensor(convResult);

        // Define weight and bias tensors
        var weights = new Tensor<float>[]
        {
        new Tensor<float>(new float[,,] { { { 1.0f, 0.0f }, { 0.0f, 1.0f } } }),
        new Tensor<float>(new float[,,] { { { 0.0f, 1.0f }, { 1.0f, 0.0f } } })
        };
        var denseBiases = new Tensor<float>(new float[] { 0.0f, 0.0f });

        // Apply dense layer
        var denseResult = Layers.Dense(input, weights, denseBiases);
        Console.WriteLine("Dense Result:");
        PrintTensor(denseResult);

        // Apply ReLU activation
        var reluResult = Layers.ReLU(input);
        Console.WriteLine("ReLU Result:");
        PrintTensor(reluResult);

        // Apply Sigmoid activation
        var sigmoidResult = Layers.Sigmoid(input);
        Console.WriteLine("Sigmoid Result:");
        PrintTensor(sigmoidResult);
    }

    private static void PrintTensor(Tensor<float> tensor)
    {
        for (int d = 0; d < tensor.Depth; d++)
        {
            for (int y = 0; y < tensor.Height; y++)
            {
                for (int x = 0; x < tensor.Width; x++)
                {
                    Console.Write($"{tensor[x, y, d]} ");
                }
                Console.WriteLine();
            }
            Console.WriteLine();
        }
    }
}

Language / 语言: English · 简体中文

Home

首页

Clone this wiki locally