Module_DeepLearning_Audio

Location: Home / Deep learning module / Audio integration

Deep learning × signal processing integration

This document summarizes the full Signal → BatchTensor → Trainer pipeline (phases A–D). Primary APIs live in Vorcyc.Mathematics.DeepLearning.Integration and Integration.Frontends.

Data layout (NHWC)

Use	Shape	Typical entry
Waveform batch	N×1×1×L	BatchTensorSignalExtensions.FromSignalVectors
Spectrogram batch	N×T×F×1	SpectrogramBatchBuilder.FromSignals
Periodogram / feature vector	N×1×1×F	FeatureBatchBuilder.FromPeriodograms / FromExtractorMeans
Frame feature sequence	N×T×F×1	FeatureBatchBuilder.FromExtractors
Classification labels	N×1×1×C	AudioTrainingSamples.CreateOneHotTargets
Regression scalar	N×1×1×1	AudioTrainingSamples.CreateScalarTargets

Dependency direction: DeepLearning → SignalProcessing (DSP core does not reference DL).

Phase A: bridging and training samples

var sample = AudioTrainingSamples.SpectrogramClassification(
    signals, labels, numClasses: 2, stft);

trainer.FitBatchSequential(model, loss, optimizer, [sample], epochs: 100);

Phase B: feature packing

var mfcc = AudioTrainingSamples.CreateDefaultMfccExtractor(8000);
var features = FeatureBatchBuilder.FromExtractorMeans(signals, mfcc); // N×1×1×13

Phase C: network-embeddable DSP frontends

var stack = AudioFrontendLayers.CreateMelSpectrogramStack(8000, 64, 32, melBands: 24);
var model = new BatchSequential<float>(stack[0], stack[1], stack[2], /* CNN head */);

• BatchStftMagnitudeLayer backward is stop-gradient (CNN head only).
• BatchPreEmphasisLayer and BatchMelFilterbankLayer support gradients (linear).

Phase D: experimental CurveFitter and regression

CurveFitter<T>.NeuralNetwork now accepts Signal input, packs via Integration internally, then calls MlpRegressor:

var options = new SignalNeuralNetworkOptions
{
    FeatureMode = SignalNeuralNetworkFeatureMode.Periodogram,
    Stft = new Stft(64, 32)
};

var fit = CurveFitter<float>.NeuralNetwork(signals, targets, signalOptions: options);
var y = fit.Predict(probeSignal);

Waveform → scalar regression (BatchSequential):

var sample = AudioTrainingSamples.WaveformRegression(signals, scalarTargets);
trainer.FitBatchSequential(model, new BatchMeanSquaredErrorLoss<float>(), optimizer, [sample], epochs);

Test index

Test	Description
SignalBatchTensor_test	Waveform roundtrip
SpectrogramBatch_training_test	Spectrogram CNN classification
FeatureBatchBuilder_test	Feature shapes
MfccMean_training_test	MFCC mean + MLP
AudioFrontend_training_test	Waveform → frontend → CNN
CurveFitterSignalNeuralNetwork_test	Periodogram-feature frequency regression
WaveformScalarRegression_test	Waveform → amplitude regression

See Module_DeepLearning.md for the full module manual.