Module_DeepLearning_Audio

Location: Home / Deep learning module / Audio integration

Deep learning × signal processing integration

This document summarizes the full Signal → BatchTensor → Trainer pipeline (phases A–D). Primary APIs live in Vorcyc.Mathematics.DeepLearning.Integration and Integration.Frontends.

Data layout (NHWC)

Use	Shape	Typical entry
Waveform batch	N×1×1×L	BatchTensorSignalExtensions.FromSignalVectors
Spectrogram batch	N×T×F×1	SpectrogramBatchBuilder.FromSignals
Periodogram / feature vector	N×1×1×F	FeatureBatchBuilder.FromPeriodograms / FromExtractorMeans
Frame feature sequence	N×T×F×1	FeatureBatchBuilder.FromExtractors
Classification labels	N×1×1×C	AudioTrainingSamples.CreateOneHotTargets
Regression scalar	N×1×1×1	AudioTrainingSamples.CreateScalarTargets

Dependency direction: DeepLearning → SignalProcessing (DSP core does not reference DL).

Phase A: bridging and training samples

var sample = AudioTrainingSamples.SpectrogramClassification(
    signals, labels, numClasses: 2, stft);

trainer.FitBatchSequential(model, loss, optimizer, [sample], epochs: 100);

Phase B: feature packing

var mfcc = AudioTrainingSamples.CreateDefaultMfccExtractor(8000);
var features = FeatureBatchBuilder.FromExtractorMeans(signals, mfcc); // N×1×1×13

Phase C: network-embeddable DSP frontends

var stack = AudioFrontendLayers.CreateMelSpectrogramStack(8000, 64, 32, melBands: 24);
var model = new BatchSequential<float>(stack[0], stack[1], stack[2], /* CNN head */);

• BatchStftMagnitudeLayer backward is stop-gradient (CNN head only).
• BatchPreEmphasisLayer and BatchMelFilterbankLayer support gradients (linear).

Phase D: neural regression and waveform regression

For neural-network curve fitting, use MlpRegressor (DeepLearning.Training) — see Examples/DeepLearning_example/CurveFitDemo.cs. For waveform → scalar regression, build a BatchSequential and train with a batch MSE loss:

var sample = AudioTrainingSamples.WaveformRegression(signals, scalarTargets);
trainer.FitBatchSequential(model, new BatchMeanSquaredErrorLoss<float>(), optimizer, [sample], epochs: 100);

Test index

Test	Description
SignalBatchTensor_test	Waveform roundtrip
SpectrogramBatch_training_test	Spectrogram CNN classification
FeatureBatchBuilder_test	Feature shapes
MfccMean_training_test	MFCC mean + MLP
AudioFrontend_training_test	Waveform → frontend → CNN
WaveformScalarRegression_test	Waveform → amplitude regression

See Module_DeepLearning.md for the full module manual.