Audio Processing Pipeline
This page follows the data from decoded waveform samples to the feature tensor consumed by the scorer model.
transcribe_audio_file first calls SoundFile with float32 output and an explicit samples-by-channels shape. If that fails, audioread supplies signed 16-bit PCM chunks. A2M validates the sample rate and dimensions, replaces non-finite values, and uses soxr when resampling is needed.
The original duration is retained so final MIDI events can be clipped to real recording time rather than padded model time.
For channel signal x, each section is standardized before windowing:
z[n] = (x[n] - μ) / (σ + ε)
Frames begin every H = 1,024 samples and contain N = 4,096 samples. At 44,100 Hz, the frame step is:
Δt = H / fₛ = 1024 / 44100 ≈ 23.22 ms
Six stored analysis windows provide complementary spectral views of each frame.
For frame m, frequency bin k, and analysis window w:
Pₘ,ₖ,w = |RFFT(zₘ[n] · Ww[n])|²
The real Fourier transform represents how strongly different frequencies are present. Squared magnitude gives power; phase is not passed to the transcription model.
The stored mel filter matrix groups FFT bins into 229 bands between 30 and 8,000 Hz:
Eₘ,b,w = Σₖ M[b,k] · Pₘ,ₖ,w
Channel contributions are averaged before the final transform. Mel spacing uses finer resolution at lower frequencies and coarser grouping higher up, producing a compact representation suited to the model.
With ε = 10⁻⁵, A2M applies the bundle-compatible mapping:
F = [ln(E + ε) - ln(ε)] / [-ln(ε)]
This compresses the large power range while preserving relative structure. The contiguous float32 tensor is passed to the scorer as mel_features.
| Property | Value |
|---|---|
| Sample rate | 44,100 Hz |
| Hop / frame step | 1,024 samples / 23.22 ms |
| FFT/window size | 4,096 samples |
| Analysis windows | 6 |
| Mel bands | 229 |
| Frequency range | 30 to 8,000 Hz |
| Section length / nominal hop | 16 s / 8 s |
These values come from the current model manifest and frontend data.
The audio is padded on both sides by the difference between section length and section movement, which is currently eight seconds. Section starts are rounded upward to a whole model step, and the final short section is padded with zeros. Overlap gives events near a handoff surrounding context. Decoder state is carried forward to help preserve continuity.
Source: A2M/a2m/core/piano_engine.py, especially transcribe_audio_file, transcribe_audio_array, _make_frames, and _extract_features.