Native Audio Analyzer

Native Audio Analyzer

native/ is an N-API addon named orchard_audio_analysis. It does the audio work that JavaScript should not: feature extraction, spectrograms, time-stretching, and offline transition rendering.

Building

npm run build:native

That runs node-gyp rebuild --directory native. Cross-compilation helpers:

npm run build:native:windows
npm run build:native:macos:cross

C++17 is required. The addon is built with NAPI_DISABLE_CPP_EXCEPTIONS, since Orchard's own native code never throws.

Vendored Rubber Band

Time-stretching uses Rubber Band, vendored under native/vendor/rubberband and licensed GPL-2.0-or-later. This is one of the reasons Orchard is AGPL from 4.0.0 onward. See License and Legal.

Rubber Band ships a single-file build unit, single/RubberBandSingle.cpp, which #includes the rest of the library itself. Only that one file appears in binding.gyp.

Rubber Band does throw internally, unlike Orchard's own code, so exceptions have to work at the compiler level even though NAPI_DISABLE_CPP_EXCEPTIONS is set:

• On non-Windows, -fno-exceptions is removed and -fexceptions added.
• On Windows, ExceptionHandling: 1 is set explicitly, because node-addon-api sometimes suppresses it alongside NAPI_DISABLE_CPP_EXCEPTIONS.
• On macOS, RubberBandSingle.cpp selects the vDSP FFT, so the Accelerate framework is linked.

native/transition/rubberband_stretch.cpp catches anything that reaches it rather than letting an exception escape across the worker-thread boundary.

Exported functions

Export	Purpose
analyze	Full track analysis
beatSpectrogram	Log-mel input for the beat model
vocalSpectrogram	Input for the vocal-separation model
timeStretch	Rubber Band time-stretch
renderTransition	Offline render of a complete transition overlap
analysisVersion	Version stamp on every analysis result

All heavy work runs on AsyncWorkers off the environment thread. The PCM snapshot and its result are owned by the worker until OnOK runs.

What analyze returns

Analysis produces one object per track. Grouped by what it describes:

Tempo and grid

bpm, beatInterval, firstBeat, beatConfidence, beats[], downbeats[], phraseBoundaries[]

Harmony

key, keyConfidence, chroma[]

Structure

audibleStartTime, pickupTime, pickupConfidence, mixInTime, mixInConfidence, introEndTime, outroStartTime, contentEndTime, mixOutTime, phrases[] (each with start, end, type, confidence), mixInCandidates[] and mixOutCandidates[] (each with time, score, type)

Loudness

loudnessLufs, peakDbfs, dynamicRangeDb

Content

vocalProbability, instrumentalProbability, vocalActivityMask[]

Energy

energyCurve[], plus band-split lowEnergyCurve[], midEnergyCurve[], highEnergyCurve[], each an array of { time, energy }

Every result carries analysisVersion and duration. The version stamp is what lets Orchard invalidate stored analyses when the analyzer changes, so a cached result from an older build is never silently trusted by newer policy code.

Note the shape that Smart Crossfade relies on: confidences travel with the values they qualify. beatConfidence, keyConfidence, mixInConfidence, and per-phrase confidence all exist so downstream policy can grade its own ambition rather than treating every number as fact.

Model inference

Two models run outside the addon, in separate child processes so a crash cannot take down playback:

Model	Purpose	Host
Beat This!	Beat and downbeat tracking	electron/audio/beatModelHost.js
open-unmix	Vocal presence	electron/audio/vocalMaskHost.js

The addon computes each model's spectrogram input (beatSpectrogram, vocalSpectrogram), the child process runs inference, and the results merge back into the analysis. Process lifecycle is managed by electron/audio/modelProcessHost.js.

Fetch the beat model with:

npm run fetch:beat-model

See mobile/docs/BEAT_MODEL.md for the model and its quantization.

Essentia is used for an independent beat-confidence cross-check (electron/audio/essentiaBeatConfidence.js).

Transition rendering

renderTransition produces the entire overlap between two tracks as one finished stereo buffer: time-stretch, beat alignment, filter sweep, and bass handover baked in. The renderer enforces kMaxTransparentRatioDeviation in native/transition/wsola.h, the four percent stretch limit mirrored in JavaScript as MAX_STRETCH_DEVIATION.

Doing this offline is what lets the renderer play a beat-matched transition as a scheduling problem rather than a real-time DSP problem. See Smart Crossfade.

Testing

npm run test:native

test/nativeAudioAnalysis.test.js exercises the addon directly.

A measurement note

When measuring analysis output against reference material, sample rate matters more than it looks like it should. Analysis and reference must be compared at the same rate, or the numbers do not mean what they appear to mean.

Source map

Path	Role
native/binding/addon.cpp	N-API surface and AsyncWorkers
native/analyzer/audio_analysis.cpp	Full-track analysis
native/analyzer/tempo_analysis.cpp	Tempo and beat grid
native/analyzer/mel_spectrogram.cpp	Beat model input
native/analyzer/vocal_spectrogram.cpp	Vocal model input
native/transition/transition_render.cpp	Offline overlap render
native/transition/rubberband_stretch.cpp	Rubber Band wrapper
native/binding.gyp	Build configuration
electron/audio/	JavaScript hosts and services