MediaStream linked to a MediaRecorder without an AudioContext in between results in a runaway recording

[CallMeMSL]

Hi there,
thank you for your work on this great library!

I want to use this library for a Uni Project where I do live transcription and analyze spoken sentences with an LLM.
Somehow recording the mic doesn't seem to work.
This code produces an 11 Minute .wav file that blasts your ears away with garbage:

fn main() {
    let mic = media_devices::get_user_media_sync(MediaStreamConstraints::Audio);
    let recorder = MediaRecorder::new(&mic);
    let recording: Arc<Mutex<Vec<u8>>> = Default::default();
    let arc = Arc::clone(&recording);
    recorder.set_ondataavailable(move |mut event| {
        let mut r = arc.lock().unwrap();
        r.append(&mut event.blob);
    });
    recorder.start();
    std::thread::sleep(std::time::Duration::from_secs(4));
    recorder.stop();
    std::thread::sleep(std::time::Duration::from_secs(1));
    let data = recording.lock().unwrap();
    let mut file = fs::OpenOptions::new()
        .create(true)
        .write(true)
        .open("unprocessed.wav").unwrap();
    file.write_all(&data).unwrap();
}

Is there something obvious that I am missing? The example for get_user_media_sync for the docs works without any problems and I can hear myself clearly and without any latency.

I am using Linux with PipeWire.

Kind Regards,
David

[orottier]

Thanks for the report. This is very interesting since you are combining the MediaDevices API and the MediaRecorder API, without an AudioContext in between. This is of course fine on the web, but not something we have really accounted for in our implementation - which main focus is the Web Audio API.

Tacking an AudioContext in between makes it work:

use std::sync::{Arc, Mutex};
use web_audio_api::context::{AudioContext, AudioContextLatencyCategory, AudioContextOptions};
use web_audio_api::media_devices;
use web_audio_api::media_devices::MediaStreamConstraints;
use web_audio_api::media_recorder::MediaRecorder;
use web_audio_api::node::AudioNode;

// If you are on Linux and use ALSA as audio backend backend, you might want to run
// the example with the `WEB_AUDIO_LATENCY=playback ` env variable which will
// increase the buffer size to 1024
//
// `WEB_AUDIO_LATENCY=playback cargo run --release --example recorder
fn main() {
    let latency_hint = match std::env::var("WEB_AUDIO_LATENCY").as_deref() {
        Ok("playback") => AudioContextLatencyCategory::Playback,
        _ => AudioContextLatencyCategory::default(),
    };

    let context = AudioContext::new(AudioContextOptions {
        latency_hint,
        ..AudioContextOptions::default()
    });

    // connect the mic via an AudioContext media_stream_source to an media_stream_destination
    let mic = media_devices::get_user_media_sync(MediaStreamConstraints::Audio);
    let stream_source = context.create_media_stream_source(&mic);
    let dest = context.create_media_stream_destination();
    stream_source.connect(&dest);

    // record the media_stream_destination
    let recorder = MediaRecorder::new(dest.stream());
    let recording: Arc<Mutex<Vec<u8>>> = Default::default();
    let arc = Arc::clone(&recording);
    recorder.set_ondataavailable(move |mut event| {
        let mut r = arc.lock().unwrap();
        r.append(&mut event.blob);
    });
    recorder.start();
    std::thread::sleep(std::time::Duration::from_secs(4));
    recorder.stop();
    std::thread::sleep(std::time::Duration::from_secs(1));
    let data = recording.lock().unwrap();
    let mut file = std::fs::OpenOptions::new()
        .create(true)
        .write(true)
        .open("unprocessed.wav")
        .unwrap();
    file.write_all(&data).unwrap();
}

You might wonder what's happening in your example and why it produces a large wav with garbage. The AudioContext in our library is responsible for timing of the render quantums to produce e.g. 48000 samples per second. We have taken some shortcuts in implementing the MediaStream API that results in them supplying samples on demand, instead of having an internal clock themselves. When you hook the mic directly to the recorder, the recorders polls for samples continuously. The stream is trying to provide them ASAP, and inserting silent frames whenever the underlying source (mic in this case) cannot provide them. Since there is no AudioContext clock to tame the recorder polling, the result is a giant wav file (the faster your computer, the larger the file). The noise is because it is mixing true signal with silent gaps.

In any case this is something we need to address. For now remember to actually use the Web Audio API when using our library :)

NB: the examples/recorder.rs file has a somewhat better implementation than your 1000ms delay to flush the recorder.
NB2: some audio players (e.g. iTunes) don't display the length of the wav files correctly in case of 'streaming wavs'. Don't let this confuse you, it plays just fine