Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client disconnects , and refreshes after the audio stream is recorded and sent to the server #14

Open
qxprakash opened this issue Apr 7, 2024 · 3 comments
Open

Client disconnects , and refreshes after the audio stream is recorded and sent to the server #14

qxprakash opened this issue Apr 7, 2024 · 3 comments

Comments

@qxprakash
Copy link

hello @alesaccoia , thanks for the work on VoiceStreamAI , I was trying to make this work but , after setting the config and successfully establishing the websocket connection , when I click on start streaming , once the audio is recorded and sent to the server my client refreshes and I get Connection with ad63d0ab-a123-476a-8d4e-83b2b02d24c4 closed: received 1001 (going away); then sent 1001 (going away) in my server logs , I modified processAudio and convertFloat32ToInt16 funciton inside utils.js with more logs and error handling to better understand what's happening ,

here is my implementation of 2 functions

function processAudio(e) {
  try {
    console.log("processAudio function called");

    if (!context) {
      throw new Error("Audio context is not initialized.");
    }

    const inputSampleRate = context.sampleRate;
    console.log(`Input sample rate: ${inputSampleRate}`);

    const outputSampleRate = 16000; // Target sample rate
    console.log(`Output sample rate: ${outputSampleRate}`);

    const left = e.inputBuffer.getChannelData(0);
    const downsampledBuffer = downsampleBuffer(
      left,
      inputSampleRate,
      outputSampleRate
    );

    const audioData = convertFloat32ToInt16(downsampledBuffer);

    if (!websocket) {
      throw new Error("WebSocket is not initialized.");
    }

    if (websocket.readyState === WebSocket.OPEN) {
      websocket.send(audioData);
      console.log(`Audio data sent: ${audioData.byteLength} bytes`);
    } else {
      console.log(
        "WebSocket is not open. Current state: " + websocket.readyState
      );
    }
  } catch (error) {
    console.error("Error in processAudio:", error);
  }
}


function convertFloat32ToInt16(buffer) {
  try {
    console.log("convertFloat32ToInt16 function called");

    if (!buffer) {
      throw new Error("Input buffer is null or undefined.");
    }

    let l = buffer.length;
    console.log(`Input buffer length: ${l}`);

    const buf = new Int16Array(l);
    while (l--) {
      // Clamp the float value within the [-1.0, 1.0] range before conversion
      const clampedValue = Math.max(-1, Math.min(1, buffer[l]));
      buf[l] = clampedValue * 0x7fff;
    }

    console.log(`Converted Int16 buffer length: ${buf.length}`);
    return buf.buffer;
  } catch (error) {
    console.error("Error in convertFloat32ToInt16:", error);
    // Return an empty ArrayBuffer on error to maintain function signature consistency
    return new ArrayBuffer(0);
  }
}



here are my console logs 



```utils.js:18 Attempting to initialize WebSocket connection...
utils.js:32 WebSocket connection established
utils.js:105 start recording function called 
utils.js:106 recording function called --------------------------------------------
utils.js:113 starting recording
utils.js:124 [Deprecation] The ScriptProcessorNode is deprecated. Use AudioWorkletNode instead. (https://bit.ly/audio-worklet)
(anonymous) @ utils.js:124
Promise.then (async)
startRecording @ utils.js:121
onclick @ VoiceStreamAI_Client.html:125
utils.js:166 send audio config function called 
utils.js:167 Sending audio configuration to server...
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
utils.js:245 processAudio function called
utils.js:252 Input sample rate: 44100
utils.js:255 Output sample rate: 16000
utils.js:200 downsampling buffer function called 
utils.js:295 convertFloat32ToInt16 function called
utils.js:302 Input buffer length: 1486
utils.js:311 Converted Int16 buffer length: 1486
utils.js:272 Audio data sent: 2972 bytes
Navigated to http://127.0.0.1:5500/client/VoiceStreamAI_Client.html
Navigated to http://127.0.0.1:5500/client/VoiceStreamAI_Client.html




as you can see in the end the client navigates back to the same url
@alesaccoia
Copy link
Owner

Hello! I checked the error and sincerely I don't see anything that could point me to a solution.
Did you try on another browser, and on another PC?

@qxprakash
Copy link
Author

Did you also faced this error or everything is working fine ? I even tried with the docker image got the same result , not sure what I'm doing wrong

@alesaccoia
Copy link
Owner

It's working fine here on several setups

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alesaccoia @qxprakash