Audio Sources?

[radry]

What audio sources can this capture? Only the microphone?
Can it capture audio from other software, for example Zoom or MS Teams?

[CorentinvdBdO]

You can send audio bytes from any source. For example I read TV streams, extract the audio and send it to the server that then sends back live subtitles.

[radry]

@CorentinvdBdO
How are you feeding audio to the server?
Is it possible to pipe an audio source to the server via terminal or is it necessary to use the python API?

[CorentinvdBdO]

You need to open a websocket connection with the server, you link to it actions when it opens, closes and receive messages:

ws = ws_client.WebSocketApp(
                url,
                on_open=self.on_open,
                on_message=self.on_message,
                on_error=self.on_error,
                on_close=self.on_close
            )

Then you keep sending audio data:

                audio_b64 = base64.b64encode(audio_data).decode("utf-8")
                message = {
                    "type": "audio_chunk",
                    "data": {"chunk": audio_b64}
                }
               
                self.ws.send(json.dumps(message))

And then you keep receiving messages on the other end:

    def on_message(self, ws, message):
        try:
            content = json.loads(message)
            log_message(content, RECEIVED_LOG_FILE)
            if content["type"] == "transcript":
                self.transcript_queue.put(content)
                if 'message_trigger' in st.session_state:
                    st.session_state.message_trigger = time.time()
            elif content["type"] == "translation":
                self.translation_queue.put(content)
                if 'message_trigger' in st.session_state:
                    st.session_state.message_trigger = time.time()

I'm using a custom server so it will not work like this with the basic repository. I don't think I'll have the time to publish proper standalone examples. To be honest I'm really new to using webrtc. I am pretty sure you need to build a custom ingestion pipeline, it wouldn't read a stream if you send one. (doesn't have to be python either, any language would do, even embedded in a simple html front end would do).

[radry]

@CorentinvdBdO Thank you. I see now, it's necessary to create an own application. I thought this repository was a usable standalone server/backend that we can interact with but it seems it's barely a proof of concept and a library that needs to be implemented elsewhere.

[QuentinFuxa]

@CorentinvdBdO Thank you. I see now, it's necessary to create an own application. I thought this repository was a usable standalone server/backend that we can interact with but it seems it's barely a proof of concept and a library that needs to be implemented elsewhere.

This repo contains a standalone backend server, AND an optional frontend for input mic OR chrome extension for tab audio, that can be connected to that backend. if you have other usages you have to dev your own frontend, that will send the websocket stream to the backend

[radry]

@QuentinFuxa I still don't understand fully how do access that backend server other than through the provided web frontend. Is it only possible through the python API as shown in the readme or does it expose a web API or something else? I would appreciate if you could show an example.