Transcribing live stream calls from Twilio in python

[dquang-tran]

Hi team,

Love the library — amazing job!

I am currently attempting to implement live streaming audio transcription.
This article gives an example of transcribing from live streaming audio: https://developers.deepgram.com/docs/getting-started-with-live-streaming-audio

I would like to implement something similar but with Twilio audio as the source. However, I'm currently running into the issue where TRANSCRIPT_RECEIVED event handler doesn't trigger: https://replit.com/join/tfgzvbjuws-quangtran73

Curious if I'm doing something wrong and whether there are tips to making this work?

[nikolawhallon]

This is a guide for streaming live Twilio calls to Deepgram in Python: https://deepgram.com/learn/deepgram-twilio-streaming

It uses Twilio's streaming API, which will fork Twilio audio to your server, which then processes the audio, sends it to Deepgram via websockets, and gets the live streaming transcription results. Can you share more about what you have tried? My first guess is that, Twilio media messages can't be directly parsed by Deepgram, they are streamed as json text, where the audio payloads are base64-encoded mulaw, so some processing of Twilio's media messages needs to happen before sending the audio to Deepgram (that guide should cover this).

[dquang-tran]

Can you share a request id at this point too? If the transcript is gibberish, it's likely the audio is still gibberish/corrupt/not formatted well for some reason, see the below point too:

Yeah! It's 2419dccc-5a23-4c24-bf6a-6564acfe06b1

Twilio can drop packets, and when it does audio will come in with timestamps that seem to skip, like 0, 20, 40, 80, 100 (notice the 60-80ms audio was skipped)

Gotcha, copied your logic over to handle this case.

Last question, just to cover the bases, can you share your TwiML Bin (minus whatever information in it might be sensitive)?

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Start>
        <Stream url="wss://[NGROK_SERVER]/media" track="both_tracks"/>
     </Start>
     <Dial>[PHONE_NUMER]</Dial>
</Response>

^ Was partially following https://www.twilio.com/docs/voice/tutorials/consume-real-time-media-stream-using-websockets-python-and-flask

Will check out the test suite as well

[nikolawhallon]

Hold that thought, we're good. I checked 2419dccc-5a23-4c24-bf6a-6564acfe06b1 and it looks great, I get transcripts like:

one
two
three
four five
six
one two three
hello
one two
three four five
seven
eight
...

Your current issue is that you have:

      'multichannel': True,
      'channels': 2,

but you are only sending the inbound channel, so Deepgram is interpreting that single channel of audio as two channels, chopping it up, and probably producing terrible transcripts on the chopped up audio. Given you have track="both_tracks" in your TwiML Bin, I imagine you want to transcribe both channels - great - but before doing that, can you just turn multichannel off and set 'channels': 1 to make sure you can get to the point where you are getting transcripts like I pasted above?

[nikolawhallon]

(sending both inbound an outbound channels is tricky - Twilio streams them separately, but you need to stitch them together into valid stereo audio - the Twilio blog post goes over this, but I'd recommending trying that out after making sure you are getting valid transcripts for just the one channel first)

[dquang-tran]

That fixed it! Yeah, I initially had it multi channel because I eventually plan to take input from both, but was only feeding inbound for now for testing purposes. It's a bit counter-intuitive that it would result in such vastly different output.

Really appreciate your help!

[dquang-tran]

Will be happy to contribute the code as sample when I have time — feels like this is a fairly common use case