Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure Twilio to record voicemail and send it by email #6

Closed
eric-brechemier opened this issue Mar 14, 2019 · 8 comments
Closed
Assignees

Comments

@eric-brechemier
Copy link
Owner

eric-brechemier commented Mar 14, 2019

When the software phone is closed on my computer, or when I am away and unable to answer a call, I wish to direct the caller to voicemail.

A notification of the call including the date/time, the phone number of the caller and a link to the recording should then be sent to me by email.

@eric-brechemier eric-brechemier self-assigned this Mar 14, 2019
@eric-brechemier eric-brechemier changed the title Call is rejected immediately when VOIP software phone is not connected Configure Twilio to record voicemail and send it by email May 10, 2019
@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 13, 2019

How I Failed to Configure a Voicemail Using TwiML Alone

Starting from a script as simple as possible to forward calls to my SIP endpoint, I first added a message, read by a friendly female robot using <Say>, followed with audio recording for the voicemail, using <Record>:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Dial>
    <Sip>sip:me@1-202-555-0162.sip.us1.twilio.com;transport=tls</Sip>
  </Dial>
  <Say voice="woman" language="en">
    Hi, Jane Doe is not available at the moment.
    You may leave a message after the beep.
  </Say>
  <Record maxLength="60" />
</Response>

A few notes:

This script handles correctly two out of three cases:

  1. ✓ voicemail triggers when software phone is closed or computer is unreachable
  2. ✓ voicemail triggers when software phone is open but I don't pick up the call
  3. 𐄂 voicemail also triggers, sadly, at the end of a successful call, if the caller stays on the line after I have hung up.

We would like to trigger the voicemail only for failed calls, not for successful calls.

Unfortunately, there is no way in TwiML to make a distinction between a failed call and a successful one:

When no action attribute is set on the <Dial> element, the end of the document executes whether the call was successful or not.

When a URL is provided in the action attribute, it is triggered with a parameter, DialCallStatus which indicates whether the call failed (busy, no-answer, failed, canceled) or was successful (completed, answered). But TwiML has no construct to test for a particular status value; using Mustache templates, we can only apply a different logic based on falsy values (boolean false, empty list or missing value) or truthy values (everything else).

It is also possible to set a callback URL in the StatusCallbackEvent attribute of the <Sip> element. But here again, when a call is completed, the same handler is called whether the call was successful or not, and we need to examine the value of the CallStatus parameter to determine whether to trigger the voicemail or not.

Lacking a separate action which would trigger only for failed calls, we cannot rely on TwiML to start the voicemail only when appropriate.

@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 13, 2019

Twimlets to the Rescue!

After TwiML Bins, the next simplest option is Twimlets, a set of ready-made scripts developed and hosted by Twilio.

We first need a Twimlet which checks whether the call failed or succeeded before it triggers the voicemail. To that end, we will use a little trick with the Forward Twimlet. That Twimlet is designed to call a regular phone number:

Forward will forward a call to another phone number. If the call isn't answered or the line is busy, the call is optionally forwarded to a new URL (or Twimlet).
https://www.twilio.com/labs/twimlets/forward

But we are forwarding the call to a SIP Phone Number, and thus we are only interested in the fallback part of the Twimlet. To skip the first part, we will set the Dial property of the Twimlet, which is a flag that the Twimlet adds to its own URL to indicate that the attempt to call has already occurred. When the flag is present, the Forward Twimlet checks the status in the DialCallStatus property, and in case the call failed, redirects to the URL provided in the FailUrl parameter:

https://twimlets.com/forward?Dial=true&FailUrl=...

In addition, we will use the Voicemail Twimlet which seems to match our needs perfectly:

  1. it plays a message (either using text-to-speech or by playing an audio file)
  2. it records voicemail after the beep
  3. it sends a link to the recording by email

Optionally, it can also transcribe the audio recording to text, but after a few tests, I did not find this option very useful because the transcription always assumed an English speaker even when I called from a French number, and the transcription was far from accurate, even in English.

For example:

Nature said that actually sheds some pebble ascension us and fish..

Just to learn you know I want one. To mull room so. He could come back and then we see the tax cheats..

It's not me it's age cheers..

I'll let you ponder what the original sentences where...

At the end of the page for the Voicemail Twimlet, you find a form, the Twimlet Generator, which lets you build a URL with parameters to customize the behavior of the Twimlet:

Twimlet Generator

Fill in your email and set the Transcribe option to false (it is enabled by default):

Voicemail Twimlet Settings

You get the resulting URL, in red, at the bottom of the generator:

http://twimlets.com/voicemail?Email=jane.doe%40bezen.org&Transcribe=false&

Let's change the protocol from http to https for safety. I also prefer to remove the redundant & at the end of the URL. With these two changes, the above URL becomes:

https://twimlets.com/voicemail?Email=jane.doe%40bezen.org&Transcribe=false

If you leave the Message field empty, the default message will be played:

Please leave a message after the beep.

This is a bit dry. You can customize this message, adding your name or your phone number. Your message can span several lines, for example:

Hi, Jane Doe is not available at the moment.
You may leave her a message after the beep.

Type your text in the Message field and you get a more complicated URL:

Custom Voicemail Message

Change it to https and remove the final & and you get the callback URL that we will use in the FailUrl parameter of the Forward Twimlet:

https://twimlets.com/voicemail?Email=jane.doe%40bezen.org&Message=Hi%2C%20Jane%20Doe%20is%20not%20available%20at%20the%20moment.%0AYou%20may%20leave%20her%20a%20message%20after%20the%20beep.&Transcribe=false

Sadly, it is not possible to select the voice and language for the text-to-speech conversion. As a result, this method is only suitable for messages in English.

If you want to play a message in a different language or in your own voice, you will need to play an audio recording instead. In the next three sections below, I will describe:

  1. how to record your greeting and export it as an audio file in the right format
  2. how to host the audio file as an Asset on Twilio
  3. how to configure the Voicemail Twimlet to play your audio file as greeting.

You can skip these three sections if you are using the text-to-speech conversion, and jump directly to the section after, Redirecting Missed Calls to Voicemail, which describes how to chain our TwiML script which handles incoming calls with the Forward Twimlet and the Voicemail Twimlet.

@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 13, 2019

Recording Your Voicemail Greeting

Here I am using Audacity to record the greeting message in WAV format. Alternatively, you can use any audio recorder capable of saving files in WAV or AIFF format. Twilio also supports playing files in MP3 format, but these are already compressed, and their conversion to the low bitrate format actually played on telephone networks is slower and may yield poor results. Thus Twilio recommends to record at 8kbps in a lossless format.

I do not think that they actually mean 8kbps uncompressed, but rather 8kHz with a depth of 8 bits which gives 64kbps uncompressed:

In telephony, a standard audio signal for a single phone call is encoded as 8,000 analog samples per second, of 8 bits each, giving a 64 kbit/s digital signal known as DS0.
https://en.wikipedia.org/wiki/Pulse-code_modulation#Processing_and_coding

When you first open Audacity, there are no audio tracks yet:

Starting Audacity

In the top part of the window, you can select the Audio Host and Recording Device, e.g. the Built-In Microphone. If your computer does not have a built-in microphone, make sure that you plug one in before recording.

Select Recording Device

Next on the right on the same line, reduce the number of channels from 2 (Stereo) to 1 (Mono):

Change Stereo to Mono

Then at the bottom-left of the window, change the Project Rate from 44100 Hz to 8000 Hz:

Change Project Rate to 8Hz

Write down or make a mental note of what you are going to say in your greeting, and say it out loud a few times, trying to articulate as much of possible. You can safely exaggerate a lot before you actually over-articulate.

If you want to sound more natural, you can put the photograph of a person in front of you. Imagine a first sentence spoken by that person, then look at the person in the photograph while recording your greeting, as if you were answering them. Make sure that you do not leave long pauses before you announce that you are not available, however, or a caller might start speaking, thinking that you are actually on the line.

When you are ready to record, you should first check the recording level. The input level at the top of the window bears the instruction "Click to Start Monitoring".

Click to Start Monitoring Recording Level

Click it and start talking; the recording level is shown as a green bar which grows from left to right. The maximum level reached is shown as a blue line:

Monitor Recording Level

Look at the meter as you repeat your greeting. If you speak louder and louder, the green bar changes to yellow (high but ok), orange (dangerously high) and red (too high, sound will be distorted). You want the green bar to grow as much as possible without getting into the red, to capture the whole dynamics of your voice without clipping.

Instead of speaking louder, speak normally and adjust the recording volume on the left of the window. It starts at 0.5, which is typically too low:

Initial Recording Volume

Audacity manual recommends to aim for a maximum level of approximately -6 dB. To reach this level in my case, I moved the recording volume close to 1.0:

Set recording volume

You can now press the Record button to start the recording and Stop to stop it.

It can be cumbersome to look for the buttons to start/end the recording while you are focusing on your performance. You can use keyboard shortcuts instead: R to record and space to stop the recording. You can then press space again to play the recording, and space again to stop playing.

On my computer, there was a little latency before the actual recording started. If it is your case as well, do not speak immediately after hitting Record, but wait one or two seconds before you start to speak.

If you are not satisfied with a recording, press the close button at the top-left of the Audio Track to remove it, then start a new recording.

Once you are satisfied with a recording, start by saving the project to keep a copy of the original:

Save Project

then enter a name for your project:

File Name

and click the Save button to save the project.

The project name is now displayed in the title bar at the top of the window.

Title Bar

Press the space bar or click the Play button to play the recording.

You can trim silent parts at the start and end of the recording, which often contain a bit of noise related to the manipulations to start and end the recording. You can also reduce long breaks in the middle of the recording. A period of silence appears as a flat line in the waveform visualization.

Select Range

Select a range with the mouse, then press delete key to remove it. The recording mends itself, with the parts before and after the cut now joined together:

Range Deleted

You can move the cursor back to the beginning of the track with the mouse or by pressing the Home key on the keyboard.

The infamous beep which signals the beginning of the recording will be played right after the end of your greeting, without any pause. I would rather leave a slight delay, to let the caller decide whether to leave a message or hang up before the beep. After trimming the end of the recording to remove the noise, you may append one to two seconds of silence at the end of the recording. Move the cursor to the end of the recording with a click or by pressing the End key on the keyboard. Then open the menu Generate > Silence...:

Generate Silence

Select the duration of silence; I chose 1.5 seconds:

Choose Silence Duration

The click the 14-ok button and silence gets added to the end of the track:

Silence Added to End of Track

When you are done editing, save the project with a new name, e.g. "Voice-Mail-EDIT", to preserve the unedited original.

Save Edited Project

You can now export the recording to WAV:

Export as WAV

The default is a WAV format with 16 bits of depth. But we want 8 bits of depth. Select Other uncompressed files as File format. You can then select further file format options:

Export Format Options

  • Header: WAV (Microsoft)
  • Encoding: U-Law

Strangely enough, the file extension gets changed to aiff, but the file will actually be saved with the wav extension.

Updated File Name

You can customize the file name and location, then click the Save button. A further window allows to set metadata to describe the recording:

Metadata

You can leave all the fields blank or, optionally, set your name as Artist Name, give a Track Title, the current Year, the Genre, and so on:

Track Metadata

Then click the OK button to save the exported file.

@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 14, 2019

Hosting the Audio File as an Asset on Twilio

The Voicemail Twimlet can play audio files accessible from any URL. If you have your own Web server, you may host it there. I find it more convenient to keep everything in the same place though, and to let Twilio host the audio file for the greeting as well.

  1. In the Twilio dashboard, go to Assets section.

  2. Open the Assets (Classic) subsection.

  3. Click the button Add an Asset.

  4. A file selection dialog opens. Navigate to the location of your audio file and select it for upload. Make sure to select the file with the wav extension, and not the file of the same name with the aup extension, which is the Audacity project.

  5. A dialog opens to confirm the asset upload:

Confirm Asset Upload

Do not check the box to make the asset private, which would prevent the Voicemail Twimlet from accessing it:

Private Assets are accessible to your Twilio Functions via code. We recommend using private assets for storing sensitive configuration you don't want hosted publicly. For example an XML or JSON file containing URLs for backend services

while

Public Assets are served over HTTPS from the Twilio CDN to ensure it is highly available and secure. Typically, Public Assets are used to store static files that support an application. For example, Public Assets are perfect for hosting .mp3 audio files used in TwiML or serve images sent through MMSes.
https://www.twilio.com/docs/runtime/assets

8-list-of-assets

  1. Click the Upload button to confirm the upload.

  2. The file is added to the list of assets, with a progress bar showing its deployment to Twilio's Content Delivery Network (CDN):

Upload Progress

  1. Once the deployment is complete, the progress bar is replaced with the path to access the file through Twilio Assets Gateway:

List of Assets

  1. Click the Copy icon on the right of the path to copy the URL of the asset to the clipboard.

@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 14, 2019

Using the Voicemail Twimlet with an Audio Greeting

We can now configure the Voicemail Twimlet to play our recorded greeting instead of a synthesized message.

  1. Open the page of the Voicemail Twimlet

  2. At the bottom of the page, fill in the Twimlet Generator form. Add your email and set the Transcribe option to false, as before, but this time, paste the URL of the recorded greeting into the Message box instead of typing a text message:

Voicemail Twimlet with Audio Greeting

  1. Copy the Resulting URL, e.g.
http://twimlets.com/voicemail?Email=jane.doe%40bezen.org&Message=https%3A%2F%2Fyour-runtime-domain.twil.io%2Fassets%2FVoicemail-Greeting-EDIT.wav&Transcribe=false&
  1. Replace http with https at the start of the URL and remove the final &:
https://twimlets.com/voicemail?Email=jane.doe%40bezen.org&Message=https%3A%2F%2Fyour-runtime-domain.twil.io%2Fassets%2FVoicemail-Greeting-EDIT.wav&Transcribe=false

We will now set this URL to the FailUrl parameter of the Forward Twimlet.

@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 14, 2019

Redirecting Missed Calls to Voicemail

At this point, you have a URL for the Voicemail Twimlet with your custom configuration, either using a text message in English or a link to an audio greeting that you recorded yourself.

  1. Copy this URL and go to the page of the Forward Twimlet

  2. At the bottom of the page, you find the Twimlet Generator form:

Forward Twimlet Generator

  1. Paste your customized Voicemail Twimlet URL in the FailUrl field of the generator:

Forward Twimlet Generator Filled

  1. Copy the Resulting URL generated for the Forward Twimlet:
http://twimlets.com/forward?FailUrl=https%3A%2F%2Ftwimlets.com%2Fvoicemail%3FEmail%3Djane.doe%2540bezen.org%26Message%3Dhttps%253A%252F%252Fyour-runtime-domain.twil.io%252Fassets%252FVoicemail-Greeting-EDIT.wav%26Transcribe%3Dfalse&
  1. Change the start of the URL from http to https and remove the final &:
https://twimlets.com/forward?FailUrl=https%3A%2F%2Ftwimlets.com%2Fvoicemail%3FEmail%3Djane.doe%2540bezen.org%26Message%3Dhttps%253A%252F%252Fyour-runtime-domain.twil.io%252Fassets%252FVoicemail-Greeting-EDIT.wav%26Transcribe%3Dfalse
  1. Add the flag Dial=true to the end of the URL. Because this URL will be used in a TwiML script this time, and TwiML is an XML dialect, we will encode the & separator before this flag as &amp;. Thus we add &amp;Dial=true to the end of the URL, which becomes in my case:
https://twimlets.com/forward?FailUrl=https%3A%2F%2Ftwimlets.com%2Fvoicemail%3FEmail%3Djane.doe%2540bezen.org%26Message%3Dhttps%253A%252F%252Fyour-runtime-domain.twil.io%252Fassets%252FVoicemail-Greeting-EDIT.wav%26Transcribe%3Dfalse&amp;Dial=true
  1. Go back to the TwiML Bins section in the Twilio dashboard.

  2. In the list of TwiML Bins, click the link to access the script to handle incoming calls:

Twiml Bin for Incoming Calls

  1. Add an action attribute to the <Dial> element, and set its value to the customized URL of the Forward Twimlet ending with the Dial flag &amp;Dial=true:

Twiml with Forward Action

  1. Click the Save button to save it.

Note: If you forgot to replace the & in the URL with &amp;, your TwiML will be displayed as invalid, without any further explanation, and you will be strongly discouraged, if not prevented, from saving it:

Do Not Confirm Saving Invalid TwiML

Do not save the invalid TwiML, but cancel and check that the & in the URL has been escaped as &amp;. The TwiML should then be valid and ready to save properly.

  1. You can now test the proper activation of the voicemail. Close the VOIP software phone on your computer, then call your Twilio number from a regular phone.

  2. After a few seconds, the call fails to connect and you are redirected to voicemail. You can leave a message to yourself.

  3. After the call, go check your emails. You should find an email from Twilio similar to this one:

From: voicemail@twimlets.com
Subject: New Voicemail Message from 336[...]
To: Me jane.doe@bezen.org

You have a new voicemail from 336[...]

Click this link to listen to the message: https://api.twilio.com/2010-04-01/Accounts/[...]/Recordings/[...].mp3

  1. Follow the link and you should hear the message that you left on your own voicemail.

@eric-brechemier
Copy link
Owner Author

eric-brechemier commented May 14, 2019

Managing Voicemails

You can find the list of recorded voicemails in your Twilio dashboard:

  1. Open the Logs section in the Monitor tab.

  2. Go to the Call recordings subsection.

  3. Here you find a list of all your voicemail recordings, which you can filter by date:

Voicemail Recordings

Sadly, there is no easy option to delete all these recordings at once. Fortunately, Twilio will store up to 10,000 minutes of voicemail for free (that's more than 6 days!), and only charges $0.0005/minute per month beyond that point.

If you only wish to delete a few voicemail recordings, you can do it one by one:

  1. Click the recording date to access the recording details:

Recording Date

  1. At the bottom of the recording details, a fixed panel displays a link to delete the recording:

Delete this Recording

  1. Click the link and a dialog opens to confirm the deletion:

Confirm Delete Recording

  1. Click the Yes button and the deletion is confirmed:

Recording Deleted Successfully

  1. The recording is now marked as Deleted in the list of recordings:

Deleted Recording

Otherwise, you can use the Twilio API to list and delete voicemail recordings. Using the API, you can list up to 10,000 recordings in a single request, then delete them one by one.

You can find an example script in PHP and Python in this blog post.

I would personally write a shell script instead, using curl and jq:

#!/bin/sh
# Delete all voicemail recordings from your Twilio account
#
# Note:
# a single run of this script will delete up to 10,000 voicemails.
#
# Requires:
#   * curl - transfer a URL
#   * jq - Command-line JSON processor
#
cd "$(dirname "$0")"

accountSID='' # REQUIRED
authToken='' # REQUIRED
test -f ./auth.sh && . ./auth.sh

if test -z "$accountSID" -o -z "$authToken"
then
  cat <<EOF >&2
accountSID and authToken must be set at the top of this script
or in a file called auth.sh in the same directory as this script.
They can be found in your Twilio dashboard:
https://www.twilio.com/console
EOF
  exit 1
fi

if test -z "$(which curl)" -o -z "$(which jq)"
then
  cat <<EOF >&2
curl and jq are required.
curl: https://curl.haxx.se/
jq: https://stedolan.github.io/jq/
EOF
  exit 2
fi

api="https://api.twilio.com/2010-04-01/Accounts/$accountSID"
auth="$accountSID:$authToken"

curl "$api/Recordings.json" -u "$auth" -s |
jq '.recordings[] | .sid' --raw-output |
while read -r recordingSID
do
  echo "Delete recording $recordingSID..."
  curl -X DELETE "$api/Recordings/$recordingSID.json" -u "$auth" -s
done

echo 'Done.'

eric-brechemier added a commit that referenced this issue May 15, 2019
I enhanced the recording of the greeting and introduced the Forward
Twimlet between the TwiML <Dial> and the Voicemail Twimlet.
@eric-brechemier
Copy link
Owner Author

You may leave your comments below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant