Build & Repeat: CI & CD with Twilio Studio Flows and Serverless Runtime Projects

This repo and its accompanying instructions outline the pathway to setting up an end-to-end testing, integration and deployment workflow. We will be using bash scripts, the twilio CLI, APIs and some Node.js code.

In our example, we will be starting a new Twilio project from scratch. The project will build a Customer Satisfaction Survey via SMS. You can see the architecture diagram below.

When using this repo as a reference for your future working, you can substitute the placeholder values for the account SID and auth token for your development, staging and production environments.

In this instance, we’ve opted to use three subaccounts to simulate the progress from environment to environment.

Note regarding external services: We will not be developing from the Strapi-side. Strapi’s headless CMS service will serve as an example for an external API-enabled database containing customer details i.e. name, job number etc. You can find more about Strapi in their website.

Download code repository

Download your own local copy of this repo to your machine using [.Zip ]

We will call this copy's root folder 'csat-form-master' from now on.

Unzip the file into your local user directory.

Navigate to the file in your directory and open in VSCode:

$ cd csat-form-master
$ code

Installation

Firstly, start by installing the latest versions of Twilio CLI and Serverless Plugins:

$ npm install twilio-cli -g

$ twilio plugins:install @twilio-labs/plugin-serverless

If you already have Twilio CLI installed, run the following:

$ brew upgrade twilio-cli

Or if you’re a node developer running a version later than 10.0

$ npm i twilio-cli

Setup

First up, login to you Twilio dev account via the command line:

$ twilio profiles:list
$ twilio profiles:use 

If you need to create a new profile from scratch in which to host your dev environment (recommended) then use the following command:

$ twilio profiles:create

You will need your Twilio Account SID and Auth Token for the above action.

Initialise the serverless plugin and create your new project to kick off the project structure creation.

$ twilio serverless:init myproject ← Feel free to name your project whatever you want

Navigate towards said project:

$ cd myproject

And initialise the server:

$ npm start

Now if you navigate to http://localhost:3000/, your Twilio Serverless project should be live!

Next, you’ll want to delete the pre-made assets (_myproject > assets > _delete _index.html, message.private.js _and style.css), Function Templates (_myproject > functions > hello-world.js _and private-message.js) and the .twilio-functions file.

With your project environment cleared up, it’s time to install your dependencies. In this instance, we’ll be using the following services:

$ npm install concurrently
$ npm install custom-env
$ npm install ngrok
$ npm install axios 

Now, with your dependencies installed on your development environment using the Twilio Serverless Runtime project structure, you’ll need to look at the pre-made repo and use the .env.dev.example to set your environment variables inside your project.

Here’s a quick guide to which credentials/SIDs/tokens you should use in each instance:

Key	Reference
NGROK_TOKEN=1avsgIzuHYUIKppdmP6rG6gN2hy_ECXsCMBQUkLb7hsW49Kk	This is the token for the Ngrok account that allows us to set subdomain - note that this is not essential. You can configure your own non-custom ngrok tunnel for this process to work
TESTER_URL=testerSignal2020	Ngrok subdomain that’s surfacing your testing server - change to suit your project name [for free account holders, see note above]
flowSid/ flowSidSource=FWxxxxxxxxxxx	Unique identifier of the Studio flow that you’re trying to replicate
accountSIDFlow=ACxxxxxxxxx	Unique identifier of the originating account that you’re cloning your Studio flow from
authTokenFlow=xxxxxxxxxxx	Auth token for the originating account that you’re cloning your Studio flow from
targetAccountSid=ACxxxxxx	Unique account SID identifier of the project you’ve created for your dev environment
targetAccountToken=xxxxxxxxxxx	Auth token for the project you’ve created for your dev environment
LOGGER_SERVICE_URL=https://csat.ngrok.io/logs?token=abcdefghijk	The externally reachable URL for your logger service

Save your newly updated “.env.dev” file and delete the example.

Firstly, copy the logger.js function from the pre-prepared repo, and paste it in the functions folder within your project.

$ cd functions
$ cp logger.js /Users/usr/csat-sms-form/myproject/functions

One of the last steps that you need to take before running your first bash script will be to set up a mock server. Both the logger service (logger.js) and the Studio HTTP widget will be connected to the mock server in its first instance. This mock service will simply respond with a 200/OK response whenever reached. You can find information about how to spin up your own mock service here.

The idea behind the way we’re incorporating services, is step by step. We want to up the complexity of our program in a very incremental manner.

With your mock service up and running, copy the folder bash_scripts to your new project, navigate to the root of the project, and then run this command:

$ bash bash_scripts/deployDevEnv.sh

The above command will have deployed the logger function to your Twilio dev environment.

Next, step into your Twilio console, to view your newly deployed service!

Jump into the Studio flow and assign the logger service to the endpoint within your Studio flow. You’ll need to make sure that your Studio Flow is completely free of any external URLs.

While you’re there, also update your environment, and function usage:

And as always, hit **save **and **publish. **

With your function deployed to your dev environment, credentials updated and dependencies installed, you’ll lastly need to configure the phone number that you’ll be testing from (your one key static asset in this process) to the desired Studio flow that you’d like to alter.

This can be done in the console, under the numbers icon or via the CLI.

Creating your Twilio testing account

Now, in order to simulate user behaviour within your development environment, you’ll firstly need to create an additional account that's completely independent from your project account with its own SID and auth token, purely for programmatic testing.

Once you have you testing Twilio account set, run:

$ twilio profiles:create testing

Next, head to the console to purchase a number.

As always, there’s a CLI shortcut for that! Run the below command replacing the placeholder with a number available to purchase.

$ twilio api:core:incoming-phone-numbers:create \
  --phone-number="+[available phone number in e.164 format]"

You’ll need these details in the following section on testing to create a functioning test environment from which to alter your flow.

Testing

Copy across the testing service folder /test_service to your project.

The test service itself contains a file that spins up a server that will write the message body of the SMS messages that your Twilio testing phone number receives from your flow, and print them to a file.

The programme will parse your ngrok auth token, and use this to assign a custom domain name to your ngrok tunnel in order to keep it consistent, and externally accessible.

It’s possible to spin up your testing service using a free standard ngrok account, but you’ll have to bear in mind that ngrok assigns a one-time session ID for your url. As a reminder, this externally accessible URL is referenced as the TESTER_URL throughout the dependency documentation. We recommend you use a fixed URL, it will simplify the process.

For the test service to operate, you must assign the ngrok public url as a webhook for the Twilio test account SMS number. Now, every time that test number gets an SMS, your test service will write the content of the SMS into a file called test.txt .

At your first phase of testing, you’ll need to install the corresponding testing Library, which is in this case, Cypress.

Once installed, you will have to clean the cypress structure from the example tests (delete file inside integration folder) and copy tests from the original:

$ npm install cypress --save-dev

Next, we need to import our pre-made tests, also stored in the root directory. Copy the contents of the **cypress > integration **folder from the pre-prepared repository.

$ cd ..
$ cd cypress/integration
$ cp *.* myproject/cypress/integration

These scripts will enable you to harness Cypress’ GUI to test the functionality of your SMS flow.

Before you run these tests, make sure to check the phone number variables in your code and ensure that the number referenced in your code matches the number configured to your flow within the numbers section of the console of your Twilio development account.

And lastly, head to the package.json file within the repo, and copy across the below line of code to the scripts package within the package.json file within your project - and of course, hit save!

"test-dev": "concurrently --kill-others \"$(npm bin)/cypress open\" \"node test_service/server.js\""

The above script will use concurrently to run two separate programs. One of them is the Cypress testing tool (which will be reading the test.txt file to see what has been received) and the other is the test_service which will be writing what the test number simulating a person is receiving.

Now, for Cypress to be able to test against your flow, you’ll need to copy the cypress.env example from the root to your project and update it based on the below guide:

Key	value
accountSID	Unique account SID identifier of the project you’ve created for your testing environment account
authToken	Unique auth token for your testing environment account
testAccountNumber	Phone number tied to testing account
flowSid	Unique Flow SID identifier
accountSIDFlow	Account SID for the development account hosting the flow
authTokenFlow	Auth token for the development account hosting the flow
flowNumber	Phone number configured to the flow
csatDB	External ngrok tunnel to the strapi database
loggerDb	External ngrok tunnel to logger function

Save the newly updated file within your project as cypress.env.json.

You will also have to copy the contents of the 'csat-form-master > Cypress > Integration > command.js into myproject > Cypress > Integration > Command.js

Cypress is a testing framework that originally targets Web interfaces. It is a very flexible framework and allows you to expand testing into SMS or any other communication interface by adding key communication modules into the Command.js file. In our project we have expanded Cypress to be able to support SMS communications and Studio flow testing.

To start up your testing server while having Cypress running at the same time, run the following command:

$ npm start test-dev

Your tests should show your testing number interacting with your studio flow within the Cypress testing framework, and work to automate the entire process of a user going through both the HAPPY and UNHAPPY customer experience paths.

This is the environment for you to develop your tests according to your user stories for the project at hand, and then work to develop your flow in a manner that meets your testing criteria.

Remember - think testing and automation first. This will make your final build more robust and give it the strong foundation it needs to be continually repeated, improved and redeployed.

Pushing to staging environment

Now we’ve reached the staging environment, it’s important to remember that we’re working from the minimal amount of dependencies, to the maximum. So the order of actions will be:

1. Get the Strapi or other database URLs for logging and storing the results of the SMS survey
2. Deploy the logger Function into Staging
3. Replicate the Studio flow from Development to Staging
4. Connect the Studio Flow Webhook to the SMS phone number that we have in Staging

In order to push to the staging environment, you’ll need to add the **studio **folder and its contents to your project. Studio should not be copied or imported manually from environment to environment. Instead, we will use the Studio API and node.js to automatically import the flow from the development environment, update the dependencies (changes from dev to staging: new logger service and database URLs) and add the webhook to the phone number.

You’ll also need to update the **.env.staging **file using the previously provided key/value tables for reference in filling out the credentials and add this to your project folder using the credentials of your **staging account **as the target account.

Firstly, ensure that your terminal is using your appropriate Twilio staging profile, and in doing so, surface the account details for this account.

In order to do this, run the following command:

$ twilio profiles:list

And if you see the account you’d like to use for the staging portion of the build, use the below command:

$ twilio profiles:use <staging-env>

If not, you can go ahead and create this now by running the below:

$ twilio profiles:create staging

You can head to your console to grab your auth token and account SID for your staging environment and add these into your .env.staging file.

You’ll need to export the SID and token for the staging environment using these commands. This will inform Cypress that we’re now testing our flow within the staging environment.

$ export ACCOUNTSID_STAGING= 
$ export TOKENSID_STAGING=

Now, ahead of running your most complex bash script, you’ll need to purchase an available number, which your bash script will use to attach to your flow.

$ twilio api:core:incoming-phone-numbers:create \
  --phone-number="+[available phone number in e.164 format]"

Add this number into your script as a variable, replacing the example:

mobile="+XXXXXXXXXX"

Once you’re all ready, deploy your bash script to push you to the staging environment:

$ bash bash/deployStagingEnv.sh 

The script will assign your staging environment variables manually, before opening and running Cypress.

Here’s the line of code within your bash script which assigns the corresponding staging environment variables for you:

$(npm bin)/cypress open  --env flowNumber=$mobile,flowSid=$flowId,accountSIDFlow=$ACCOUNTSID_STAGING,authTokenFlow=$TOKEN_STAGING

Perform your staging environment tests.

Pushing to the production environment

For the final stage of your build, you’ll be pushing production-ready code. This deployment will be very similar to staging, except in this instance, you won’t be performing any testing.

First up, if you don’t have your production profile set-up and ready to go, here’s a refresher on how to create that:

$ twilio profiles:list
$ twilio profiles:create production

Next, you’ll need to surface both the account SID and auth token and get these added to your new environment file. Remember also to update the script with the phone number of your production environment.

Use the .**env.production.example **as a guide:

Key	value
flowSidSource	Unique Flow SID identifier
accountSIDFlow	Account SID for the development account hosting the flow
authTokenFlow	Auth token for the development account hosting the flow
targetAccoundSid	Account SID for the production account hosting the flow
targetAccountToken	Auth token for the production account hosting the flow
Logger_service_URL=	External ngrok tunnel to logger function

With your environment ready to go, navigate to the root of your project and run the final deployment script to push to production 😎

$ bash bash_scripts/deployProductionEnv.sh