SMS Communication

Makefile

@@ -51,9 +51,13 @@ send-email:
 send-sms:
 	npm run send-test-sms
 
-.PHONY: send-dispatch ## Send an email or sms with the higher-level dispatch function (for testing).
-send-dispatch:
-	npm run send-test-dispatch
+.PHONY: send-dispatch-email ## Send an email with the higher-level dispatch function (for testing).
+send-dispatch-email:
+	TEST_DISPATCH_CHANNEL='email' npm run send-test-dispatch
+
+.PHONY: send-dispatch-sms ## Send an sms with the higher-level dispatch function (for testing).
+send-dispatch-sms:
+	TEST_DISPATCH_CHANNEL='sms' npm run send-test-dispatch
 
 ####
 

docs/inbound-sms.md

@@ -0,0 +1,56 @@
+# Inbound SMS
+
+Gov Flow supports multiple "inbound channels" for service requests. The basic inbound channel is submitting data via a POST request, as generally done from a web form (and, we call this channel `webform` in the data model ([ref.](https://github.com/govflow/govflow/blob/main/src/core/service-requests/models.ts#L20))).
+
+Gov Flow also supports SMS as a channel for service requests, and in this implementation, an SMS can be sent to a special email address, and in turn, data from that email gets sent to Gov Flow, processed, and turned into a service request. There are several aspects of this integration to create structured data out of an SMS, and here we will go over all the details.
+
+## Configure a Twilio Number for the GovFlow instance
+
+At the level of the GovFlow instance (the server), we need one Twilio Number configured, and set at `TWILIO_FROM_PHONE`.
+
+This number is used to emit SMS messages for Service Request submissions as a fallback, and irregardless of whether a given Jurisdiction has opted in to Inbound SMS ot Two-way SMS. Think if it like `webmaster@localhost` for SMS in GovFlow.
+
+This number needs to be configured to send messages (default when you acquire a Twilio Number), and also it is highly recommended to provide an autoresponder from this number so if anyone attempts to respond back to an SMS, it is clear that it is a "no reply" number.
+
+The core configuration for Twilio is to set `TWILIO_ACCOUNT_SID ` and `TWILIO_AUTH_TOKEN` with valid credentials from a Twilio account, in order to interact with Twilio.
+
+### Set up an auto response with Studio Flow
+
+[Studio Flow](https://console.twilio.com/us1/develop/studio?frameUrl=%2Fconsole%2Fstudio%2Fdashboard%3Fx-target-region%3Dus1) is Twilio's UI for creating logic around events in there system. We can use it to set an auto response on our GovFlow instance number.
+
+- Create a new flow (e.g.: called "GovFlow Instance Auto Response")
+- Choose the "Message Auto Responder" template
+- When the editor opens up, it creates a widget with a default name and message body. Edit it as desired. (e.g: "This number does not accept replies. Please contact the Jursidiction where you submitted your request directly.")
+- After you save your widget, click on the "trigger" component, and then, on the right had side edit panel, you will see a link "Using this flow with a Phoen Number". Click that link and follow the stesp to associate it with your GovFlow Instance Number
+- Click on the Twilio Number that is your instance number, go down to messaging, and change "A Message Comes in" to Studio Flow and select to Flow you just created.
+- Now, this number will be used for sending system messages, but if anyone tries to respond to such messages, they will receive a response back "This number does not accept replies. Please contact the Jursidiction where you submitted your request directly."
+
+## Configure a Twilio Number for a Jurisidiction
+
+Ok, so now with the basics of what the GovFlow server needs done, we can move onto Jurisdiction-level Twilio Number configuration for inbound SMS.
+
+The core configuration for Twilio is to set `TWILIO_ACCOUNT_SID ` and `TWILIO_AUTH_TOKEN` with valid credentials from a Twilio account.
+
+The Twilio setup steps are:
+
+- Create a new number, and give it a friendly name (e.g.: "GovFlow Demo")
+- As with the instance-level number, create an auto response (as this is one-way communication, we want it to be clear to submitters).
+  - It is recommended to create a generic "jursidiction no reply" response. You can do that by duplicating the "GovFlow Instance Auto Response" we created above, call it "GovFlow Jurisdiction No Reply" and write your message.
+  - We will attach all Jurisdiction Numbers to this by default, and only change that if (i) a Jurisdiction wants a custom message, or (ii), the **common case**, if two-way communication via SMS is enabled.
+
+Next, we need to configure the use of this Number for the Jurisdiction in two places:
+
+1. `Jurisdiction.sendFromPhone` - This needs to be set with the new number. When this is set, outgoing messages for the jurisdiction will be sent from this number rather than `TWILIO_FROM_PHONE`. This also includes the sending of messages when `Jursidcition.replyToServiceRequestEnabled` is enabled for two-way communication.
+2. Creating a new `InboundMap` record to route incoming SMS from this number to the correct Jurisdiction (this is the equivalent of how InboundMap is used for [Inbound Email](./inbound-email.md)).
+  - Note that we **don't** simply use `Jurisdiction.sendFromPhone` because it is possible to create any number of InboundMap records with different Twilio Numbers and/or with different rules. However, the happy path for small Jurisdictions will be to simply create a single InboundMap record associating the Twilio Number and the Jurisdiction, and have all inbound requests come via it.
+
+```http
+POST {{ host }}/communications/create-map?jurisdictionId={{ jurisdictionId }}
+content-type: application/json
+
+{
+    "jurisdictionId": "{{ jurisdictionId }}",
+    "channel": "sms",
+    "id": "+1-111-11111"
+}
+```

src/cli/send-test-dispatch.ts

@@ -18,7 +18,7 @@ import { DispatchConfigAttributes } from '../types';
     } = app.config;
     const { communicationRepository, emailStatusRepository } = app.repositories;
     const dispatchConfig = {
-        channel: 'email', // can manually change to sms to test that
+        channel: process.env.TEST_DISPATCH_CHANNEL,
         sendGridApiKey: sendGridApiKey as string,
         toEmail: testToEmail as string,
         fromEmail: sendGridFromEmail as string,

src/core/communications/helpers.ts

@@ -9,7 +9,7 @@ import striptags from 'striptags';
 import { sendEmail } from '../../email';
 import logger from '../../logging';
 import { sendSms } from '../../sms';
-import { CommunicationAttributes, DispatchConfigAttributes, DispatchPayloadAttributes, EmailEventAttributes, ICommunicationRepository, IEmailStatusRepository, InboundEmailDataToRequestAttributes, InboundMapInstance, JurisdictionAttributes, ParsedForwardedData, ParsedServiceRequestAttributes, PublicId, ServiceRequestAttributes, TemplateConfigAttributes, TemplateConfigContextAttributes } from '../../types';
+import { ChannelType, CommunicationAttributes, DispatchConfigAttributes, DispatchPayloadAttributes, EmailEventAttributes, ICommunicationRepository, IEmailStatusRepository, InboundEmailDataToRequestAttributes, InboundMapInstance, InboundSmsDataToRequestAttributes, JurisdictionAttributes, ParsedForwardedData, ParsedServiceRequestAttributes, PublicId, ServiceRequestAttributes, TemplateConfigAttributes, TemplateConfigContextAttributes } from '../../types';
 import { SERVICE_REQUEST_CLOSED_STATES } from '../service-requests';
 import { InboundMapRepository } from './repositories';
 
@@ -32,6 +32,10 @@ export function makeRequestURL(appClientUrl: string, appClientRequestsPath: stri
 export async function loadTemplate(templateName: string, templateContext: TemplateConfigContextAttributes, isBody = true): Promise<string> {
     const filepath = path.resolve(`${__dirname}/templates/${templateName}.txt`);
     const [templateType, ..._rest] = templateName.split('.');
+    let lineBreak = "\n";
+    if (templateType === "email") {
+        lineBreak = '<br />';
+    }
     try {
         await fs.access(filepath, fsConstants.R_OK | fsConstants.W_OK);
     } catch (error) {
@@ -46,12 +50,12 @@ export async function loadTemplate(templateName: string, templateContext: Templa
     if (isBody) {
         const poweredBy = path.resolve(`${__dirname}/templates/${templateType}.powered-by.txt`);
         const poweredByBuffer = await fs.readFile(poweredBy);
-        appendString = `<br />${poweredByBuffer.toString()}<br />`;
+        appendString = `${lineBreak}${poweredByBuffer.toString()}${lineBreak}`;
         const unsubscribe = path.resolve(`${__dirname}/templates/${templateType}.unsubscribe.txt`);
         const unsubscribeBuffer = await fs.readFile(unsubscribe);
         replyAboveLine = templateContext.jurisdictionReplyToServiceRequestEnabled
-            ? `${emailBodySanitizeLine}<br /><br />` : '';
-        appendString = `${appendString}<br />${unsubscribeBuffer.toString()}<br />`;
+            ? `${emailBodySanitizeLine}${lineBreak}${lineBreak}` : '';
+        appendString = `${appendString}${lineBreak}${unsubscribeBuffer.toString()}${lineBreak}`;
     }
 
     const fullTemplateString = `${replyAboveLine}${templateString}${appendString}`;
@@ -217,9 +221,8 @@ export function extractToEmail(inboundEmailDomain: string, headers: string, toEm
     return address as addrs.ParsedMailbox
 }
 
-export async function findIdentifiers(toEmail: addrs.ParsedMailbox, InboundMap: InboundMapRepository): Promise<InboundMapInstance> {
-    const { local } = toEmail;
-    const record = await InboundMap.findOne(local) as InboundMapInstance;
+export async function findIdentifiers(id: string, channel: ChannelType, InboundMap: InboundMapRepository): Promise<InboundMapInstance> {
+    const record = await InboundMap.findOne(id, channel) as InboundMapInstance;
     return record;
 }
 
@@ -241,30 +244,64 @@ export function extractDescriptionFromInboundEmail(emailSubject: string, emailBo
     return `${prefix}${cleanText}`;
 }
 
-export function extractPublicIdFromInboundEmail(emailSubject: string): string | undefined {
+export function _extractPublicIdFromText(text: string): string | undefined {
     let publicId;
-    const match = emailSubject.match(publicIdSubjectLinePattern);
+    const match = text.match(publicIdSubjectLinePattern);
     if (match && match.length == 2) { publicId = match[1]; }
     return publicId;
 }
 
+export function extractPublicIdFromInboundEmail(emailSubject: string): string | undefined {
+    return _extractPublicIdFromText(emailSubject);
+}
+
+export function extractPublicIdFromInboundSms(smsBody: string): string | undefined {
+    return _extractPublicIdFromText(smsBody);
+}
+
 export function extractCreatedAtFromInboundEmail(headers: string): Date {
     const dateStr = extractDate(headers);
     return new Date(dateStr);
 }
 
+export async function extractServiceRequestfromInboundSms(data: InboundSmsDataToRequestAttributes, InboundMap: InboundMapRepository):
+    Promise<[ParsedServiceRequestAttributes, PublicId]> {
+    const { To, From, Body } = data;
+    const inputChannel = 'sms';
+
+    const publicId = extractPublicIdFromInboundSms(Body);
+    const { jurisdictionId, departmentId, staffUserId, serviceRequestId, serviceId } = await findIdentifiers(
+        To, inputChannel, InboundMap
+    );
+    return [{
+        jurisdictionId,
+        departmentId,
+        assignedTo: staffUserId,
+        serviceRequestId,
+        serviceId,
+        firstName: '',
+        lastName: '',
+        email: '',
+        phone: From,
+        description: Body,
+        inputChannel,
+        createdAt: new Date(),
+    }, publicId];
+}
+
 export async function extractServiceRequestfromInboundEmail(data: InboundEmailDataToRequestAttributes, inboundEmailDomain: string, InboundMap: InboundMapRepository):
     Promise<[ParsedServiceRequestAttributes, PublicId]> {
     let firstName = '',
         lastName = '',
         email = '';
+    const phone = '';
     const inputChannel = 'email';
     const { to, cc, bcc, from, headers } = data;
     let { subject, text } = data;
     let fromEmail = extractFromEmail(from);
     const toEmail = extractToEmail(inboundEmailDomain, headers, to, cc, bcc);
     const { jurisdictionId, departmentId, staffUserId, serviceRequestId, serviceId } = await findIdentifiers(
-        toEmail, InboundMap
+        toEmail.local, inputChannel, InboundMap
     );
 
     const maybeForwarded = extractForwardDataFromEmail(text, subject);
@@ -294,6 +331,7 @@ export async function extractServiceRequestfromInboundEmail(data: InboundEmailDa
             firstName,
             lastName,
             email,
+            phone,
             description,
             inputChannel,
             createdAt

src/core/communications/models.ts

@@ -106,6 +106,11 @@ export const InboundMapModel: ModelDefinition = {
             allowNull: false,
             primaryKey: true,
         },
+        channel: {
+            type: DataTypes.ENUM('email', 'sms'),
+            allowNull: false,
+            defaultValue: 'email',
+        },
         staffUserId: { // is not a foreignKey as commonly customised
             type: DataTypes.STRING,
             allowNull: true,
@@ -137,6 +142,10 @@ export const InboundMapModel: ModelDefinition = {
             {
                 unique: true,
                 fields: ['id', 'jurisdictionId', 'serviceRequestId']
+            },
+            {
+                unique: true,
+                fields: ['id', 'channel', 'jurisdictionId']
             }
         ]
     }

src/core/communications/repositories.ts

@@ -3,6 +3,7 @@ import { appIds } from '../../registry/service-identifiers';
 import type {
     AppConfig, ChannelIsAllowed, ChannelStatusCreateAttributes,
     ChannelStatusInstance,
+    ChannelType,
     CommunicationAttributes,
     CommunicationCreateAttributes,
     CommunicationInstance,
@@ -129,10 +130,10 @@ export class InboundMapRepository implements IInboundMapRepository {
         return await InboundMap.create(data) as InboundMapInstance;
     }
 
-    async findOne(id: string): Promise<InboundMapInstance | null> {
+    async findOne(id: string, channel: ChannelType): Promise<InboundMapInstance | null> {
         const { InboundMap } = this.models;
         const record = await InboundMap.findOne(
-            { where: { id }, order: [['createdAt', 'DESC']] }
+            { where: { id, channel }, order: [['createdAt', 'DESC']] }
         ) as InboundMapInstance | null;
         return record;
     }

src/core/communications/routes.ts

@@ -9,6 +9,23 @@ import { EMAIL_EVENT_IGNORE } from './models';
 
 export const communicationsRouter = Router();
 
+// public route for web hook integration
+communicationsRouter.post('/inbound/sms', multer().none(), wrapHandler(async (req: Request, res: Response) => {
+    const { jurisdictionRepository } = res.app.repositories;
+    const { outboundMessageService, inboundMessageService } = res.app.services;
+    const [record, recordCreated] = await inboundMessageService.createServiceRequest(req.body);
+    const jurisdiction = await jurisdictionRepository.findOne(record.jurisdictionId) as JurisdictionAttributes;
+
+    let eventName = 'serviceRequestCreate';
+    if (recordCreated) {
+        GovFlowEmitter.emit(eventName, jurisdiction, record, outboundMessageService);
+    } else {
+        eventName = 'serviceRequestCommentBroadcast';
+        GovFlowEmitter.emit(eventName, jurisdiction, record, outboundMessageService);
+    }
+    res.status(200).send({ data: { status: 200, message: "Received inbound SMS" } });
+}))
+
 // public route for web hook integration
 communicationsRouter.post('/inbound/email', multer().none(), wrapHandler(async (req: Request, res: Response) => {
     const { jurisdictionRepository } = res.app.repositories;

src/core/communications/services.ts

@@ -9,13 +9,16 @@ import type {
     IInboundMessageService,
     InboundEmailDataAttributes,
     InboundMapAttributes,
+    InboundSmsDataAttributes,
     IOutboundMessageService,
     JurisdictionAttributes,
+    ParsedServiceRequestAttributes,
+    PublicId,
     RecipientAttributes,
     Repositories,
     ServiceRequestAttributes, ServiceRequestCommentAttributes, ServiceRequestInstance, StaffUserAttributes
 } from '../../types';
-import { canSubmitterComment, dispatchMessage, extractServiceRequestfromInboundEmail, getReplyToEmail, getSendFromEmail, makeRequestURL } from './helpers';
+import { canSubmitterComment, dispatchMessage, extractServiceRequestfromInboundEmail, extractServiceRequestfromInboundSms, getReplyToEmail, getSendFromEmail, makeRequestURL } from './helpers';
 
 @injectable()
 export class OutboundMessageService implements IOutboundMessageService {
@@ -460,28 +463,55 @@ export class InboundMessageService implements IInboundMessageService {
         this.config = config;
     }
 
-    async createServiceRequest(inboundEmailData: InboundEmailDataAttributes):
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    inboundIsEmail(data: any): data is InboundEmailDataAttributes {
+        const valids = ['to', 'from', 'subject', 'text', 'envelope', 'dkim', 'SPF'];
+        const properties = Object.keys(data)
+        return valids.every(value => properties.includes(value))
+    }
+
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    inboundIsSms(data: any): data is InboundSmsDataAttributes {
+        const valids = ['To', 'From', 'Body', 'MessageSid', 'AccountSid', 'SmsMessageSid', 'MessagingServiceSid'];
+        const properties = Object.keys(data)
+        return valids.every(value => properties.includes(value))
+    }
+
+    async createServiceRequest(inboundData: InboundEmailDataAttributes | InboundSmsDataAttributes):
         Promise<[ServiceRequestAttributes, boolean]> {
         const { serviceRequestRepository, staffUserRepository, inboundMapRepository } = this.repositories;
-        const { subject, to, cc, bcc, from, text, headers } = inboundEmailData;
+
         const { inboundEmailDomain } = this.config;
         let intermediateRecord: ServiceRequestAttributes;
         let recordCreated = true;
-        const [cleanedData, publicId] = await extractServiceRequestfromInboundEmail(
-            { subject, to, cc, bcc, from, text, headers }, inboundEmailDomain, inboundMapRepository
-        );
+        let cleanedData: ParsedServiceRequestAttributes;
+        let publicId: PublicId;
+
+        if (this.inboundIsEmail(inboundData)) {
+            const { subject, to, cc, bcc, from, text, headers } = inboundData;
+            [cleanedData, publicId] = await extractServiceRequestfromInboundEmail(
+                { subject, to, cc, bcc, from, text, headers }, inboundEmailDomain, inboundMapRepository
+            );
+        } else if (this.inboundIsSms(inboundData)) {
+            const { To, From, Body } = inboundData;
+            [cleanedData, publicId] = await extractServiceRequestfromInboundSms(
+                { To, From, Body }, inboundMapRepository
+            );
+        } else {
+            throw Error("Received an invalid inbound data payload");
+        }
+
         if (publicId || cleanedData.serviceRequestId) {
             const [staffUsers, _count] = await staffUserRepository.findAll(
                 cleanedData.jurisdictionId, { whereParams: { isAdmin: true } }
             );
-            // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-            // @ts-ignore
-            const staffEmails = staffUsers.map((user: StaffUserInstance) => { return user.email }) as string[];
+            const staffEmails = staffUsers.map((user: StaffUserAttributes) => { return user.email }) as string[];
             let addedBy = '__SUBMITTER__';
             if (staffEmails.includes(cleanedData.email)) {
-                // eslint-disable-next-line @typescript-eslint/ban-ts-comment
-                // @ts-ignore
-                addedBy = _.find(staffUsers, (u) => { return u.email === cleanedData.email }).id
+                const staffUserMatch = _.find(
+                    staffUsers, (u) => { return u.email === cleanedData.email }
+                ) as StaffUserAttributes;
+                addedBy = staffUserMatch.id;
             }
             if (cleanedData.serviceRequestId) {
                 intermediateRecord = await serviceRequestRepository.findOne(

src/core/jurisdictions/models.ts

@@ -28,6 +28,10 @@ export const JurisdictionModel: ModelDefinition = {
             type: DataTypes.BOOLEAN,
             defaultValue: false,
         },
+        sendFromPhone: {
+            allowNull: true,
+            type: DataTypes.STRING,
+        },
         sendFromEmail: {
             allowNull: true,
             type: DataTypes.STRING,