two way sms - message disambiguation, and delivery event handling

docs/two-way-email.md → docs/two-way-email-and-sms.md

@@ -1,21 +1,43 @@
-# Two-way email
+# Two-way SMS and email
 
-Extending the functionality described in [Inbound Email](inbound-email.md), [Email Delivery Management](email-delivery-management.md), and [core email dispatch functionality](https://github.com/govflow/govflow/blob/main/src/email/index.ts#L9), Gov Flow supports full two way communication between Staff Users and Submitters around a Service Request.
+Extending the functionality described in [Inbound Email](inbound-email.md), [Inbound SMS](inbound-sms.md), [Email Delivery Management](email-delivery-management.md), [core email dispatch functionality](https://github.com/govflow/govflow/blob/main/src/email/index.ts#L9) and [core SMS dispatch functionality](https://github.com/govflow/govflow/blob/main/src/sms/index.ts), Gov Flow supports full two way communication between Staff Users and Submitters around a Service Request.
 
-This means that, as well as [commenting on a Service Request](https://github.com/govflow/govflow/blob/main/src/core/service-requests/routes.ts#L72) via the API (used by client apps that implement a dashboard, for example), an email can be sent, to a specifically configured [inbound email address](), and that email will be turned into a comment on the service request, with provenance information such as who made the comment and when.
+This means that, as well as [commenting on a Service Request](https://github.com/govflow/govflow/blob/main/src/core/service-requests/routes.ts#L72) via the API (used by client apps that implement a dashboard, for example), an email or an SMS can be sent, to a specifically configured email or phone, and that email or SMS will be turned into a comment on the service request, with provenance information such as who made the comment and when.
 
 ## How it works
 
-There is a jurisdiction setting: `jursidiction.replyToServiceRequestEnabled`. If this is `false` (the default). Emails emitted from the system will not have the Service Request reply to inbound email, and they will just fall back to the (i) jursidiction reply to, or (ii) the system reply to. The rest works when this is `true`.
+There is a jurisdiction setting: `jursidiction.replyToServiceRequestEnabled`.
 
-- Comments POSTed with `broadcastToSubmitter` as `true` are send by email to the original submitter of the request. Additionally, the submitter can reply to the email and it will write her comment into Gov Flow as a new Service Request Comment.
+If this is `false` (the default), emails emitted from the system will not have the Service Request reply to inbound email, and they will just fall back to the (i) jursidiction reply to, or (ii) the system reply to. Likewise, SMS emitted from the system will not handle replies. The rest works when this is `true`.
+
+- Comments POSTed with `broadcastToSubmitter` as `true` are send by email or SMS to the original submitter of the request. Additionally, the submitter can reply to the email and it will write her comment into Gov Flow as a new Service Request Comment.
 - Comments POSTed with `broadcastToAssignee` as `true` are send by email to the current assignee of the request. Additionally, the assignee can reply to the email and it will write her comment into Gov Flow as a new Service Request Comment.
 - Comments POSTed with `broadcastToStaff` as `true` are send by email to the staff of GovFlow of the request. Depending on the jurisdiction configuration, these currently means one of two scenarios:
   - All Staff Users which are admins (the current behaviour of Gov Flow for notifications around a request)
   - If `jurisdiction.enforceAssignmentThroughDepartment` is `true`, and if the Service Request has `departmentId` set, then, only Staff Users which are leads of a department receive the broadcast to email.
 
   `ServiceRequestComment.addedBy` is a field for a string, and generally used to store an ID of a staff user. With the new two-way communication features, this field can also take a constant `__SUBMITTER__` to signify when the comment was created by the submitter.
 
+## When and how SMS or Email chosen as the broadcast channel?
+
+- Submitters can be messaged via email or SMS. If they provided an email address, email will be used. If they provide both email and phone, email will be used. If they provide only phone, SMS will be used.
+- Staff Users will always only have boradcasts via email (there is not way for such users to add a phone number at present).
+
+## SMS and message disambiguation
+
+With email, we issue unique email handles which allow us to route messages efficiently and correctly.
+
+With SMS this is not so trivial. See [Inbound SMS](inbound-sms.md) for info on the various layers of SMS setup.
+
+So, how do we route SMS messages to their correct destination? It basically works like this:
+
+- GovFlow sees an incoming message
+- The incoming message is automatically associated with a Jurisdiction due to a matching InboundMap, and optionally with deeper context (eg, a department) based on the InboundMap configuration
+- If the submitter has no OPEN requests for this InboundMap context, then, create a new Service Request (we disambiguated the incoming message without user involvement)
+- Otherwise, we disambiguate the new message by asking the user in response:
+  - case/ user has one or many tickets: ask her to select from an incremented number list, where each entry is an existing service request (# and snippet of text), and the last entry is "This is a new request"
+- To support the manual disambiguation flow, we store message state in the MessageDisambiguation table. Once a message has been routed correctly, the disambiguation record that backed the interaction with the user is closed.
+
 ## Example request
 
 ```http

src/cli/send-test-dispatch.ts

@@ -13,6 +13,7 @@ import { DispatchConfigAttributes } from '../types';
         twilioAccountSid,
         twilioAuthToken,
         twilioFromPhone,
+        twilioStatusCallbackURL,
         testToEmail,
         testToPhone
     } = app.config;
@@ -24,6 +25,7 @@ import { DispatchConfigAttributes } from '../types';
         fromEmail: sendGridFromEmail as string,
         twilioAccountSid: twilioAccountSid as string,
         twilioAuthToken: twilioAuthToken as string,
+        twilioStatusCallbackURL: twilioStatusCallbackURL as string,
         fromPhone: twilioFromPhone as string,
         toPhone: testToPhone as string
     } as DispatchConfigAttributes;

src/cli/send-test-sms.ts

@@ -6,13 +6,20 @@ import { AppConfig } from '../types';
 
 (async () => {
     const config = await initConfig();
-    const { twilioAccountSid, twilioAuthToken, twilioFromPhone, testToPhone } = config as AppConfig;
+    const {
+        twilioAccountSid,
+        twilioAuthToken,
+        twilioFromPhone,
+        testToPhone,
+        twilioStatusCallbackURL
+    } = config as AppConfig;
     const response = await sendSms(
         twilioAccountSid as string,
         twilioAuthToken as string,
         testToPhone as string,
         twilioFromPhone as string,
         'Test message body.',
+        twilioStatusCallbackURL,
     );
     console.log(response);
 })();

src/config/index.ts

@@ -21,6 +21,7 @@ const defaultConfig: Partial<AppConfig> = {
     'twilioAccountSid': process.env.TWILIO_ACCOUNT_SID || '',
     'twilioAuthToken': process.env.TWILIO_AUTH_TOKEN || '',
     'twilioFromPhone': process.env.TWILIO_FROM_PHONE || '',
+    'twilioStatusCallbackURL': process.env.TWILIO_STATUS_CALLBACK_URL || '',
     'testToEmail': process.env.TEST_TO_EMAIL || '',
     'testToPhone': process.env.TEST_TO_PHONE || '',
     'appPort': parseInt(process.env.APP_PORT || '3000', 10),

src/core/communications/helpers.ts

@@ -142,6 +142,7 @@ export async function dispatchMessage(
             dispatchPayload.toPhone as string,
             dispatchPayload.fromPhone as string,
             dispatchPayload.textBody as string,
+            dispatchPayload.twilioStatusCallbackURL as string,
         );
     } else {
         const errorMessage = `Unknown communication dispatch channel`;
@@ -362,12 +363,17 @@ export async function extractServiceRequestfromInboundEmail(data: InboundEmailDa
     ];
 }
 
-export function canSubmitterComment(submitterEmail: string, validEmails: string[]): boolean {
+export function canSubmitterComment(
+    submitterEmail: string, submitterPhone: string, validEmails: string[], validPhones: string[]
+): boolean {
+    let canSubmit = false;
     if (validEmails.includes(submitterEmail)) {
-        return true
-    } else {
-        return false;
+        canSubmit = true
+    }
+    if (!canSubmit && validPhones.includes(submitterPhone)) {
+        canSubmit = true
     }
+    return canSubmit;
 }
 
 export function verifySendGridWebhook(
@@ -413,4 +419,21 @@ export function getSendFromEmail(jurisdiction: JurisdictionAttributes, defaultSe
 
 export function getSendFromPhone(jurisdiction: JurisdictionAttributes, defaultSendFromPhone: string) {
     return jurisdiction.sendFromPhone ? jurisdiction.sendFromPhone : defaultSendFromPhone;
-}
+}
+
+export function parseDisambiguationChoiceFromText(text: string): number | null {
+    let parsedValue = null;
+    const integerString = text.replace(/\D/g, '');
+    if (integerString) { parsedValue = parseInt(integerString); }
+    return parsedValue;
+}
+
+export function makeDisambiguationMessage(msg: string, choiceMap: Record<number, string>): string {
+    let disambiguationMessage = `${msg}\n\n`;
+    for (const [key, value] of Object.entries(choiceMap)) {
+        let line = `Existing Request #${value}`;
+        if (value == 'New Request') { line = value; }
+        disambiguationMessage = `${disambiguationMessage}${key}. ${line}\n`;
+    }
+    return disambiguationMessage;
+}

src/core/communications/models.ts

@@ -2,6 +2,22 @@ import { DataTypes } from 'sequelize';
 import validator from 'validator';
 import type { ModelDefinition } from '../../types';
 
+export const SMS_EVENT_MAP = {
+    'accepted': null,
+    'queued': null,
+    'sending': null,
+    'receiving': null,
+    'received': null,
+    'canceled': null,
+    'failed': false,
+    'undelivered': false,
+    'sent': true,
+    'delivered': true,
+    'read': true, // whatapp only
+} as Record<string, boolean | null>;
+
+export const SMS_EVENT_MAP_KEYS = Object.keys(SMS_EVENT_MAP);
+
 export const EMAIL_EVENT_MAP = {
     'processed': null, // Message has been received and is ready to be delivered.
     'deferred': null, // Receiving server temporarily rejected the message.
@@ -21,6 +37,16 @@ export const EMAIL_EVENT_IGNORE = ['processed', 'deferred', 'open', 'clicked']
 
 export const EMAIL_EVENT_MAP_KEYS = Object.keys(EMAIL_EVENT_MAP);
 
+export const MESSAGE_DISAMBIGUATION_STATUS_KEYS = [
+    'open',
+    'closed'
+]
+
+export const MESSAGE_DISAMBIGUATION_RESULT_KEYS = [
+    'new-request',
+    'existing-request'
+]
+
 export const CommunicationModel: ModelDefinition = {
     name: 'Communication',
     attributes: {
@@ -198,3 +224,54 @@ export const ChannelStatusModel: ModelDefinition = {
         }
     }
 }
+
+export const MessageDisambiguationModel: ModelDefinition = {
+    name: 'MessageDisambiguation',
+    attributes: {
+        id: {
+            type: DataTypes.UUID,
+            defaultValue: DataTypes.UUIDV4,
+            allowNull: false,
+            primaryKey: true,
+        },
+        submitterId: {
+            type: DataTypes.STRING,
+            allowNull: false,
+        },
+        status: {
+            allowNull: false,
+            type: DataTypes.ENUM(...MESSAGE_DISAMBIGUATION_STATUS_KEYS),
+            defaultValue: MESSAGE_DISAMBIGUATION_STATUS_KEYS[0],
+        },
+        result: {
+            allowNull: true,
+            type: DataTypes.ENUM(...MESSAGE_DISAMBIGUATION_RESULT_KEYS),
+            defaultValue: MESSAGE_DISAMBIGUATION_RESULT_KEYS[0],
+        },
+        originalMessage: {
+            type: DataTypes.TEXT,
+            allowNull: false,
+        },
+        disambiguationFlow: {
+            type: DataTypes.ARRAY(DataTypes.TEXT),
+            allowNull: true,
+        },
+        choiceMap: {
+            type: DataTypes.JSONB,
+            allowNull: true,
+        }
+    },
+    options: {
+        freezeTableName: true,
+        indexes: [
+            {
+                unique: true,
+                fields: ['submitterId']
+            },
+            {
+                unique: true,
+                fields: ['submitterId', 'status']
+            },
+        ]
+    }
+}

src/core/communications/repositories.ts

@@ -11,12 +11,18 @@ import type {
     ICommunicationRepository,
     IEmailStatusRepository,
     IInboundMapRepository,
+    IMessageDisambiguationRepository,
     InboundMapCreateAttributes,
     InboundMapInstance,
+    ISmsStatusRepository,
     LogEntry,
-    Models
+    MessageDisambiguationAttributes,
+    MessageDisambiguationCreateAttributes,
+    MessageDisambiguationInstance,
+    Models,
+    SmsEventAttributes
 } from '../../types';
-import { EMAIL_EVENT_MAP } from './models';
+import { EMAIL_EVENT_MAP, SMS_EVENT_MAP } from './models';
 
 @injectable()
 export class CommunicationRepository implements ICommunicationRepository {
@@ -111,6 +117,68 @@ export class EmailStatusRepository implements IEmailStatusRepository {
 
 }
 
+@injectable()
+export class SmsStatusRepository implements ISmsStatusRepository {
+
+    models: Models;
+    config: AppConfig;
+
+    constructor(
+        @inject(appIds.Models) models: Models,
+        @inject(appIds.AppConfig) config: AppConfig,
+    ) {
+        this.models = models;
+        this.config = config
+    }
+
+    async create(data: ChannelStatusCreateAttributes): Promise<ChannelStatusInstance> {
+        const { ChannelStatus } = this.models;
+        return await ChannelStatus.create(data) as ChannelStatusInstance;
+    }
+
+    async createFromEvent(data: SmsEventAttributes): Promise<ChannelStatusInstance> {
+        const { ChannelStatus } = this.models;
+        const { From, MessageStatus } = data;
+        const eventKey = MessageStatus;
+        const isAllowed = SMS_EVENT_MAP[eventKey];
+        const logEntry = [eventKey, SMS_EVENT_MAP[eventKey]] as LogEntry;
+        const exists = await ChannelStatus.findOne({ where: { id: From } }) as ChannelStatusInstance;
+        let record;
+        if (exists) {
+            exists.isAllowed = isAllowed
+            exists.log.unshift(logEntry)
+            record = exists.save()
+        } else {
+            const log = [logEntry];
+            record = await ChannelStatus.create(
+                { id: From, channel: 'sms', isAllowed, log }
+            ) as ChannelStatusInstance;
+        }
+        return record;
+    }
+
+    async findOne(phone: string): Promise<ChannelStatusInstance | null> {
+        const { ChannelStatus } = this.models;
+        const record = await ChannelStatus.findOne(
+            { where: { id: phone, channel: 'sms' }, order: [['createdAt', 'DESC']] }
+        ) as ChannelStatusInstance | null;
+        return record;
+    }
+
+    async isAllowed(base64Phone: string): Promise<ChannelIsAllowed> {
+        const email = Buffer.from(base64Phone, 'base64url').toString('ascii');
+        const existingRecord = this.findOne(email);
+        let isAllowed = true;
+        // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+        // @ts-ignore
+        if (existingRecord && existingRecord.isAllowed === false) {
+            isAllowed = false;
+        }
+        return { id: base64Phone, isAllowed };
+    }
+
+}
+
 @injectable()
 export class InboundMapRepository implements IInboundMapRepository {
 
@@ -138,4 +206,46 @@ export class InboundMapRepository implements IInboundMapRepository {
         return record;
     }
 
+}
+
+@injectable()
+export class MessageDisambiguationRepository implements IMessageDisambiguationRepository {
+
+    models: Models;
+    config: AppConfig;
+
+    constructor(
+        @inject(appIds.Models) models: Models,
+        @inject(appIds.AppConfig) config: AppConfig,
+    ) {
+        this.models = models;
+        this.config = config
+    }
+
+    async create(data: MessageDisambiguationCreateAttributes): Promise<MessageDisambiguationAttributes> {
+        const { MessageDisambiguation } = this.models;
+        return await MessageDisambiguation.create(data) as MessageDisambiguationInstance;
+    }
+
+    async findOne(jurisdictionId: string, submitterId: string): Promise<MessageDisambiguationAttributes> {
+        const { MessageDisambiguation } = this.models;
+        const params = { where: { jurisdictionId, submitterId, status: 'open' } }
+        const record = await MessageDisambiguation.findOne(params) as MessageDisambiguationInstance;
+        return record;
+    }
+
+    async update(
+        id: string,
+        data: Partial<MessageDisambiguationAttributes>
+    ): Promise<MessageDisambiguationAttributes> {
+        const { MessageDisambiguation } = this.models;
+        const params = { where: { id } };
+        const record = await MessageDisambiguation.findOne(params) as MessageDisambiguationInstance;
+        for (const [key, value] of Object.entries(data)) {
+            // eslint-disable-next-line @typescript-eslint/ban-ts-comment
+            // @ts-ignore
+            record[key] = value;
+        }
+        return await record.save();
+    }
 }