Skip to content
This repository has been archived by the owner on Oct 19, 2021. It is now read-only.

SendGrid Responder user guide

Jump to bottom
Alex Dean edited this page Jun 19, 2017 · 6 revisions

HOME > GUIDE FOR ANALYSTS > RESPONDERS > SENDGRID RESPONDER USER GUIDE

See also: SendGrid Responder setup guide.

Contents

1. Overview

This Sauna responder lets you:

  1. Export user-level data from your event warehouse and upload this data to SendGrid for use in email marketing
  2. Send emails to recipients

2. Responder actions

Currently this responder only supports one action:

Type File landing path for Sauna Action performed in SendGrid Version
Batch com.sendgrid.contactdb/recipients Upload recipients using the Contacts API (batch) v0.1.0
Command com.sendgrid.sauna.commands/send_email Sends an email v0.3.0

2.1 Upload recipients (batch)

2.1.1 Overview

This responder path lets you upload email recipients into the SendGrid Contacts Database, which is part of the SendGrid Marketing Campaigns suite. The responder uses the SendGrid Contacts API to do so.

2.2.1 File landing path

The format for the path where files should land is:

<sauna_landing_root>/com.sendgrid.contactdb/recipients/v1/tsv:<attribute_1>,<attribute_2>,<attribute_3>,.../<sub_folders/...>

Notes on this:

  • Your <sauna_landing_root> will be an S3 bucket/path or local folder as appropriate.
  • com.sendgrid.contactdb/recipients specifies our responder action.
  • This is v1, version 1, of this responder path.
  • Currently tsv (for tab-separated values) is the only supported input format.
  • Because the SendGrid Contacts Database, lets you define custom fields, you must list each of the fields you want to upload after the tsv: prefix. You must include the email field.
  • You can add as many sub-folders as required to prevent clashes with other Sauna users in your organization.

2.2.3 File input format

At the responder path, you must upload files which contain:

  • Tab (\t)-separated values encoded in double quotation marks (");
  • Rows separated by newlines (\n).

Fields should be as follows:

"<attribute_1>"\t"<attribute_2>"\t"<attribute_3>"\t"<...>"\n

Where:

  • The order of <attribute>s exactly matches the order given in your tsv: file landing path.
  • The value for each <attribute> exactly matches the datatype in your datasource configuration.
  • One of the columns must be for the email (it doesn't have to be the first column however).

2.1.4 Response algorithm

Sauna will take the file(s) found at the file landing path and:

  1. Split each file and split it into groups of 1,000 recipients;
  2. For each group, perform a POST request to the v3/contactdb/recipients API endpoint;
  3. Transparently respect SendGrid rate limiting;
  4. Parse the SendGrid response to identify any failed rows and record them in <sauna_failure_path>. (out of scope for Sauna v1)

Let's go through each of these steps in turn:

2.1.4.1 Splitting into groups of 1,000

Sauna will split each file into groups of 1,000 email recipients.

2.1.4.2 Making the POST request

Sauna will make a POST request for each group to the v3/contactdb/recipients API endpoint.

To format the fields correctly in the POST body, Sauna will first make a GET request to the v3/contactdb/custom_fields endpoint. This will yield the type information for each custom field, which Sauna will then use to correctly format the recipient POST requests.

In terms of specific type transformations, dates and timestamps will be represented as Unix epoch timestamps.

2.1.4.3 Respecting rate-limiting

According to documentation, SendGrid only allows 3 POST requests to the recipients endpoint every 2 seconds, resulting in an effective maximum upload rate of 1,500 recipients per second. Sauna will respect this rate-limit, performing backoff-and-retry for failed POSTs.

2.1.4.4 Handling failed recipients

SendGrid's response to each POST request reports on any failures in the group, providing their index in the uploaded group and the associated error message.

2.1.5 Troubleshooting

Make sure that your listing of attributes in your file landing path exactly matches the column order of your exported data.

Don't attempt to reformat your timestamps into the format of the attribute's datetime; leave it as-is and Sauna will handle the conversion.

Remember that Sauna can only upload 1,500 recipients per second, so a large batch of recipients could take some minutes to complete.

2.2.6 Usage examples

2.2.6.1 Our custom fields

Let's imagine that we have configured the following custom fields in SendGrid:

Name Datatype
email text
birthday date
middle_name text
favorite_number number
when_promoted date

Note that there are only three datatypes, according to documentation.

Let's now populate this datasource using Sauna.

2.2.6.2 Local folder

Assuming that your Sauna root is set to local folder /opt/sauna, and the file lands as:

/opt/sauna/com.sendgrid.contactdb/recipients/v1/tsv:email,birthday,middle_name,favorite_number,when_promoted/ua-team/joe/warehouse.tsv

The contents of warehouse.tsv are as follows:

"bob@foo.com"	"1980-06-21"	"Al"	"13"	"2013-12-15 14:05:06.789"
"karl@bar.de"	"1975-07-02"	""	"12"	"2014-06-10 21:48:32.712"
"ed@me.co.uk"	"1992-09-12"	"Jo"	"98"	"2015-01-28 07:32:16.329"

Here we can see that Joe in the User Acquisition team wants to upload three contacts to SendGrid as email recipients.

Some notes on data formats:

  • The date format must be ISO 8601, i.e. YYYY-MM-DD.
  • The timestamp format must be ISO 8601 with milliseconds but no timezone information.
  • Use an empty string "" to represent a null value.
2.1.6.3 Redshift UNLOAD

UNLOAD is a Redshift SQL command which lets you export the contents of a SELECT statement to Amazon S3 for further processing.

Assuming that your Sauna root is set to S3 folder s3://my-sauna-bucket/prod and you run the following SQL:

CREATE TABLE email_recipients (
  email_address    varchar(256) NOT NULL,
  birthday         date         NOT NULL,
  middle_name      varchar(256) NULL,
  favorite_number  integer      NOT NULL,
  when_promoted    timestamp    NOT NULL,
  city             varchar(256) NOT NULL
);

INSERT INTO email_recipients VALUES
  ('bob@foo.com', '1980-06-21', 'Al', '13', '2013-12-15 14:05:06.789', 'Boston'),
  ('karl@bar.de', '1975-07-02', null, '12', '2014-06-10 21:48:32.712', 'Berlin'),
  ('ed@me.co.uk', '1992-09-12', 'Jo', '98', '2015-01-28 07:32:16.329', 'Bolton');

UNLOAD ('select email_address, birthday, middle_name, favorite_number, when_promoted from email_recipients')
  TO 's3://my-sauna-bucket/prod/com.sendgrid.contactdb/recipients/v1/tsv:email,birthday,middle_name,favorite_number,when_promoted/ua-team/joe/'
  CREDENTIALS 'aws_access_key_id=<access-key-id>;aws_secret_access_key=<secret-access-key>'
  DELIMITER AS '\t'
  ADDQUOTES
  PARALLEL OFF;

Again, this will export 3 customer profiles in a TSV to Amazon S3, where Sauna is waiting to detect this file landing and upload to Sendgrid. Some notes:

  • The order of our fields in UNLOAD ('select must match the order given in the tsv:... file landing path.
  • Although we have the city column defined in our email_recipients table, we choose not to upload it to SendGrid.
  • Note that the column email_address must be referred to as email in the Sauna landing path (tsv:email,...).
2.1.6.4 Implementation

From both of the examples above the following POST calls to SendGrid would be made, creating three new recipients in the SendGrid Contacts Database:

$ curl \
  -H "Authorization: Bearer 546ffe34ds" \
  -H "Content-Type: application/json" \
  -d '[{"email":"bob@foo.com","birthday":330393600,"middle_name":"Al","favorite_number":13,"when_promoted":1387116306},{"email":"karl@bar.de","birthday":173491200,"middle_name":null,"favorite_number":12,"when_promoted":1402436912},{"email":"ed@me.co.uk","birthday":716256000,"middle_name":"Jo","favorite_number":98,"when_promoted":1422430336}]' \
  -X POST https://api.sendgrid.com/v3/contactdb/recipients

Here is the JSON pretty-printed:

[
  {
    "email": "bob@foo.com",
    "birthday": 330393600,
    "middle_name": "Al",
    "favorite_number": 13,
    "when_promoted": 1387116306
  },
  {
    "email": "karl@bar.de",
    "birthday": 173491200,
    "middle_name": null,
    "favorite_number": 12,
    "when_promoted": 1402436912
  },
  {
    "email": "ed@me.co.uk",
    "birthday": 716256000,
    "middle_name": "Jo",
    "favorite_number": 98,
    "when_promoted": 1422430336
  }
]

2.2 Send email (real-time)

2.2.1 Overview

This responder command lets you send an email to recipients using SendGrid's Web API v3.

Like all real-time responder actions, this action is triggered by a well-structured JSON command being received by a compatible observer.

2.2.3 Command format

The command must be configured using a self-describing JSON Schema which validates against this schema:

[iglu:com.opsgenie.sauna.commands/send_email/jsonschema/1-0-0][schema]

This is a very feature-rich command. For a full explanation of all of these parameters, please see the excellent SendGrid guide to the available Request Body Parameters.

NOTE TO TREVOR: PLEASE ADD AS MANY OF THE REQUEST BODY PARAMETERS AS YOU CAN TO THIS COMMAND'S JSON SCHEMA

2.2.3 Example commands

2.2.3.1 Simple send email

A simple SendGrid send_email command will look as follows:

{
  "schema": "iglu:com.opsgenie.sauna.commands/send_email/jsonschema/1-0-0",

  "data": {
    "personalizations": [
      {
        "to": [
          {
            "email": "john@example.com"
          }
        ],
        "subject": "Hello, World!"
      }
    ],
    "from": {
      "email": "from_address@example.com"
    },
    "content": [
      {
        "type": "text/plain",
        "value": "Hello, World!"
      }
    ]
  }
}

Where:

  • personalizations is a non-empty array of up to 1,000 messages and their metadata, like the "envelope" for your email. See the SendGrid documentation on Personalizations for more information
  • from is an object containing the email address and name of the sender; Unicode is not supported here
  • content is the an array in which you may specify the contents of your email, providing the mime type and actual content

This command will send a simple "Hello world" email to a single recipient, john@example.com.

2.2.3.2 Send template to multiple recipients

Here is a more complex example:

SOMETHING WITH MULTIPLE RECIPIENTS AND A TEMPLATE

2.2.4 Response algorithm

As with all RT responders, Sauna will take each command and:

  1. Validate it as a valid SendGrid send_email command
  2. If it is not valid, this will be reported to any configured Sauna loggers
  3. If it is valid, Sauna will attempt to trigger the event using the Mail component of the SendGrid Web API v3
  4. If the API reports success (200), this will be reported to any configured Sauna loggers
  5. If the API reports failure (400/403/etc), this will be reported to any configured Sauna loggers but no retry will be attempted

In the case of failure, we do not attempt retry, even in the case of us temporarily exceeding API rate limits, because this could block other non-SendGrid-related commands in the observed stream from executing.

2.2.5 Usage examples

2.2.5.1 Kinesis stream

Assuming that the Amazon Kinesis Observer receives the following command:

{
  "schema": "iglu:com.snowplowanalytics.sauna.commands/command/jsonschema/1-0-0",

  "data": {
    "envelope": {
      "schema": "iglu:com.snowplowanalytics.sauna.commands/envelope/jsonschema/1-0-0",
      "data": {
        "commandId": "9dadfc92-9311-43c7-9cee-61ab590a6e81",
        "whenCreated": "2017-01-02T19:14:42Z",
        "execution": {
          "semantics": "AT_LEAST_ONCE",
          "timeToLive": 1200000
        },
        "tags": {}
      }
    },
    "command": {
      "schema": "iglu:com.sendgrid.sauna.commands/send_email/jsonschema/1-0-0",
      "data": {
        "personalizations": [
          {
            "to": [
              {
                "email": "john@example.com"
              }
            ],
            "subject": "Hello, World!"
          }
        ],
        "from": {
          "email": "from_address@example.com"
        },
        "content": [
          {
            "type": "text/plain",
            "value": "Hello, World!"
          }
        ]
      }
      }
    }
  }
}

And assuming that the current time is within 20 minutes (1,200,000 ms) of 2017-01-02T19:14:42Z, then:

  • Sauna will send a simple "Hello world" email to a single recipient, john@example.com
  • Whether or not the event was successfully send will be reported to any configured Sauna loggers.

Home | Guides for: analysts § devops § developers | Copyright © 2015-2017 Snowplow Analytics Ltd

Home

Guide for Analysts

Guide for Devops

Guide for Developers

Common Terms

Clone this wiki locally