Bring Slack Conversation History into Airtable

data_source_examples/slack_conversation_history/using_airtable.js/.env.example

@@ -0,0 +1,4 @@
+AIRTABLE_API_KEY=keyXXXX
+AIRTABLE_BASE_ID=appXXXX
+AIRTABLE_TABLE_ID=tblXXXX
+AIRTABLE_UNIQUE_FIELD_NAME=Unique ID

data_source_examples/slack_conversation_history/using_airtable.js/.gitignore

@@ -0,0 +1,2 @@
+*.env
+!.env.example

data_source_examples/slack_conversation_history/using_airtable.js/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Airtable Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data_source_examples/slack_conversation_history/using_airtable.js/README.md

@@ -0,0 +1,66 @@
+# Bring Slack Conversation History into Airtable
+
+This code example can be used to import messages from a Slack conversation (public or private channel, DM, or multi-party DM) into an Airtable table. You can also schedule this script to run on a recurring to keep your table in Airtable up to date with the new messages from Slack.
+
+This code example is based on [the generic airtable.js upsert example](.../../../../../javascript/using_airtable.js/) and uses [`airtable.js`](https://github.com/airtable/airtable.js) to interact with the Airtable REST API and [`@slack/web-api`](https://slack.dev/node-slack-sdk/web-api) to interact with Slack's API.
+
+---
+
+The software made available from this repository is not supported by Formagrid Inc (Airtable) or part of the Airtable Service. It is made available on an "as is" basis and provided without express or implied warranties of any kind.
+
+---
+
+## Setup steps
+
+This section will walk you through setting up three components:
+
+- A. An Airtable base where this example script will store messages from the Slack conversation you specify
+- B. A custom Slack app and retrieving your Slack bot token, allowing you to retrieve messages from Slack
+- C. This script (which will pull messages from the Slack converstion, compare it to existing records in you base, and create-or-update records in your base so they mirror the latest messages in Slack)
+
+### A. Airtable Base Setup
+
+First, create a table in a base you have creator-level access to and create a table with the following fields: 'Channel ID + TS' (Single line text), 'Channel ID' (Single line text), 'TS' (Single line text), 'Last Edited TS' (Single line text), 'Type' (Single line text), and 'Subtype' (Single line text), 'Slack User ID' (Single line text), 'Message text' (Long text), 'Reply Count' (Number), 'Parent Message' (self-linking Linked Record), and 'Full Message Payload (JSON)' (Long text).
+
+- You may also want to add a formula field named 'TS (Human Readable)' with the formula `DATETIME_PARSE({TS}, 'X')` and another named 'Parent or Thread?' with the formula `IF({Parent Message},"Threaded reply","Parent message")`.
+- You can create a copy of a sample table with these fields [from here](https://airtable.com/shrB2653wGPc4KwoZ) by selecting "Use data" in the top right corner.
+- If you choose other field names, be sure to update the code in [index.js](./index.js) in the `convertSlackMessageToAirtableRecord` function.
+
+### B. Slack App Setup
+
+Follow these instructions to set up a Slack app, retrieve an API token, and add your new bot to the channel(s) you want to extract history from.
+
+1. Create a custom app from https://api.slack.com/apps?new_app=1, making sure to select the Slack workspace that contains the channel you want to extract messages from.
+2. Add a **bot token scope** from your new app's 'OAuth & Permissions' page. If you are extracting messages from a public mesage, you'll want to add `channels:history`. For private channels add `groups:history`. For DMs or MPDMs, use `im:history` or `mpim:history`, respectively. 
+3. Retrieve a **bot user token** for your workspace by clicking 'Install to Workspace' near the top of your app's 'OAuth & Permissions' page. This may require Slack admin approval.
+4. Once the app has been installed, be sure to add the app to the channel you want to extract messages from. You can type `/invite @<your app name here>`.
+
+### C. Configuring and Running this Script
+
+Finally, let's setup this script to run locally. You can later deploy this as a scheduled task on a server or cloud function.
+
+1. Clone/unzip code
+2. Copy `.env.example` to `.env` and populate values (see below for details on each environment variable)
+3. Install node dependencies including Airtable and Slack's official SDKs by running `npm install`
+4. Trigger the script to execute by running `npm run sync`
+5. Records in the specified Airtable base's table should be created. Try adding new messages to the Slack channel and re-run the previous step.
+
+### Key Files and Their Contents
+
+- [`index.js`](index.js) is the main code file which is executed when `npm start` is run. At a high level, it performs the following:
+  - Loads dependencies, helper functions, and configuration variables
+  - Initializes API clients for the Airtable and Slack APIs
+  - Retrieves Slack messages (both "parent" and "threaded replies") from the Slack API and transforms the objects to be flat and suitable for our base template (setup section A)
+  - Retrieves all existing records in the Airtable base and creates a mapping of the unqiue field's value to the existing record ID for later updating
+  - First loops through each of the parent messages, determining if a new record needs to be created or an existing record updated. This step then repeats for all threaded replies.
+    - Note: This example is more complicated than the generic examples as we first loop through all parent Slack messages to update-or-create them and then loop through all threaded replies so that they can be properly associated them with the correct parent Slack message. This can be observed in [index.js](./index.js); look for `// First, upsert all parent messages, and then upsert all threaded replies`
+- `*_helpers.js`
+  - [`airtable_helpers.js`](airtable_helpers.js) is referenced by [`index.js`](index.js) and contains helper functions to for batching Airtable record actions.
+  - [`slack_helpers.js`](slack_helpers.js) is referenced by [`index.js`](index.js) and contains a helper function to recursively fetch Slack messages.
+- [`.env.example`](.env.example) is an example file template to follow for your own `.env` file. The environment variables supported are:
+  - `AIRTABLE_API_KEY` - [your Airtable API key](https://support.airtable.com/hc/en-us/articles/219046777-How-do-I-get-my-API-key-); it will always start with `key`
+  - `AIRTABLE_BASE_ID` - the ID of your base; you can find this on the base's API docs from https://airtable.com/api. This will always start with `app`
+  - `AIRTABLE_TABLE_ID` - the ID of the table you want to create/update records in; you can find this in the URL of your browser when viewing the table. It will start with `tbl`
+  - `AIRTABLE_UNIQUE_FIELD_NAME` - the field name of the field that is used for determining if an existing records exists that needs to be updated (if no record exists, a new one will be created)
+  - `SLACK_BOT_TOKEN` - the Slack API key you retrieved in setup section B. It will likely start with "xoxb-"
+  - `SLACK_CHANNEL_ID` - the Slack channel ID you want to extract messages from. One way to find a channel ID is to right click the channel name, select "Copy link", and paste the value to a notepad. The channel ID will be what follows the last "/" and start with a C, D, or X. 

data_source_examples/slack_conversation_history/using_airtable.js/airtable_helpers.js

@@ -0,0 +1,36 @@
+// Helper function from https://stackoverflow.com/questions/8495687/split-array-into-chunks
+const chunkArray = function (arrayToChunk, chunkSize = 10) {
+  const arraysOfChunks = []
+  for (let i = 0; i < arrayToChunk.length; i += chunkSize) { arraysOfChunks.push(arrayToChunk.slice(i, i + chunkSize)) }
+  return arraysOfChunks
+}
+
+// Helper function to act on a chnunk of records
+const actOnRecordsInChunks = async function (table, createOrUpdate, records) {
+  console.log(`\n${createOrUpdate}'ing ${records.length} records`)
+  const arrayOfChunks = chunkArray(records)
+  for (const chunkOfRecords of arrayOfChunks) {
+    console.log(`\tProcessing batch of ${chunkOfRecords.length} records`)
+    if (createOrUpdate === 'create') {
+      await table.create(chunkOfRecords, { typecast: true }) // typecast=true so that we can specify values instead of record IDs with the Airtable REST API
+    } else if (createOrUpdate === 'update') {
+      await table.update(chunkOfRecords, { typecast: true })
+    } else {
+      throw new Error(`Unexpected value for createOrUpdate: ${createOrUpdate}`)
+    }
+  }
+}
+
+// Helper function that takes an array of records and returns a mapping of primary field to record ID
+const createMappingOfUniqueFieldToRecordId = function (records, fieldName) {
+  const mapping = {}
+  for (const existingRecord of records) {
+    mapping[existingRecord.fields[fieldName]] = existingRecord.id
+  }
+  return mapping
+}
+
+module.exports = {
+  createMappingOfUniqueFieldToRecordId,
+  actOnRecordsInChunks
+}

data_source_examples/slack_conversation_history/using_airtable.js/index.js

@@ -0,0 +1,121 @@
+// Use dotenv to read the .env file and load into process.env
+require('dotenv').config()
+
+// Load external dependencies
+const Airtable = require('airtable')
+const { WebClient: SlackWebClient } = require('@slack/web-api')
+
+// Load helper functions from *_helpers.js
+const { createMappingOfUniqueFieldToRecordId, actOnRecordsInChunks } = require('./airtable_helpers')
+const { getMessages } = require('./slack_helpers')
+
+// Define variables and initialize Airtable client
+const { AIRTABLE_API_KEY, AIRTABLE_BASE_ID, AIRTABLE_TABLE_ID, AIRTABLE_UNIQUE_FIELD_NAME } = process.env
+Airtable.configure({ apiKey: AIRTABLE_API_KEY })
+const base = Airtable.base(AIRTABLE_BASE_ID)
+const table = base(AIRTABLE_TABLE_ID)
+
+// Define variables and initialize Slack WebClient
+const { SLACK_BOT_TOKEN, SLACK_CHANNEL_ID } = process.env;
+const slackWebClient = new SlackWebClient(SLACK_BOT_TOKEN);
+
+// Define a helper function to convert a Slack message object to an Airtable record
+const convertSlackMessageToAirtableRecord = function (msg) {
+  return {
+    [AIRTABLE_UNIQUE_FIELD_NAME]: `${SLACK_CHANNEL_ID}-${msg.ts}`,
+    'Channel ID': SLACK_CHANNEL_ID,
+    'TS': msg.ts,
+    'Type': msg.type,
+    'Subtype': msg.subtype,
+    'Slack User ID': msg.user,
+    'Message Text': msg.text,
+    'Reply Count': msg.reply_count,
+    'Last Edited TS': msg.edited ? msg.edited.ts : null,
+    'Full Message Payload (JSON)': JSON.stringify(msg, null, 2),
+    // If msg.thread_ts exists and is different from msg.ts, then this is a threaded message; set the parent message accordingly:
+    'Parent Message': (msg.thread_ts && (msg.thread_ts != msg.ts)) ? [`${SLACK_CHANNEL_ID}-${msg.thread_ts}`] : null,
+  }
+}
+
+  ;
+
+(async () => {
+
+  // Get all parent messages from Slack for the given channel
+  const allParentMessages = await getMessages(slackWebClient, 'conversations.history', { channel: SLACK_CHANNEL_ID })
+
+  // Create an array of threaded replies too
+  const allParentMessagesWithReplies = allParentMessages.filter(message => message.reply_count > 0)
+  const allThreadedReplies = await Promise.all(allParentMessagesWithReplies.map(async message => {
+    const replies = await getMessages(slackWebClient, 'conversations.replies', { ts: message.ts, channel: SLACK_CHANNEL_ID })
+    const repliesWithoutOriginalMessage = replies.filter(reply => reply.thread_ts !== reply.ts)
+    return repliesWithoutOriginalMessage
+  }))
+
+  // Define two new inputRecords arrays that converts the objects received from the Slack API to use Airtable field names
+  const parentMessageInputRecords = allParentMessages.map(msg => convertSlackMessageToAirtableRecord(msg))
+  const threadedRepliesInputRecords = allThreadedReplies.flat().map(msg => convertSlackMessageToAirtableRecord(msg))
+
+  ////////////////////////////////////////////////////////////////////
+  // Note: you should not need to edit the code below this comment
+  ////////////////////////////////////////////////////////////////////
+
+  // Retrieve all existing records from the base through the Airtable REST API
+  const existingRecords = await table.select().all()
+
+  // Create an object mapping of the primary field to the record ID
+  // Remember, it's assumed that the AIRTABLE_UNIQUE_FIELD_NAME field is truly unique
+  const mapOfUniqueIdToExistingRecordId = createMappingOfUniqueFieldToRecordId(existingRecords, AIRTABLE_UNIQUE_FIELD_NAME)
+
+  // First, upsert all parent messages, and then upsert all threaded replies
+  // (This way threaded replies can be properly linked to their parent message
+  //  which will exist in the base by the time threaded replies are upserted)
+  for (const input of Object.entries({ parentMessageInputRecords, threadedRepliesInputRecords })) {
+    const [inputRecordType, inputRecords] = input
+
+    // Create two arrays: one for records to be created, one for records to be updated
+    const recordsToCreate = []
+    const recordsToUpdate = []
+
+    // For each input record, check if it exists in the existing records. If it does, update it. If it does not, create it.
+    console.log(`Processing ${inputRecords.length} ${inputRecordType} to determine whether to update or create`)
+    for (const inputRecord of inputRecords) {
+      const recordUniqueFieldValue = inputRecord[AIRTABLE_UNIQUE_FIELD_NAME]
+      console.debug(`\tProcessing record w/ '${AIRTABLE_UNIQUE_FIELD_NAME}' === '${recordUniqueFieldValue}'`)
+      // Check for an existing record with the same unique ID as the input record
+      const recordMatch = mapOfUniqueIdToExistingRecordId[recordUniqueFieldValue]
+
+      if (recordMatch === undefined) {
+        // Add record to list of records to update
+        console.log('\t\tNo existing records match; adding to recordsToCreate')
+        recordsToCreate.push({ fields: inputRecord })
+      } else {
+        // Add record to list of records to create
+        console.log(`\t\tExisting record w/ ID ${recordMatch} found; adding to recordsToUpdate`)
+        recordsToUpdate.push({ id: recordMatch, fields: inputRecord })
+      }
+    }
+
+    // Read out array sizes
+    console.log(`\nRecords to create: ${recordsToCreate.length}`)
+    console.log(`Records to update: ${recordsToUpdate.length}\n`)
+
+    try {
+      // Perform record creation
+      await actOnRecordsInChunks(table, 'create', recordsToCreate)
+
+      // Perform record updates on existing records
+      await actOnRecordsInChunks(table, 'update', recordsToUpdate)
+      console.log(`\n\nFinished processing ${inputRecordType}`)
+
+    } catch (error) {
+      console.error('An error occured while creating or updating records using the Airtable REST API')
+      console.error(error)
+      throw (error)
+    }
+
+  }
+
+  console.log('\n\nScript execution complete')
+
+})()