Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial commit of ruby example #13

Merged
merged 4 commits into from
Apr 6, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions ruby/using_airrecord/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
AIRTABLE_API_KEY=keyXXXX
AIRTABLE_BASE_ID=appXXXX
AIRTABLE_TABLE_ID=tblXXXX
AIRTABLE_UNIQUE_FIELD_NAME=Unique ID
2 changes: 2 additions & 0 deletions ruby/using_airrecord/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.env
!.env.example
3 changes: 3 additions & 0 deletions ruby/using_airrecord/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
source 'https://rubygems.org'
gem 'airrecord'
gem 'dotenv'
29 changes: 29 additions & 0 deletions ruby/using_airrecord/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
GEM
remote: https://rubygems.org/
specs:
airrecord (1.0.9)
faraday (>= 0.10, < 3.0)
faraday-net_http_persistent
net-http-persistent
connection_pool (2.2.5)
dotenv (2.7.6)
faraday (2.2.0)
faraday-net_http (~> 2.0)
ruby2_keywords (>= 0.0.4)
faraday-net_http (2.0.1)
faraday-net_http_persistent (2.0.1)
faraday-net_http
net-http-persistent (~> 4.0)
net-http-persistent (4.0.1)
connection_pool (~> 2.2)
ruby2_keywords (0.0.5)

PLATFORMS
x86_64-darwin-21

DEPENDENCIES
airrecord
dotenv

BUNDLED WITH
2.3.10
21 changes: 21 additions & 0 deletions ruby/using_airrecord/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2021 Airtable Labs

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
39 changes: 39 additions & 0 deletions ruby/using_airrecord/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Airtable Upsert Example w/ [airrecord](https://github.com/sirupsen/airrecord)

This example uses [airrecord](https://github.com/sirupsen/airrecord), a community supported Ruby API library for the [Airtable REST API](https://airtale.com/api), to process input records and compare a unique field with existing records in an Airtable base. If the unique value is present in an existing record, the existing record will be updated. If the unique value is not found, a new record will be created.

The example code in this repository assumes your base has a table with the following fields: First Name (Single line text), Last Name (Single line text), Uniue ID (Single line text), Job Title (Single line text), and Hire Number (Number). You can creatqe a copy of a sample base with 200 records prepopulated [here](https://airtable.com/shrgakIqrpwtkQL2p).

---

The software made available from this repository is not supported by Formagrid Inc (Airtable) or part of the Airtable Service. It is made available on an "as is" basis and provided without express or implied warranties of any kind.

---

### Local setup
1. Clone/unzip code
2. Copy `.env.example` to `.env` and populate values (details below)
3. Install [Bundler](https://bundler.io/) and run `bundle install` to install the dependencies listed in the `Gemfile`
4. (Optional) Modify `input_records` in `example.rb` with new static values or dynamically fetched values from your source of choice (API, file, etc.)
5. Run `ruby example.rb` to run the script

### Key files and their contents
- [`example.rb`](example.rb) is the main code file which is executed when `ruby example.rb` is run. At a high level, it performs the following:
- Loads dependencies and configuration variables
- Defines a sample `input_records` array which should be modified to reference an external data source
- Retrieves all existing records in the Airtable base and creates a mapping of the unqiue field's value to the existing record ID for later updating
- Loops through each record from `input_records` array and determines if an existing record should be updated or a new one should be created
- In chunks of 10, updates existing and creates new records
- [`.env.example`](.env.example) is an example file template to follow for your own `.env` file. The environment variables supported are:
- `AIRTABLE_API_KEY` - [your Airtable API key](https://support.airtable.com/hc/en-us/articles/219046777-How-do-I-get-my-API-key-); it will always start with `key`
- `AIRTABLE_BASE_ID` - the ID of your base; you can find this on the base's API docs from https://airtable.com/api. This will always start with `app`
- `AIRTABLE_TABLE_ID` - the ID of the table you want to create/update records in; you can find this in the URL of your browser when viewing the table. It will start with `tbl`
- `AIRTABLE_UNIQUE_FIELD_NAME` - the field name of the field that is used for determining if an existing records exists that needs to be updated (if no record exists, a new one will be created)

### Notes
- [airrecord](https://github.com/sirupsen/airrecord) [handles API rate limiting](https://github.com/sirupsen/airrecord#throttling)
- [airrecord](https://github.com/sirupsen/airrecord) does not support [bulk operations](https://github.com/sirupsen/airrecord/issues/66) and so each update and create request is made
- The field used for uniqueness does not have to be the primary field.
- The field name for the unique field is expected to remain consistent. If it changes, update the environment variable
- Each existing and new record is expected to have a value for the field used for uniqueness.
- [Mockaroo](https://www.mockaroo.com/) was used to generate example data used in this example.
93 changes: 93 additions & 0 deletions ruby/using_airrecord/example.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Load dependencies
require 'bundler/setup'
require 'airrecord' # to interact with Airtable REST API
require 'dotenv' # to load .env files with environment variables

# Load .env file
Dotenv.load

# Configuration variables for Airtable
AIRTABLE_API_KEY = ENV.fetch('AIRTABLE_API_KEY')
AIRTABLE_BASE_ID = ENV.fetch('AIRTABLE_BASE_ID')
AIRTABLE_TABLE_ID = ENV.fetch('AIRTABLE_TABLE_ID')
AIRTABLE_UNIQUE_FIELD_NAME = ENV.fetch('AIRTABLE_UNIQUE_FIELD_NAME')

# Initialize Airtable client using Airrecord's 'ad-hoc API' (https://github.com/sirupsen/airrecord#ad-hoc-api)
Table = Airrecord.table(AIRTABLE_API_KEY, AIRTABLE_BASE_ID, AIRTABLE_TABLE_ID)

# Define input records(from the source system). This would usually be an API call or reading in a CSV or other format of data.
input_records = [
# Existing person in the table, if using the sample data linked to in the README
{
'First Name': 'Juliette',
'Last Name': 'Schimmang',
'Unique ID': '16a05ea5-7bbd-4353-bc25-878a2245835e',
'Job Title': 'Account Executive II'
},
# New user to be added to the table
{
'First Name': 'Marsha',
'Last Name': 'Rickeard',
'Unique ID': 'bf68da9d-805b-4117-90dc-d54eb46db19f',
'Job Title': 'CTO',
'Hire Number': 201
}
]

# Retrieve all existing records from the base through the Airtable REST API
all_existing_records = Table.all
puts "#{all_existing_records.count} existing records found"

# Create an object mapping of the primary field to the record ID
# Remember, it's assumed that the AIRTABLE_UNIQUE_FIELD_NAME field is truly unique
upsert_field_value_to_existing_record = {}
all_existing_records.each do |existing_record|
upsert_field_value_to_existing_record[existing_record[AIRTABLE_UNIQUE_FIELD_NAME]] = existing_record
end

# Create two arrays: one for records to be created, one for records to be updated
records_to_create = []
records_to_update = []

# # For each input record, check if it exists in the existing records. If it does, update it. If it does not, create it.
puts "\nProcessing #{input_records.count} input records to determine whether to update or create"
input_records.each do |input_record|
record_unique_value = input_record[AIRTABLE_UNIQUE_FIELD_NAME.to_sym]
puts " Processing record w / \'#{AIRTABLE_UNIQUE_FIELD_NAME}\' == \'#{record_unique_value}\'"
existing_record_based_on_upsert_field_value_maybe = upsert_field_value_to_existing_record[record_unique_value]

# and if the upsert field value matches an existing one...
if existing_record_based_on_upsert_field_value_maybe
# Add record to list of records to update
puts " Existing record w / ID #{existing_record_based_on_upsert_field_value_maybe.id} found; adding to records_to_update"
records_to_update.push({:existing_record => existing_record_based_on_upsert_field_value_maybe, :input_record => input_record})
else
# Otherwise, add record to list of records to create
puts ' No existing records match; adding to records_to_create'
records_to_create.push(input_record)
end
end

# Read out array sizes
puts "\n#{records_to_create.count} records to create"
puts "#{records_to_update.count} records to update"

# Perform record creation, one at a time (Airrecord does not support batch updates at the moment)
records_to_create.each do |record_to_create|
Table.create(record_to_create)
end

# Perform record updates on existing records, one at a time (Airrecord does not support batch updates at the moment)
records_to_update.each do |record_to_update|
existing_record = record_to_update[:existing_record]
input_record = record_to_update[:input_record]

# Loop through all input records' fields and set the existing record's fields to the input record's fields
input_record.keys.each do |field_name|
existing_record[field_name.to_s] = input_record[field_name]
end
# Save the record
existing_record.save
end

puts "\n\nScript execution complete!"