Skip to content

How do I import external data into a database?

Jump to bottom
Joel "The Gregarious" Meador edited this page Jun 4, 2026 · 1 revision

In the event that you should need to import data from an external source (e.g. Airtable) into a database for a SmartLogic project, what are some common "gotchas" you should be aware of? And what are some good practices to ensure data integrity?

  1. Never make assumptions about data integrity from a legacy system
  • It's possible that a field that is now not-nullable was previously allowed to be null, etc.
  1. Assume there is hidden complexity
  • This is a good rule of thumb for any work.
  1. Utilize the client's knowledge about the data
  • More than likely they will know more about it and be able to answer any questions you have! Utilize their knowledge.
  1. Map old data to new data in a spreadsheet with the client ahead of the import.
  • However you go about implementing the import, we want it to be smart about how it associates old data to new.

  1. If importing and honoring the old primary keys, use them to make it idempotent even if we don’t expect to run it more than once

  2. Validate the length of the data you get from the old system

  • For example, when importing orders from Airtable for World Central Kitchen, we could see in Airtable that there were ~37,000 records.
  • When downloading the CSV of orders to import, we could validate the number of rows in the CSV to make sure the CSV contained all records.
  1. Within the import code, set inserted_at and updated_at to the historic values.
  • A record that was inserted 5 years ago should still reflect that original date.
  1. It's OK to plan for the client to have to do some post-import data cleanup.
  • When planning the import with the client, if you identify certain fields that will need to be verified/cleaned by them after the fact, and the client is OK with it, then having them perform that task is OK.
  • The client will have more context for historical data so this cleanup could be vital.
  1. When you're all ready to import -- TELL the customer when the import is happening!
  • Don't surprise them by running the import without checking with them first.

🏳️‍🌈🦄 Sharing is caring. 🦄🏳️‍🌈

Home

TODO: can we recreate the tag system here roughly with sections about certain subjects?

Clone this wiki locally