This AWS Lambda function loads [gzipped] JSON data from S3 into a Redshift table, making use of a staging table to avoid duplication.
git clone <repo>
cd <dir>
nvm install 4.3 # version supported by AWS Lambda
nvm use 4.3
npm install
cp config.example.json config.json # and edit to suit needs
npm run setup # optional, to create staging table
npm run simulate # optional, to simulate Lambda function (WARNING: accesses _real_ S3/Redshift resources)
npm run zip # package for upload to AWS
The following keys in config.json
must be customized to suit your environment:
awsAccessKeyId
: AWS access key with access to the S3 source locationawsSecretAccessKey
: AWS secret key matching the above access keyconnectionString
: A valid connection string to your Redshift destination database (e.g.,postgres://user:pass@dbhost:5439/dbname
)destinationTable
: Name of the Redshift table to load data intostagingTable
: Name of a Redshift table where this function can track in-progress work (once configuration is set, you can create this table withnode setup
)
Copyright 2015 Joey Aghion, Artsy [MIT license]