Pinster - ImageService

Design

Pinster's image handling service operates using several AWS technologies:

• Lambda
• StepFunction
• APIGateway
• S3
• Rekognition
• CloudFormation
• CloudWatch

The basic flow of the service is as follows:

1. Client POSTs image to the image uploading api

   1. The POST body is validated against the expected structure:

      {
        "data": {
          "image": "a base64 encoded image"
        }
      }
      

   2. The metadata is built form the JWT given from PinsterApi

   3. The image is saved to s3 with accompanying metadata under a key that is the md5 of the image itself.

2. S3 fires an event which kicks off a lambda that starts the step function to handle the new file
   1. The first step of the step function is to validate that the image is safe for work
      1. If the image is nsfw then the step function will call a lambda that will post to PinsterApi that a user's image upload was flagged -- Does not work yet. Just posts on SNS instead.
   2. The validated image event is passed to the generateThumbnail function which will generate thumbnails of preconfigured sizes into the s3 bucket
   3. PinsterApi is notified of the new image and accompanying thumbnails.

Testing

To run the tests, make sure you have deployed to dev and then run npm test

Deploying

Checklist:

1. Generate new keys for pinsterapi and image service
2. Set environment variables in secrets manager

Configuring your environment

Runtimes

• Ruby > 2.1 (for deploying)
• Nodejs ~8.10
  • Serverless

To get all the dependencies run npm install

Credentials

In order to deploy, you will need to have aws credentials on your local machine. In whatever aws account you have, make a user by following this guide and then save those credentials into your ./aws/credentials file under a profile of either dev or prod

Actual Deployment Instructions

Since serverless framework doesn't support programmatic changing of yaml keys with their templating engine, I wrote a script to handle this for me. It is rudimentary but effective.

To deploy, run ruby serverless.rb deploy _stage_ where _stage_ is either dev or prod

• This could fail if the bucket configured in serverless_template.yml already exists elsewhere as bucket names are global.

The downside of this is that serverless.yml cannot be saved in the project and must be generated from serverless_template.yml

Environment variables

Use Secrets Manager to configure environment variables.

Un-deploying

Don't forget to empty the image upload bucket of files before you tear it down, otherwise the removal will fail. I may add support for this to serverless.rb but it will need to be refactored first.

Backing up images

aws s3 cp --recursive s3://image-service-dev.pinster.io /dev/null --exclude "*_*" --dryrun

Purging all thumbnails

aws s3 rm --recursive s3://image-service-dev.pinster.io --exclude "*" --include "*_*" --dryrun

Credits:

• Foo Bar videos:

  • AWS Step Functions
  • S3 in AWS Lambda

• Serverless blog: Step functions with Serverless Framework

• How to resize images in AWS lambda: https://github.com/awslabs/serverless-image-resizing

• Aws labs examples