Secure 'Serverless' File Uploads with AWS Lambda, S3, and Zappa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Secure 'Serverless' File Uploads with AWS Lambda, S3, and Zappa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Serverless File Upload Architecture Diagram](\t\n",
    "https://s3.amazonaws.com/stratospark/images/serverless_file_upload.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As I've been experimenting with [AWS Lambda](https://aws.amazon.com/lambda/), I've found the need to accept file uploads from the browser in order to kick off asynchronous Lambda functions. For example, allowing a user to directly upload in an S3 bucket from the browser, which would trigger a Lambda function for image processing.\n",
    "\n",
    "I decided to use the [Zappa](https://github.com/Miserlou/Zappa) framework, as it allows me to leverage my existing Python WSGI experience, while also providing a number of **awesome** features such as:\n",
    "\n",
    "* Access to powerful, prebuilt Python packages such as Numpy and scikit-learn\n",
    "* Automatic Let's Encrypt SSL registration and renewal\n",
    "* Automatic scheduled job to keep the Lambda function warm\n",
    "* Ability to invoke arbitrary Python functions within the Lambda execution environment (great for debugging)\n",
    "* Deploy bundles larger than 50 megs through a Slim Handler mechanism"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This walkthrough will cover deploying an SSL-encrypted S3 signature microservice and integrating it with the browser-based [Fine Uploader](http://fineuploader.com/) component. In an upcoming post, I will show how to take the file uploads and process them with an additional Lambda function triggered by new files in an S3 bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy Zappa Lambda Function for Signing S3 Requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the steps I took to create a secure file upload system in the cloud:\n",
    "\n",
    "* [Sign up for a domain using Namecheap](https://ap.www.namecheap.com/Profile/Tools/Affiliate)\n",
    "* Follow [these instructions](https://github.com/Miserlou/Zappa/blob/master/docs/domain_with_free_ssl_dns.md) to create a Route 53 Hosted Zone, update your domain DNS, and generate a Let's Encrypt account.key\n",
    "* Create S3 bucket to hold uploaded files, with the policy below. **Note: do not use periods in the bucket name if you want to be able to use SSL, as [explained here](http://stackoverflow.com/questions/39396634/fine-uploader-upload-to-s3-over-https-error)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{\n",
    "    \"Version\": \"2008-10-17\",\n",
    "    \"Id\": \"policy\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"allow-public-put\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"AWS\": \"*\"\n",
    "            },\n",
    "            \"Action\": \"s3:PutObject\",\n",
    "            \"Resource\": \"arn:aws:s3:::BUCKET_NAME_HERE/*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Activate CORS for the S3 bucket. You may want to update the AllowedOrigin tag to limit the domains you are allowed to upload from."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
    "<CORSConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">\n",
    "    <CORSRule>\n",
    "        <AllowedOrigin>*</AllowedOrigin>\n",
    "        <AllowedMethod>POST</AllowedMethod>\n",
    "        <AllowedMethod>PUT</AllowedMethod>\n",
    "        <AllowedMethod>DELETE</AllowedMethod>\n",
    "        <MaxAgeSeconds>3000</MaxAgeSeconds>\n",
    "        <ExposeHeader>ETag</ExposeHeader>\n",
    "        <AllowedHeader>*</AllowedHeader>\n",
    "    </CORSRule>\n",
    "</CORSConfiguration>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Optionally update the Lifecycle Rules for that bucket to automatically delete files after a certain period of time.\n",
    "* Create a new IAM user specifically to create a new set of keys with limited permissions for your Lambda function:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"Stmt1486786154000\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"s3:PutObject\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::BUCKET_NAME_HERE/*\"\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Clone this Zappa project: `git clone https://github.com/stratospark/zappa-s3-signature`\n",
    "* Create a virtual environment for this project: `virtualenv myenv`. *Note, conda environments are currently unsupported, so I utilize a Docker container with a standard Python virtualenv*\n",
    "* Install packages: `pip install -r requirements.txt`. \n",
    "* Create an `s3-signature-config.json` file with the ACCESS_KEY and SECRET_KEY of the new User you created, for example:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{\n",
    "    \"ACCESS_KEY\": \"AKIAIHBBHGQSUN34COPA\",\n",
    "    \"SECRET_KEY\": \"wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Upload `s3-signature-config.json` to an S3 bucket accessible by the Lambda function, used in **remote_env** config field\n",
    "* Update the *prod* section of `zappa_settings.json` with your **aws_region**, **s3_bucket**, **cors/allowed_origin**, **remote_env**, **domain**, and **lets_encrypt_key**\n",
    "* Deploy to AWS Lambda: `zappa deploy prod`\n",
    "* Enable SSL through Let's Encrypt: `zappa certify prod`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy HTML5/Javascript Fine Uploader Page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UI](https://s3.amazonaws.com/stratospark/images/serverless_upload_ui.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following steps will allow you to host a static page that contains the Fine Uploader component. This is a very full-featured open-source component that has excellent S3 support. It also comes with pre-built UI components such as an Image Gallery, to help save time when developing prototypes.\n",
    "\n",
    "We have deployed the AWS V4 Signature Lambda function in the previous section in order to take advantage of direct Browser -> S3 uploads.\n",
    "\n",
    "You can deploy the HTML and Javascript files onto any server you have access to. However, as we have an opportunity to piggyback on existing AWS infrastructure, including SSL, we can just deploy a static site on S3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Clone the sample React Fine Uploader project: https://github.com/stratospark/react-fineuploader-s3-demo\n",
    "* Update the **request/endpoint**, **request/accessKey**, and **signature/endpoint** fields in the FineUploaderS3 constructor in App.js. Optionally update **objectProperties/region**. \n",
    "    * For example, request/endpoint should be: `https://BUCKET_NAME.s3.amazonaws.com` ...\n",
    "    **Note: the endpoints must not have trailing slashes or the signatures will not be valid!**\n",
    "* Run: ``npm build``. **Note: you need to add a `homepage` field to `package.json` if you will serve the pages at a location other than the root.**\n",
    "* Create S3 bucket and upload the contents of the build folder. **Note: once again, do not use periods in the name if you want to use HTTPS/SSL**\n",
    "* Make this S3 bucket a publically available static site. Also remember to set a policy like below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{\n",
    "    \"Version\": \"2008-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"PublicReadForGetBucketObjects\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"AWS\": \"*\"\n",
    "            },\n",
    "            \"Action\": \"s3:GetObject\",\n",
    "            \"Resource\": \"arn:aws:s3:::BUCKET_NAME/*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Access the Fine Uploader demo page in your browser, for example: https://stratospark-serverless-uploader.s3.amazonaws.com/index.html\n",
    "* Upload a file\n",
    "* Check your public uploads bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all!\n",
    "\n",
    "**Stay tuned for the next installment, where we take these uploaded files and run them through image processing, computer vision, and deep learning Lambda pipelines!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}