beginners_guide_aws_lambda.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "beginners_guide_aws_lambda.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "bq4VmHjPpMOR",
        "XFiO53b7vDpW",
        "PlhZmj8Sk-t0",
        "8ZvS58PAmbKQ",
        "G-UOsV5PJRpd",
        "zpOlnzyZvBOl",
        "IgJO70pIlSZL",
        "BF2mV8jyIAuG",
        "AswOTOWqGr2o",
        "K9rXgtFXKTCN",
        "GDVXFfrnVypD",
        "MNhWEc22cn4o",
        "-csiu0aLla1w",
        "YsEZcO69SSJI",
        "FZFaResjwAP_",
        "QwnrCOMtU1fO",
        "INrZ6YMcwFxp",
        "awt-CAP5wNOF",
        "RRw5rUM2f1rv"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/noahgift/awslambda/blob/master/beginners_guide_aws_lambda.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FuRVxRwNru_Y"
      },
      "source": [
        "# Beginner's Guide to Writing AWS Lambda Functions in Python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c_Id55m6Jsbu"
      },
      "source": [
        "### Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e5p96AqpSDZa"
      },
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n",
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bq4VmHjPpMOR"
      },
      "source": [
        "### Load AWS API Keys (Colab Notebook)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aWrzIk7WpRoh"
      },
      "source": [
        "1.  Put API keys in local or remote GDrive:  \n",
        "\n",
        "`cp ~/.aws/credentials /Users/myname/Google\\ Drive/awsml/`\n",
        "\n",
        "2.  Be very thoughtful about principle of \"least privilage\". Single purpose keys are great.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPWO_zyRopXN"
      },
      "source": [
        "#### Mount GDrive\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XI73HZNLobp4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        },
        "outputId": "af9b724d-7284-49c3-d3bd-0203a7215a44"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive', force_remount=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UNyzZwgmoxwm",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c305b89a-5e43-4f1f-f0af-2ab603dfd970"
      },
      "source": [
        "import os;os.listdir(\"/content/gdrive/My Drive/awsml\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['kaggle.json', 'credentials', 'config']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYu0ekUlqPk6"
      },
      "source": [
        "#### Install Boto"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJDDrUkWrYRY"
      },
      "source": [
        "!pip -q install boto3\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FpJhrpSQsK5E"
      },
      "source": [
        "#### Create API Config"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QxRwGOZtsN0-"
      },
      "source": [
        "!mkdir -p ~/.aws &&\\\n",
        "  cp /content/gdrive/My\\ Drive/awsml/credentials ~/.aws/credentials "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kj977UW3rph_"
      },
      "source": [
        "#### Test Comprehend API Call"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P-A8Cia-raT0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 240
        },
        "outputId": "198fe3eb-ea5c-4698-e73c-7038c28c67a3"
      },
      "source": [
        "import boto3\n",
        "comprehend = boto3.client(service_name='comprehend', region_name=\"us-east-1\")\n",
        "text = \"There is smoke in San Francisco and it makes me angry\"\n",
        "comprehend.detect_sentiment(Text=text, LanguageCode='en')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',\n",
              "   'content-length': '164',\n",
              "   'content-type': 'application/x-amz-json-1.1',\n",
              "   'date': 'Wed, 09 Jan 2019 03:16:52 GMT',\n",
              "   'x-amzn-requestid': '02748c54-13bd-11e9-9d75-77c414a099ee'},\n",
              "  'HTTPStatusCode': 200,\n",
              "  'RequestId': '02748c54-13bd-11e9-9d75-77c414a099ee',\n",
              "  'RetryAttempts': 0},\n",
              " u'Sentiment': u'NEGATIVE',\n",
              " u'SentimentScore': {u'Mixed': 0.010819978080689907,\n",
              "  u'Negative': 0.9212133288383484,\n",
              "  u'Neutral': 0.06721948087215424,\n",
              "  u'Positive': 0.0007472822326235473}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w7PAcvcVsTvs"
      },
      "source": [
        "## Using Cloud9 to Develop Python Lambda Functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hs0QBX6ewU5e"
      },
      "source": [
        "* [Watch AWS ML Video Lesson 7.5 Cloud9](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597/9780135556597-ACML_01_07_05)\n",
        "\n",
        "![Cloud9](https://user-images.githubusercontent.com/58792/50839709-bac19300-1315-11e9-9383-8e2e76bc9759.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1bBAwtru13k"
      },
      "source": [
        "### Developing with Cloud9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFiO53b7vDpW"
      },
      "source": [
        "#### Creating a Hello World Lambda Function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8X8RW6RpV73S"
      },
      "source": [
        "##### Creating Lambda Function from AWS console"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLzerLsolbk2"
      },
      "source": [
        "Best way to start with AWS Lambda is to kick the tires in the console\n",
        "\n",
        "![awslambda hello](https://user-images.githubusercontent.com/58792/53612654-2abaff80-3b88-11e9-99b8-3221d9487c24.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PlhZmj8Sk-t0"
      },
      "source": [
        "#### Cloud9 Setup Steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pfjYcMIqu8vh"
      },
      "source": [
        "*   [IAM User + Admin group](https://docs.aws.amazon.com/cloud9/latest/user-guide/setup-express.html)\n",
        "*   [Creating EC2 AWS Cloud9 development environment](https://docs.aws.amazon.com/cloud9/latest/user-guide/create-environment.html)\n",
        "\n",
        "![New Cloud9](https://user-images.githubusercontent.com/58792/53501005-4fb45300-3a60-11e9-9353-6d9a76ddbf53.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ZvS58PAmbKQ"
      },
      "source": [
        "#### Creating Cloud9 Environment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tBUnzsfPm1YM"
      },
      "source": [
        "\n",
        "\n",
        "![env cloud9](https://user-images.githubusercontent.com/58792/53502242-855a3b80-3a62-11e9-854a-9b2a8ebc6b24.png)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bq_oe5Gru84K"
      },
      "source": [
        "### Launching Cloud9 and Workspace Configuration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G-UOsV5PJRpd"
      },
      "source": [
        "#### Using the cloud9 terminal"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ctsLoxVYJVUH"
      },
      "source": [
        "* Terminal has access to aws cli\n",
        "\n",
        "![cli](https://user-images.githubusercontent.com/58792/53601859-efa5d580-3b61-11e9-8d5f-be5470961e8b.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zpOlnzyZvBOl"
      },
      "source": [
        "#### Configuring Workspace"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IgJO70pIlSZL"
      },
      "source": [
        "##### Getting Data Into Cloud9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NMRJkBx5DXwZ"
      },
      "source": [
        "\n",
        "\n",
        "*   Creating source code files\n",
        "*   Uploading files\n",
        "*   [Cloning from Github](https://docs.aws.amazon.com/cloud9/latest/user-guide/sample-github.html#sample-github-clone-repo)\n",
        "\n",
        "![git clone](https://user-images.githubusercontent.com/58792/53518359-b5ff9c80-3a85-11e9-91a1-0b312b6216db.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF2mV8jyIAuG"
      },
      "source": [
        "##### Setting up SSH Keys"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pd4Qnab-ITZA"
      },
      "source": [
        "* create public key [and put into github](https://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account)\n",
        "* allows you to read/write to github repo you clone\n",
        "\n",
        "\n",
        "![public key for github](https://user-images.githubusercontent.com/58792/53612747-97ce9500-3b88-11e9-80a7-c1702a669011.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AswOTOWqGr2o"
      },
      "source": [
        "##### Changing Project Settings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PpY0A_w1Gr2p"
      },
      "source": [
        "*   Changing Python support\n",
        "\n",
        "![python3](https://user-images.githubusercontent.com/58792/53511931-59e14c00-3a76-11e9-9be1-0cc033873ded.png)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9rXgtFXKTCN"
      },
      "source": [
        "##### Menu"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MgETt_DTVbrG"
      },
      "source": [
        "[Menu reference](https://docs.aws.amazon.com/cloud9/latest/user-guide/menu-commands.html)\n",
        "\n",
        "![menu](https://user-images.githubusercontent.com/58792/53515534-0b847b00-3a7f-11e9-87db-b1e7968b4bf2.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GDVXFfrnVypD"
      },
      "source": [
        "##### Collaborate, Outline, AWS Resources and Debugger"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qohwG2xqZz4N"
      },
      "source": [
        "[Outline window reference](https://docs.aws.amazon.com/cloud9/latest/user-guide/tutorial.html#tutorial-gutter)\n",
        "\n",
        "![local_functions](https://user-images.githubusercontent.com/58792/53516957-62d81a80-3a82-11e9-9f47-5d4da86982c5.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MNhWEc22cn4o"
      },
      "source": [
        "##### Share environment via IAM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jE4J8tMqczbL"
      },
      "source": [
        "[Shared environment reference](https://docs.aws.amazon.com/cloud9/latest/user-guide/share-environment.html)\n",
        "\n",
        "![Share environment](https://user-images.githubusercontent.com/58792/53517514-a8e1ae00-3a83-11e9-91b0-cfc2ee114076.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XCWmEIOUvBSG"
      },
      "source": [
        "### Creating and Deploying Lambda functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-csiu0aLla1w"
      },
      "source": [
        "#### Create a Lambda with Serverless Application Wizard"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qCG-xeH0mbqr"
      },
      "source": [
        "* [Creating Lambda with Wizard](https://docs.aws.amazon.com/cloud9/latest/user-guide/lambda-functions.html#lambda-functions-create)\n",
        "\n",
        "![wizard](https://user-images.githubusercontent.com/58792/53533804-18ba5d80-3ab1-11e9-8896-9c40b34ffe32.png)\n",
        "![blueprint](https://user-images.githubusercontent.com/58792/53533985-e3fad600-3ab1-11e9-8e1a-a7f31b0fdd1d.png)\n",
        "![apigateway](https://user-images.githubusercontent.com/58792/53533988-e826f380-3ab1-11e9-92e7-4e64726da3fa.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "48IMXOQLSXOl"
      },
      "source": [
        "#### Installing Python Packages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQx3cD5eTQyE"
      },
      "source": [
        "##### Upgrade pip\n",
        "\n",
        "1.  *Upgrade pip for Python3:*\n",
        "\n",
        "```bash\n",
        "sudo /usr/bin/python3 -m pip install --upgrade pip  \n",
        "```\n",
        "\n",
        "2.  *Use pip3*:\n",
        "\n",
        "\n",
        "```\n",
        "noah:~/environment $ pip3 --version\n",
        "pip 19.0.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-0e_pPprQts"
      },
      "source": [
        "##### Install package one level above function\n",
        "\n",
        "\n",
        "cd into function:\n",
        "\n",
        "```\n",
        "cd hellolambdacloud9app/hellolambdacloud9\n",
        "```\n",
        "\n",
        "install 3rd party package above:\n",
        "\n",
        "```\n",
        "pip install wikipedia --target ../\n",
        "```\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YsEZcO69SSJI"
      },
      "source": [
        "#### Running Local Lambda"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "51Be_ddFlTLx"
      },
      "source": [
        "* [Running a lambda local\n",
        "](https://docs.aws.amazon.com/cloud9/latest/user-guide/lambda-functions.html#lambda-functions-import)\n",
        "\n",
        "![run local](https://user-images.githubusercontent.com/58792/53529625-0be23d80-3aa2-11e9-8218-855da7a0b288.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FZFaResjwAP_"
      },
      "source": [
        "#### Invoking Lambda function inside (local & remote) API Gateway"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F62rE6HwwEO2"
      },
      "source": [
        "Response [must change slightly](https://docs.aws.amazon.com/cloud9/latest/user-guide/lambda-functions.html#lambda-functions-vs-api-gateway)\n",
        "\n",
        "\n",
        "```python\n",
        "import json\n",
        "import wikipedia\n",
        "\n",
        "print('Loading function')\n",
        "\n",
        "\n",
        "def lambda_handler(event, context):\n",
        "    \"\"\"Wikipedia Summarizer\"\"\"\n",
        "    \n",
        "    if 'body' in event:\n",
        "        event = json.loads(event[\"body\"])\n",
        "    entity = event[\"entity\"]\n",
        "    res = wikipedia.summary(entity, sentences=1)\n",
        "    print(f\"Response from wikipedia API: {res}\")\n",
        "    response = {\n",
        "    \"statusCode\": \"200\",\n",
        "    \"headers\": { \"Content-type\": \"application/json\" },\n",
        "    \"body\": json.dumps({\"message\": res})\n",
        "    }\n",
        "    return response  \n",
        "\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QwnrCOMtU1fO"
      },
      "source": [
        "#### Deploy from Cloud9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUR-s6fKVLpx"
      },
      "source": [
        "1.  ***Right click on local lambda to deploy***\n",
        "\n",
        "![deploy local lambda](https://user-images.githubusercontent.com/58792/53588665-54e9ce80-3b42-11e9-9771-1f3875e18995.png)\n",
        "\n",
        "2.  ***Test deploy by invoking remote API gateway***\n",
        "\n",
        "![remote api gateway](https://user-images.githubusercontent.com/58792/53588952-f8d37a00-3b42-11e9-9942-7bfb735e02b9.png)\n",
        "\n",
        "3.  ***Test remote lambda***\n",
        "\n",
        "![remote lambda](https://user-images.githubusercontent.com/58792/53589059-3e904280-3b43-11e9-8a3a-8d8edb32d77f.png)\n",
        "\n",
        "4. *** Inspect and test in in AWS Console:  https://console.aws.amazon.com/lambda/home?region=us-east-1#/applications/cloud9-hellolambdacloud9app***\n",
        "\n",
        "![test_event](https://user-images.githubusercontent.com/58792/53589401-1fde7b80-3b44-11e9-944e-ee3490e9d1b6.png)\n",
        "\n",
        "![console](https://user-images.githubusercontent.com/58792/53589403-1fde7b80-3b44-11e9-94d9-ad53edc48d24.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W9irbK4UvDvq"
      },
      "source": [
        "#### Importing (Remote) Lambda functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yj7sJxBvGB7"
      },
      "source": [
        "* Import Hello World Lambda Created earlier\n",
        "* [Importing a Lambda function](https://docs.aws.amazon.com/cloud9/latest/user-guide/lambda-functions.html#lambda-functions-import)\n",
        "\n",
        "![importing lambda](https://user-images.githubusercontent.com/58792/53524070-56a88900-3a93-11e9-9c24-7158cbe59abb.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-QGxz75FdAt"
      },
      "source": [
        "#### Case Study:  Making an API and Deploying it"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6bwPtWuFokJ"
      },
      "source": [
        "\n",
        "**Python Lambda Function**\n",
        "\n",
        "```python\n",
        "import json\n",
        "import decimal\n",
        "\n",
        "\n",
        "def lambda_handler(event, context):\n",
        "\n",
        "  print(event)\n",
        "  if 'body' in event:\n",
        "    event = json.loads(event[\"body\"])\n",
        "  \n",
        "  amount = float(event[\"amount\"])\n",
        "  res = []\n",
        "  coins = [1,5,10,25]\n",
        "  coin_lookup = {25: \"quarters\", 10: \"dimes\", 5: \"nickels\", 1: \"pennies\"}\n",
        "  coin = coins.pop()\n",
        "  num, rem  = divmod(int(amount*100), coin)\n",
        "  res.append({num:coin_lookup[coin]})\n",
        "  while rem > 0:\n",
        "    coin = coins.pop()\n",
        "    num, rem = divmod(rem, coin)\n",
        "    if num:\n",
        "      if coin in coin_lookup:\n",
        "        res.append({num:coin_lookup[coin]})\n",
        "\n",
        "  response = {\n",
        "    \"statusCode\": \"200\",\n",
        "    \"headers\": { \"Content-type\": \"application/json\" },\n",
        "    \"body\": json.dumps({\"res\": res})\n",
        "  }\n",
        "\n",
        "  return response\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKXgIaJzsTzC"
      },
      "source": [
        "## Creating Timed Lambdas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CQ45bnlfN14W"
      },
      "source": [
        "Creating Serverless Data Pipeline Producers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "INrZ6YMcwFxp"
      },
      "source": [
        "### Using AWS Lambda with Cloudwatch Events"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fo7rh-wkwJGO"
      },
      "source": [
        "Can create [cloudwatch timer](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html) to call lambda\n",
        "\n",
        "![cloudwatch event lambda](https://user-images.githubusercontent.com/58792/53612460-4c67b700-3b87-11e9-8fb9-b5d30b77431a.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "awt-CAP5wNOF"
      },
      "source": [
        "### Using AWS Cloudwatch logging with AWS Lambda"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYCfPjkGwNRB"
      },
      "source": [
        "Using cloudwatch logging is an essential step for Lambda Development\n",
        "\n",
        "![cloudwatch](https://user-images.githubusercontent.com/58792/53612528-9355ac80-3b87-11e9-8473-ab28ba860553.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6fl3b_w6wJJM"
      },
      "source": [
        "### Using AWS Lambda to populate AWS SQS (Simple Queuing Service)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tOqVcZTbwLZg"
      },
      "source": [
        "1. *** Create new Lambda with Serverless Wizard***\n",
        "2.  ***cd into lambda and install packages on level up***\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "pip3 install boto3 --target ../\n",
        "pip3 install python-json-logger --target ../\n",
        "```\n",
        "\n",
        "3.  ***Test local***\n",
        "4. *** Deploy***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Xe5mNrQIQcJ"
      },
      "source": [
        "\n",
        "\n",
        "```python\n",
        "\"\"\"\n",
        "Dynamo to SQS\n",
        "\"\"\"\n",
        "\n",
        "import boto3\n",
        "import json\n",
        "import sys\n",
        "import os\n",
        "\n",
        "DYNAMODB = boto3.resource('dynamodb')\n",
        "TABLE = \"fang\"\n",
        "QUEUE = \"producer\"\n",
        "SQS = boto3.client(\"sqs\")\n",
        "\n",
        "#SETUP LOGGING\n",
        "import logging\n",
        "from pythonjsonlogger import jsonlogger\n",
        "\n",
        "LOG = logging.getLogger()\n",
        "LOG.setLevel(logging.INFO)\n",
        "logHandler = logging.StreamHandler()\n",
        "formatter = jsonlogger.JsonFormatter()\n",
        "logHandler.setFormatter(formatter)\n",
        "LOG.addHandler(logHandler)\n",
        "\n",
        "def scan_table(table):\n",
        "    \"\"\"Scans table and return results\"\"\"\n",
        "    \n",
        "    LOG.info(f\"Scanning Table {table}\")\n",
        "    producer_table = DYNAMODB.Table(table)\n",
        "    response = producer_table.scan()\n",
        "    items = response['Items']\n",
        "    LOG.info(f\"Found {len(items)} Items\")\n",
        "    return items\n",
        "\n",
        "def send_sqs_msg(msg, queue_name, delay=0):\n",
        "    \"\"\"Send SQS Message\n",
        "\n",
        "    Expects an SQS queue_name and msg in a dictionary format.\n",
        "    Returns a response dictionary. \n",
        "    \"\"\"\n",
        "\n",
        "    queue_url = SQS.get_queue_url(QueueName=queue_name)[\"QueueUrl\"]\n",
        "    queue_send_log_msg = \"Send message to queue url: %s, with body: %s\" %\\\n",
        "        (queue_url, msg)\n",
        "    LOG.info(queue_send_log_msg)\n",
        "    json_msg = json.dumps(msg)\n",
        "    response = SQS.send_message(\n",
        "        QueueUrl=queue_url,\n",
        "        MessageBody=json_msg,\n",
        "        DelaySeconds=delay)\n",
        "    queue_send_log_msg_resp = \"Message Response: %s for queue url: %s\" %\\\n",
        "        (response, queue_url) \n",
        "    LOG.info(queue_send_log_msg_resp)\n",
        "    return response\n",
        "\n",
        "def send_emissions(table, queue_name):\n",
        "    \"\"\"Send Emissions\"\"\"\n",
        "    \n",
        "    items = scan_table(table=table)\n",
        "    for item in items:\n",
        "        LOG.info(f\"Sending item {item} to queue: {queue_name}\")\n",
        "        response = send_sqs_msg(item, queue_name=queue_name)\n",
        "        LOG.debug(response)\n",
        "\n",
        "def lambda_handler(event, context):\n",
        "    \"\"\"\n",
        "    Lambda entrypoint\n",
        "    \"\"\"\n",
        "\n",
        "    extra_logging = {\"table\": TABLE, \"queue\": QUEUE}\n",
        "    LOG.info(f\"event {event}, context {context}\", extra=extra_logging)\n",
        "    send_emissions(table=TABLE, queue_name=QUEUE)\n",
        "\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PdB-FUHUJ84S"
      },
      "source": [
        "**Successful Local Test**\n",
        "\n",
        "![test local](https://user-images.githubusercontent.com/58792/53637263-8bbdf400-3bd7-11e9-9840-0cb9851fac6a.png)\n",
        "\n",
        "**Verify Messages in SQS**\n",
        "\n",
        "![**SQS**](https://user-images.githubusercontent.com/58792/53637424-fb33e380-3bd7-11e9-8b68-021704da4ce0.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vAStK9C0NDd4"
      },
      "source": [
        "***Remote Test Needs Correct Role!!!***\n",
        "\n",
        "![role failure](https://user-images.githubusercontent.com/58792/53638025-c45ecd00-3bd9-11e9-848c-6caedc3d9011.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jyZaUa-NOWYr"
      },
      "source": [
        "#### Wire up Cloudwatch Event Trigger"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jf353R-xOcBG"
      },
      "source": [
        "1.  Enable Timed Execution of producer\n",
        "2.  Verify messages flowing into SQS\n",
        "\n",
        "![cloudwatch event trigger](https://user-images.githubusercontent.com/58792/53638200-6979a580-3bda-11e9-94ea-9008bdc9c72a.png)\n",
        "\n",
        "***SQS is populating***\n",
        "\n",
        "![alt text](https://user-images.githubusercontent.com/58792/53638351-cecd9680-3bda-11e9-85bb-f5f4bd4450ad.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_iIu63uesj5R"
      },
      "source": [
        "## Creating Event Driven Lambdas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DHqZDYOmsj8G"
      },
      "source": [
        "### Triggering AWS Lambda with AWS SQS Events"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nohx7lBy2khs"
      },
      "source": [
        "Lambda can now fire on SQS event\n",
        "\n",
        "![SQS Trigger](https://user-images.githubusercontent.com/58792/53644659-f842ee00-3beb-11e9-8527-96ec12acc5f7.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EzwngIdr2kk2"
      },
      "source": [
        "### Reading AWS SQS Events from AWS Lambda"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8hvf6Vc62ns3"
      },
      "source": [
        "\n",
        "\n",
        "```python\n",
        "def lambda_handler(event, context):\n",
        "    \"\"\"Entry Point for Lambda\"\"\"\n",
        "\n",
        "    LOG.info(f\"SURVEYJOB LAMBDA, event {event}, context {context}\")\n",
        "    receipt_handle  = event['Records'][0]['receiptHandle'] #sqs message\n",
        "    #'eventSourceARN': 'arn:aws:sqs:us-east-1:561744971673:producer'\n",
        "    event_source_arn = event['Records'][0]['eventSourceARN']\n",
        "    \n",
        "    names = [] #Captured from Queue\n",
        "    \n",
        "    # Process Queue\n",
        "    for record in event['Records']:\n",
        "        body = json.loads(record['body'])\n",
        "        company_name = body['name']\n",
        "        \n",
        "        #Capture for processing\n",
        "        names.append(company_name)\n",
        "        \n",
        "        extra_logging = {\"body\": body, \"company_name\":company_name}\n",
        "        LOG.info(f\"SQS CONSUMER LAMBDA, splitting sqs arn with value: {event_source_arn}\",extra=extra_logging)\n",
        "        qname = event_source_arn.split(\":\")[-1]\n",
        "        extra_logging[\"queue\"] = qname\n",
        "        LOG.info(f\"Attemping Deleting SQS receiptHandle {receipt_handle} with queue_name {qname}\", extra=extra_logging)\n",
        "        res = delete_sqs_msg(queue_name=qname, receipt_handle=receipt_handle)\n",
        "        LOG.info(f\"Deleted SQS receipt_handle {receipt_handle} with res {res}\", extra=extra_logging)\n",
        "    \n",
        "    # Make Pandas dataframe with wikipedia snippts\n",
        "    LOG.info(f\"Creating dataframe with values: {names}\")\n",
        "    df = names_to_wikipedia(names)\n",
        "    \n",
        "    # Perform Sentiment Analysis\n",
        "    df = apply_sentiment(df)\n",
        "    LOG.info(f\"Sentiment from FANG companies: {df.to_dict()}\")\n",
        "    \n",
        "    # Write result to S3\n",
        "    write_s3(df=df, name=names.pop(), bucket=\"fangsentiment\")\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OJLaB5KxWnU0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "829ec0d0-3bd8-4482-856e-f635e84bb702"
      },
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv(\"facebook_sentiment.csv\")\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>names</th>\n",
              "      <th>wikipedia_snippit</th>\n",
              "      <th>Sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>facebook</td>\n",
              "      <td>Facebook, Inc. is an American online social me...</td>\n",
              "      <td>NEUTRAL</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0     names                                  wikipedia_snippit  \\\n",
              "0           0  facebook  Facebook, Inc. is an American online social me...   \n",
              "\n",
              "  Sentiment  \n",
              "0   NEUTRAL  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ckRibEoh2n0q"
      },
      "source": [
        "### Writing results to AWS S3"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JDn4JdWsqrwy"
      },
      "source": [
        "write dataframe to AWS S3\n",
        "\n",
        "```python\n",
        "### S3\n",
        "def write_s3(df, name, bucket):\n",
        "    \"\"\"Write S3 Bucket\"\"\"\n",
        "\n",
        "    csv_buffer = StringIO()\n",
        "    df.to_csv(csv_buffer)\n",
        "    s3_resource = boto3.resource('s3')\n",
        "    res = s3_resource.Object(bucket, f'{name}_sentiment.csv').\\\n",
        "        put(Body=csv_buffer.getvalue())\n",
        "    LOG.info(f\"result of write name: {name} to bucket: {bucket} with:\\n {res}\")\n",
        "\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpIE4sgc2tZi"
      },
      "source": [
        "\n",
        "\n",
        "```\n",
        "noah:/tmp $ aws s3 cp --recursive s3://fangsentiment/ .                                                                                                \n",
        "download: s3://fangsentiment/netflix_sentiment.csv to ./netflix_sentiment.csv\n",
        "download: s3://fangsentiment/google_sentiment.csv to ./google_sentiment.csv\n",
        "download: s3://fangsentiment/facebook_sentiment.csv to ./facebook_sentiment.csv\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G4bMR3j7Z4oD"
      },
      "source": [
        "## Serverless AI Data Engineering Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVUQyxg7Z-FU"
      },
      "source": [
        "![architecture](https://user-images.githubusercontent.com/58792/55354483-bae7af80-547a-11e9-9909-a5621251065b.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pRZ65E8ffePo"
      },
      "source": [
        "## Homework"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oo727-tzfg-R"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRw5rUM2f1rv"
      },
      "source": [
        "## FAQ"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WC3z1ogqf92H"
      },
      "source": [
        ""
      ]
    }
  ]
}