06_deploy_and_test_logic_apps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Deploy and Test Logic Apps\n",
    "Now that we've verified that our Docker image works, this notebook will walk through how to operationalize this workflow using [Logic Apps](https://azure.microsoft.com/en-us/services/logic-apps/). \n",
    "\n",
    "For this workflow, adding a file whose name begins with 'trigger' will kick-off the entire batch style transfer process. In this architecture, we use Logic App as our trigger mechanism: when a trigger file (ex. 'trigger.txt') is uploaded into blob storage, Logic App will detect that the file has been added, and create an [Azure Container Instance (ACI)](https://azure.microsoft.com/en-us/services/container-instances/) to start creating Batch AI jobs.\n",
    "\n",
    "This notebook will rely on the file __template.logic_app.json__ to deploy the Logic App. This file is a template for a [Azure Resource Manager (ARM)](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview) json file, which we will use in this notebook to deploy the resources. In its current state, this file does not contain any of the parameters, such as those stored in the .env file, that we've set throughout the tutorial. To fill out those parameters, we will use [jinja](http://jinja.pocoo.org/) to populate the template with all the appropriate parameters and write the output to __logic_app.json__. This will be ARM json file that we use with the __az cli__ to deploy the Logic App.\n",
    "\n",
    "As part of the Logic App deployment, we'll also deploy 2 additional resources: an Azure blob storage connector and an ACI connector. These connectors will allow Logic App to talk to these two resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import utilities and load environment variables. In this notebook, we'll be using jinja2 to populate our Logic App deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import jinja2\n",
    "%load_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy Logic App"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set variables we'll need for Logic App deployment:\n",
    "\n",
    "- `logic_app` - the name of your logic app\n",
    "- `aci_container_group` - the name of your ACI \n",
    "- `aci_display_name` - a display name used for your ACI deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logic_app = \"<my-logic-app>\"\n",
    "aci_container_group = \"<my-aci-group>\"\n",
    "aci_display_name = \"<your@email.com>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Persist these variables to our `.env` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!dotenv set LOGIC_APP $logic_app\n",
    "!dotenv set ACI_CONTAINER_GROUP $aci_container_group\n",
    "!dotenv set ACI_DISPLAY_NAME $aci_display_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload our environment variables from our `.env` file so we can use these variables as enviroment variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Jinja, populate the `template.logic_app.json` file and output the new file as `logic_app.json`. This file will be saved in the working directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use jinja to fill in variables from .env file\n",
    "env = jinja2.Environment(\n",
    "    loader=jinja2.FileSystemLoader('.')\n",
    ")\n",
    "template = env.get_template('template.logic_app.json')\n",
    "\n",
    "e = os.environ\n",
    "rendered_template = template.render(env=e)\n",
    "\n",
    "out = open('logic_app.json', 'w')\n",
    "out.write(rendered_template)\n",
    "out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the `logic_app.json` file, we can use the __az cli__ to deploy the resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash \n",
    "az group deployment create \\\n",
    "    --name $LOGIC_APP \\\n",
    "    --resource-group $RESOURCE_GROUP \\\n",
    "    --template-file logic_app.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the Logic App is deployed, go into the Azure portal and open up the ACI connector and the Azure blob connector to authenticate. \n",
    "\n",
    "When you open up up the Azure ACI connector, it should look like this:\n",
    "\n",
    "![azure_aci_connector_auth](https://happypathspublic.blob.core.windows.net/assets/batch_scoring_for_dl/azure_aci_connector_auth.PNG)\n",
    "\n",
    "When you open up up the Azure blob connector, it should look like this:\n",
    "\n",
    "![azure_blob_connector_auth](https://happypathspublic.blob.core.windows.net/assets/batch_scoring_for_dl/azure_blob_connector_auth.PNG)\n",
    "\n",
    "For both of these connectors, click on the orange bar at the top to authenticate.\n",
    "\n",
    "Once authenticated, your Logic App should be all set up and ready to trigger the workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trigger Logic App\n",
    "Logic App will be triggered whenever a blob that begins with the string 'trigger' is uploaded into the Azure blob storage account. Logic App will then read the contents of the file, using the contents as the directory of individual frames to apply style transfer to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll create a file titled `trigger_0.txt` and input the name of the content directory in blob to apply style transfer to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! touch trigger_0.txt && echo $FS_CONTENT_DIR > trigger_0.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using AzCopy, upload the trigger file into blob to trigger Logic App."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "azcopy \\\n",
    "    --source trigger_0.txt \\\n",
    "    --destination https://${STORAGE_ACCOUNT_NAME}.blob.core.windows.net/${AZURE_CONTAINER_NAME}/trigger_0.txt \\\n",
    "    --dest-key $STORAGE_ACCOUNT_KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file will trigger Logic App to create an ACI that will kick off the Batch AI jobs. We can now inspect the logs from ACI to see that everything is going smoothly. If you run this command right after uploading the trigger file, the logs may not be found as it will take some time to trigger Logic App to create the ACI group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!az container logs --resource-group $RESOURCE_GROUP --name $ACI_CONTAINER_GROUP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we can inspect the Azure portal to see all the moving parts:\n",
    "- Logic Apps will be triggers and will spin up ACI\n",
    "- ACI will break up the content images in blob and create BatchAI jobs\n",
    "- The BatchAI cluster will scale up and start processing the work\n",
    "- As the style transfer script is executed in batch on BatchAI, we will see the completed images (as well as logs) saved back to blob "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this notebook, we deployed Logic App and simulated what this end-to-end workflow could look like when operationalized with an add-to-blob trigger. At this point, I'd encourage you to take a look through the portal and examine the individual components of the architecture and make sure that all the parts are operating as expected. For further customization, you can also edit the Logic App via the Logic App Designer (in the portal) so that the triggering mechanism is more fitted to your scenarios.\n",
    "\n",
    "The last step of the tutorial is to [download and re-stitch the individual frames so that we have a single style transferred video.](07_stitch_together_the_results.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:batchscoringdl]",
   "language": "python",
   "name": "conda-env-batchscoringdl-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}