azure/hdinsight/hdinsight-private-subnet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy and access Azure HDInsight in a private subnet\n",
    "\n",
    "In order to better isolate access to Azure HDInsight clusters from the public Internet and enhance the security at the networking layer, enterprises can deploy the clusters in a private subnet in a virtual network, and access them through a private endpoint.\n",
    "\n",
    "Each HDInsight cluster deployed in a virtual network has a private endpoint in the form of **https://CLUSTERNAME-int.azurehdinsight.net** as well as a public endpoint. Note the **“-int”** in this URL, this endpoint will resolve to a private IP in that virtual network and is not accessible from the public Internet.\n",
    "\n",
    "\n",
    "This lab walks you through the steps to:\n",
    "* Deploy an HDInsight cluster in a private subnet;\n",
    "* Configure the netwok security groups (NSG) that are necessary to deny access from the public Internet through the public endpoint;\n",
    "* Use an SSH tunnel with dynamic port forwarding to access the administrative web interfaces provided by the HDInsight cluster (eg. Ambari or the Spark UI).\n",
    "\n",
    "The following diagram illustrates the architecture that we will be setting up.\n",
    "\n",
    "![Architecture Diagram](images/architecture.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a resource group and a virtual network\n",
    "\n",
    "For convenience, we will be using the Azure CLI to setup our virtual networlk.\n",
    "\n",
    "[Install and configure the Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) on your machine, then log in to your Azure subscription entering the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you can create a virtual network, you have to create a resource group to host the virtual network (of course  you can reuse an existing one if you already have it - in this case you can skip the step below):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az group create --name labs-rg --location southeastasia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a virtual network with a private subnet where to deploy the HDInsight cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az network vnet create \\\n",
    "    --name dev-test-vnet \\\n",
    "    --resource-group labs-rg \\\n",
    "    --location southeastasia \\\n",
    "    --address-prefix 10.0.0.0/16 \\\n",
    "    --subnet-name hdinsight-subnet \\\n",
    "    --subnet-prefix 10.0.0.0/24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a public subnet within the virtual network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az network vnet subnet create \\\n",
    "    --name public-subnet \\\n",
    "    --vnet-name dev-test-vnet \\\n",
    "    --resource-group labs-rg \\\n",
    "    --address-prefix 10.0.128.0/20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an SSH key pair \n",
    "\n",
    "We will be using Public-key cryptography to authenticate SSH sessions, in order to avoid passwords and to secure access to both the HDInsight cluster and the bastion host (or jumpbox) that we deploy in the public subnet.\n",
    "\n",
    "Use the `ssh-keygen` command to create public and private key files. The following command generates a 2048-bit RSA key pair:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "ssh-keygen -t rsa -b 2048 -f keys/azure-ssh-keypair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the following command to set the permissions of your private key file so that only you can read it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "chmod 400 keys/azure-ssh-keypair*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the jumpbox server\n",
    "\n",
    "Follow [these instructions](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal) to create a Linux VM that will serve as a jumpbox.\n",
    "\n",
    "Just ensure that you:\n",
    "* Enable SSH authentication by providing the public key you have previously created;\n",
    "* Enable only SSH inbound connections from the Internet, selecting the SSH port (22) in the **Inbound port rules** section;\n",
    "* Deploy the jumpbox in the public subnet.\n",
    "\n",
    "In alternative, you can execute the script below to create the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az vm create \\\n",
    "    --resource-group labs-rg \\\n",
    "    --name jumpbox \\\n",
    "    --image CentOS \\\n",
    "    --location southeastasia \\\n",
    "    --public-ip-address-dns-name <JUMPBOX_DNS_NAME> \\\n",
    "    --vnet-name dev-test-vnet \\\n",
    "    --subnet public-subnet \\\n",
    "    --admin-username <ADMIN_USERNAME> \\\n",
    "    --ssh-key-value \"$(cat keys/azure-ssh-keypair.pub)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the HDInsight cluster\n",
    "\n",
    "Follow [these instructions](https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql-use-portal) to create an Apache Spark cluster in HDInsight using the Azure portal.\n",
    "\n",
    "While configuring the cluster, make sure that you enable SSH authentication by providing the public key you have create at the previous step:\n",
    "\n",
    "![Enable SSH authentication with public key](images/screenshot-01.png)\n",
    "\n",
    "Moreover, ensure that you deploy the cluster in the private subnet you have created, as exemplified in the following screenshot:\n",
    "\n",
    "![Deployu the cluster in the private subnet](images/screenshot-02.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and configure Network Security Groups (NSG)\n",
    "\n",
    "Network security groups (NSG) allow you to filter inbound and outbound traffic to the network.\n",
    "\n",
    "We need to create a NSG to secure access to the HDInsight cluster, and configure it with a set of rules that:\n",
    "* Allow traffic from the Azure health and management services to reach HDInsight clusters on port 443;\n",
    "* Allow traffic between VMs inside the virtual network we have setup;\n",
    "* Deny everything else.\n",
    "\n",
    "The IP addresses of the management services to allow in tbhe Southeast Asia region are the following (for other regions, you can consult this [documentation link](https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-extend-hadoop-virtual-network#hdinsight-ip-1)):\n",
    "\n",
    "| Source IP address | Destination port | Direction |\n",
    "|--------------|---|-------|\n",
    "|168.61.49.99  |443|Inbound|\n",
    "|23.99.5.239   |443|Inbound|\n",
    "|168.61.48.131 |443|Inbound|\n",
    "|138.91.141.162|443|Inbound|\n",
    "|13.76.245.160 |443|Inbound|\n",
    "|13.76.136.249 |443|Inbound|\n",
    "|168.63.129.16 |443|Inbound|\n",
    "\n",
    "The following commands in Azure CLI helps you to setup the NSG (alternatively you could configure the same NSG through the [Azure Portal](https://portal.azure.com)) and associate it to the private subnet where the HDInsight cluster is deployed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az network nsg create --resource-group labs-rg --name hdinsight-nsg --location southeastasia "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az network nsg rule create \\\n",
    "    --name AllowAzureMgmtServicesInBound \\\n",
    "    --priority 100 \\\n",
    "    --nsg-name hdinsight-nsg \\\n",
    "    --resource-group labs-rg \\\n",
    "    --access Allow \\\n",
    "    --source-address-prefixes 168.61.49.99 23.99.5.239 168.61.48.131 138.91.141.162 13.76.245.160 13.76.136.249 168.63.129.16 \\\n",
    "    --source-port-ranges 443 \\\n",
    "    --destination-address-prefixes '*' \\\n",
    "    --destination-port-ranges 443 \\\n",
    "    --protocol Tcp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "az network vnet subnet update \\\n",
    "    --resource-group labs-rg \\\n",
    "    --vnet-name dev-test-vnet \\\n",
    "    --name hdinsight-subnet \\\n",
    "    --network-security-group hdinsight-nsg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup an SSH tunnel with dynamic port forwarding\n",
    "\n",
    "You can setup and use an SSH tunnel with dynamic port forwarding to access the administrative web interfaces provided by the HDInsight cluster (eg. Ambari or the Spark UI)\n",
    "\n",
    "If you use an SSH tunnel with dynamic port forwarding, you must use a SOCKS proxy management add-on to control the proxy settings in your browser. Using a SOCKS proxy management tool allows you to automatically filter URLs based on text patterns and to limit the proxy settings to domains that match the form of the master node's public DNS name. The browser add-on automatically handles turning the proxy on and off when you switch between viewing websites hosted on the master node and those on the Internet. To manage your proxy settings, configure your browser to use an add-on such as FoxyProxy (with Google Chrome) or SwitchyOmega (with Firefox).\n",
    "\n",
    "The following example demonstrates a [SwitchyOmega](https://addons.mozilla.org/en-US/firefox/addon/switchyomega/) configuration using Firefox.\n",
    "\n",
    "Set the proxy server to localhost with the port set to 8157 (you should set this value to the local port number that you will use to establish the SSH tunnel).\n",
    "\n",
    "![Setup the SOCKS proxy](images/screenshot-03.png)\n",
    "\n",
    "Whitelist the following URL patterns (as illustrated by the screenshot below):\n",
    "\n",
    "* The **\\*-int.azurehdinsight.net\\*** matches the DNS name of the private endpoint of the HDInsight cluster;\n",
    "\n",
    "* The **10.0.\\*** pattern provides access to the private IP addresses of the compute resources deployed within the virtual network you have setup, including the HDInsight cluster. Alter this filter if it conflicts with your network access plan.\n",
    "\n",
    "![Whitelist URL patterns](images/screenshot-04.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the private SSH key you have created to the authentication agent, with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "ssh-add -k <PRIVATE_SSH_KEY>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Connect to the jumpbox server, creating the SSH tunnel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "ssh -A -D 8157 <ADMIN_USERNAME>@<IP_ADDRESS_OR_DNS_NAME_OF_JUMPBOX_SERVER>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open your browser and connect to the HDInsight private endpoint (**https://CLUSTERNAME-int.azurehdinsight.net**).\n",
    "\n",
    "If you have configured everything properly, the browser should be route the requests through the proxy server and you should be able to access the HDInsight cluster administrative itnerfaces as illustrated by the following screenshot.\n",
    "\n",
    "![HDInsight cluster web UI](images/screenshot-05.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}