GALA_Coding_Club_15.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xENUtv6_G90f"
   },
   "source": [
    "# Google Colab\n",
    "As I promised, I figured out the file import protocols for **colab.research.google.com**.\n",
    "\n",
    "### NOTE This Link Only Works Using Colab Jupyter Notebooks:\n",
    "\n",
    "[Open in Colab](https://chrome.google.com/webstore/detail/open-in-colab/) is a Chrome extension that will allow you to open notebooks from GitHub, then save them to your google drive.  \n",
    "\n",
    "## Saving Notebooks To GitHub or Drive\n",
    "\n",
    "Any time you open a GitHub hosted notebook in Colab, it opens a new editable view of the notebook. You can run and modify the notebook without worrying about overwriting the source.\n",
    "\n",
    "If you would like to save your changes from within Colab, you can use the File menu to save the modified notebook either to Google Drive or back to GitHub. Choose **File→Save a copy in Drive** or **File→Save a copy to GitHub** and follow the resulting prompts. To save a Colab notebook to GitHub requires giving Colab permission to push the commit to your repository.\n",
    "\n",
    "After you have installed ***Open in Colab*** let's test it with this week's notebook:\n",
    "\n",
    "[GALA Coding Club 15.ipynb](https://github.com/BrianArbuckle/GALA/blob/master/GALA_Coding_Club_15.ipynb)\n",
    "\n",
    "Now you can save a local copy, which will be in your google drive. \n",
    "\n",
    "# Files and File Paths\n",
    "\n",
    "I have glossed over this topic, as we have been importing ZIP Code data from a local source, that I have always created.  Starting to store our own data is crucial next step to working with the data in our project. \n",
    "\n",
    "Variables are a great way to store data while your program or notebook is running, but if you want\n",
    "your data to persist even after your program has finished, you need to save it to a file.  \n",
    "\n",
    "Think of our large shape files that we imported from the US Census data. We are in the process of reducing the data to a smaller 'workable' data frame just the unique zip zip codes.  We will need to save this reduced size if we want to continue working with the data. \n",
    "\n",
    "\n",
    "### Properties\n",
    "A file has two main properties: a **filename** (typically written as one word) and a **path**. The path specifies the location of a file on the computer.   The part of the filename after the last period is called the file’s **extension** and tells you a file’s type. **project.docx** is a Word document, and our first zip code data that we imported **free-zipcode-database-Primary.csv** is a csv, or column separated values file.  \n",
    "\n",
    "Let's look at our import portion of out notebook from last week *GALA Coding Club 14* if you run this, is will not work as you do not have a local copy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N_9opgekG90g"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "zip_df = pd.read_csv(\"data/free-zipcode-database-Primary.csv\",index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5tw9O-p0G90k"
   },
   "source": [
    "Take note of the *data* before our csv and the shape file.  'data' is a ***folder***, or more often in computer programing, it is called a **directory**.  Also notices the slash.\n",
    "\n",
    "### Backslash on Windows and Forward Slash on OS X and Linux\n",
    "On Windows, paths are written using backslashes (\\\\) as the separator between folder names. OS X and Linux, however, use the forward slash (/) as their path separator. If you want your programs to work on all operating systems, you will have to write your Python scripts to handle both cases.  We will not worry about it for a long time, but the good news is there is a module that helps with this process, which is called the **<code>os</code>** module.  In addition to slashes, there fantastic things the os module does, which we will dive into now.  First, as we have for other modules or *libraries*, we will need to import the **os** module.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Tb5bp_OLG90k"
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OSKOIjjWG90m"
   },
   "source": [
    "### Jupyter Notebooks and Directories. \n",
    "\n",
    "Trying to find our current working location can be quite important when writing a program. The nice thing when working with Jupyter Notebooks is our working directory, unless we change it, is the same as our working directory. Let's confirm that with the **<code>os.getcwd()</code>**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "cg_9uYs1G90n",
    "outputId": "8bbefcac-5e4f-4c5d-b937-f24eef3e8558"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/content'"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G3qQX1nPG90s"
   },
   "source": [
    "In GALA Coding Club 14 we listed all the various files needed when importing the single <code>shape</code>.  We did this with the **<code>os.listdir()</code>**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "hCff6-35G90t",
    "outputId": "1a16bbfe-53bc-4ee8-92e2-ff87fdbf27f4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.config', 'sample_data']"
      ]
     },
     "execution_count": 3,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EMDli2V0G90v"
   },
   "source": [
    "We can also create a new director with the **<code>os.mkdir()</code>**, which need a string argument for the name of the folder / directory.  Since we will be needing the data files for our project, lets create a new directory called data<br>\n",
    "PLEASE RUN IF THIS IS THE FIRST TIME USING THIS NOTEBOOOK:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hVzpWlv9G90w"
   },
   "outputs": [],
   "source": [
    "os.mkdir('data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pFdlGq_XG90z"
   },
   "source": [
    "We can also change the working directory with the **<code>os.chdir()</code>** to the data folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Gki1b6suG90z"
   },
   "outputs": [],
   "source": [
    "os.chdir('data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YZxjJa9_G901"
   },
   "source": [
    "Let's check inside with os.listdir() again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "6o_yYgLmG902",
    "outputId": "b970ae09-4e42-49e4-cd60-ad773f917056"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kTASGws_G905"
   },
   "source": [
    "Look what happens when we try and change the the directory to `data` again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 164
    },
    "colab_type": "code",
    "id": "hrqzH02UG906",
    "outputId": "e5f88ff9-5095-46ab-8957-83954fd65ff6"
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-6920d9013d82>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'data'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data'"
     ]
    }
   ],
   "source": [
    "os.chdir('data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p-Z8mqGqG90-"
   },
   "source": [
    "# Absolute vs. Relative Paths\n",
    "There are two ways to specify a file path.\n",
    "\n",
    "* An **absolute path**, which always begins with the root folder, on my computer that is 'Users'<br>\n",
    "* However an **absolute path**, though is the same in colab, has a complicated twist.  Your \"root\" folder, is technically the computer that is assigned to you when you start a new colab notebook.  When you close it and start it again, it become a new computer.  So it is important to note that we treat the '/content' as our root directory. <br>\n",
    "* A **relative path**, which is relative to the program’s current working directory.\n",
    "\n",
    "By changing the path to 'data' without the full path, python assumes that it is relative to the current working directory.  \n",
    "\n",
    "For now, since it is easy to do in Jupyter Notebooks, we will be using the relative path.  Before we do that, we need to change back to our main folder:\n",
    "\n",
    "    '/content'\n",
    "    \n",
    "We coud use the Absolute Path, but there is a quicker way to move to the director containing the current director, and that is '..':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lAuf9QxSG90_"
   },
   "outputs": [],
   "source": [
    "os.chdir('..')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "p6ZCerODG91C",
    "outputId": "112621a5-3463-48f9-ad48-e51c74a8a09c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.config', 'data', 'sample_data']"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IOfJIFGBJjxZ"
   },
   "source": [
    "# Getting Data Into Colab\n",
    "\n",
    "Once again we will use a module, this time it is **<code>urllib</code>** module. And all we need inside the module is a sub-library called **<code>request</code>** and a single function **<code>urlretrieve</code>**. And we will assign the csv link, from our GitHub repository and assign it to the variable **url**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EmUudCPCG91F"
   },
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "\n",
    "#Assign url of file: url\n",
    "url = \"http://federalgovernmentzipcodes.us/free-zipcode-database-Primary.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "byEZZ4uTOAXW"
   },
   "source": [
    "# Save file locally\n",
    "\n",
    "We will use the **<code>urlretrieve</code>** module, which requires two arguments. First is the web address of the the file that we want to copy. The second, is the destination path, notice that I have the **<code>data/</code>** in the path name.  Since we changed to our main directory, <code>/content</code> and we created the new directory 'data'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "kTL2wphyG91H",
    "outputId": "2c993685-ccd2-4e9c-8c72-dfda1fa2420c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('data/free-zipcode-database-Primary.csv',\n",
       " <http.client.HTTPMessage at 0x7fb494b3e518>)"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urlretrieve(url, 'data/free-zipcode-database-Primary.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9jSDNOkIN_Cz"
   },
   "source": [
    "Now Let's list the contents of the data directory.  This time, we will not change our working directory, but simiply pass the 'data' folder name as the argument in the **<code>os.listdir()</code>. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "YZEB6NqQG91J",
    "outputId": "07998717-6716-44f1-f556-b799d27c7c23"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['free-zipcode-database-Primary.csv']"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir('data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KeUiRAAiQN4Q"
   },
   "source": [
    "### Now we can use the data (This is the code from above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GLjIHdyAG91N"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "zip_df = pd.read_csv(\"data/free-zipcode-database-Primary.csv\",index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 235
    },
    "colab_type": "code",
    "id": "DRIfoN-3G91P",
    "outputId": "96d70d09-4cd9-4df4-d724-1f15ed5fd453"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ZipCodeType</th>\n",
       "      <th>City</th>\n",
       "      <th>State</th>\n",
       "      <th>LocationType</th>\n",
       "      <th>Lat</th>\n",
       "      <th>Long</th>\n",
       "      <th>Location</th>\n",
       "      <th>Decommisioned</th>\n",
       "      <th>TaxReturnsFiled</th>\n",
       "      <th>EstimatedPopulation</th>\n",
       "      <th>TotalWages</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zipcode</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>705</th>\n",
       "      <td>STANDARD</td>\n",
       "      <td>AIBONITO</td>\n",
       "      <td>PR</td>\n",
       "      <td>PRIMARY</td>\n",
       "      <td>18.14</td>\n",
       "      <td>-66.26</td>\n",
       "      <td>NA-US-PR-AIBONITO</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>610</th>\n",
       "      <td>STANDARD</td>\n",
       "      <td>ANASCO</td>\n",
       "      <td>PR</td>\n",
       "      <td>PRIMARY</td>\n",
       "      <td>18.28</td>\n",
       "      <td>-67.14</td>\n",
       "      <td>NA-US-PR-ANASCO</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611</th>\n",
       "      <td>PO BOX</td>\n",
       "      <td>ANGELES</td>\n",
       "      <td>PR</td>\n",
       "      <td>PRIMARY</td>\n",
       "      <td>18.28</td>\n",
       "      <td>-66.79</td>\n",
       "      <td>NA-US-PR-ANGELES</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>612</th>\n",
       "      <td>STANDARD</td>\n",
       "      <td>ARECIBO</td>\n",
       "      <td>PR</td>\n",
       "      <td>PRIMARY</td>\n",
       "      <td>18.45</td>\n",
       "      <td>-66.73</td>\n",
       "      <td>NA-US-PR-ARECIBO</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601</th>\n",
       "      <td>STANDARD</td>\n",
       "      <td>ADJUNTAS</td>\n",
       "      <td>PR</td>\n",
       "      <td>PRIMARY</td>\n",
       "      <td>18.16</td>\n",
       "      <td>-66.72</td>\n",
       "      <td>NA-US-PR-ADJUNTAS</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        ZipCodeType      City State LocationType    Lat   Long  \\\n",
       "Zipcode                                                          \n",
       "705        STANDARD  AIBONITO    PR      PRIMARY  18.14 -66.26   \n",
       "610        STANDARD    ANASCO    PR      PRIMARY  18.28 -67.14   \n",
       "611          PO BOX   ANGELES    PR      PRIMARY  18.28 -66.79   \n",
       "612        STANDARD   ARECIBO    PR      PRIMARY  18.45 -66.73   \n",
       "601        STANDARD  ADJUNTAS    PR      PRIMARY  18.16 -66.72   \n",
       "\n",
       "                  Location  Decommisioned  TaxReturnsFiled  \\\n",
       "Zipcode                                                      \n",
       "705      NA-US-PR-AIBONITO          False              NaN   \n",
       "610        NA-US-PR-ANASCO          False              NaN   \n",
       "611       NA-US-PR-ANGELES          False              NaN   \n",
       "612       NA-US-PR-ARECIBO          False              NaN   \n",
       "601      NA-US-PR-ADJUNTAS          False              NaN   \n",
       "\n",
       "         EstimatedPopulation  TotalWages  \n",
       "Zipcode                                   \n",
       "705                      NaN         NaN  \n",
       "610                      NaN         NaN  \n",
       "611                      NaN         NaN  \n",
       "612                      NaN         NaN  \n",
       "601                      NaN         NaN  "
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zip_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YUzXQDQnG91S"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X8dRpX9wbLpm"
   },
   "source": [
    "# Save the data once\n",
    "\n",
    "We have been using the <code>free-zipcode-database-Primary.csv</code> in many of our notebooks, but only need to copy it and save it to our data folder once. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lpQwwFY9G91V"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vs7pfpY9G91W"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XxUoqNP_G91g"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "GALA Coding Club 15.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}