2018_summer/synch_session_agendas/synch_10_python_requests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Demonstration of how to use the Python requests module \n",
    "* pulling down a plain vanilla web page (html, javascript, etc.)\n",
    "  * checking the return code\n",
    "  * getting the headers\n",
    "  * getting the web page content (html, javascript, etc.)\n",
    "* web api calls\n",
    "  * passing parameters in json format\n",
    "  * receiving responses in json format\n",
    "* downloading a text file or binary file of any type from the internet\n",
    "* downloading a zip file from the internet and unzipping it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import os\n",
    "import zipfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plain Vanilla Web Page\n",
    "We will start with a plain vanilla web page using the google.com landing page.  \n",
    "The get() method corresponds to the http GET."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = requests.get(\"https://google.com\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the status code, 200 means ok, errors would be like 401, 404, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily retrieve the http headers in dictionary form in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Date': 'Thu, 12 Jul 2018 18:02:39 GMT', 'Expires': '-1', 'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/html; charset=ISO-8859-1', 'P3P': 'CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"', 'Content-Encoding': 'gzip', 'Server': 'gws', 'X-XSS-Protection': '1; mode=block', 'X-Frame-Options': 'SAMEORIGIN', 'Set-Cookie': '1P_JAR=2018-07-12-18; expires=Sat, 11-Aug-2018 18:02:39 GMT; path=/; domain=.google.com, NID=134=TWpGIEamkY9a7ijN-cfEffhiJoNoTutoOgWwycoNZIiwFD5r-L-lofjUUoPqFie8EwOuLMjHWFUwUqFpGmWR-0ogUOf1z0PlYUVlvPvJeC3M5ZDYrWqaM5r4LAvUncWp; expires=Fri, 11-Jan-2019 18:02:39 GMT; path=/; domain=.google.com; HttpOnly', 'Alt-Svc': 'quic=\":443\"; ma=2592000; v=\"43,42,41,39,35\"', 'Transfer-Encoding': 'chunked'}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily retrive the text of the webpage (html, http, etc.)  With google.com, as you can see the html is really long, so we will only look at the first 500 characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10762"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"en\"><head><meta content=\"Search the world\\'s information, including webpages, images, videos and more. Google has many special features to help you find exactly what you\\'re looking for.\" name=\"description\"><meta content=\"noodp\" name=\"robots\"><meta content=\"text/html; charset=UTF-8\" http-equiv=\"Content-Type\"><meta content=\"/images/branding/googleg/1x/googleg_standard_color_128dp.png\" itemprop=\"image\"><title>Google</title>'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# it's long so let's just see the first 500 characters of the html returned\n",
    "r.text[0:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Web API Call\n",
    "We will now make a web API call. We will send json as the \"payload\" and receive json back.\n",
    "The post() method corresponds to an http POST. The data keywork argument is a python dictionary with the payload.\n",
    "The json() method takes the response text and formats it into a python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = requests.post('http://httpbin.org/post', data = {'key':'value'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"args\":{},\"data\":\"\",\"files\":{},\"form\":{\"key\":\"value\"},\"headers\":{\"Accept\":\"*/*\",\"Accept-Encoding\":\"gzip, deflate\",\"Connection\":\"close\",\"Content-Length\":\"9\",\"Content-Type\":\"application/x-www-form-urlencoded\",\"Host\":\"httpbin.org\",\"User-Agent\":\"python-requests/2.12.4\"},\"json\":null,\"origin\":\"12.228.204.18\",\"url\":\"http://httpbin.org/post\"}\\n'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'args': {},\n",
       " 'data': '',\n",
       " 'files': {},\n",
       " 'form': {'key': 'value'},\n",
       " 'headers': {'Accept': '*/*',\n",
       "  'Accept-Encoding': 'gzip, deflate',\n",
       "  'Connection': 'close',\n",
       "  'Content-Length': '9',\n",
       "  'Content-Type': 'application/x-www-form-urlencoded',\n",
       "  'Host': 'httpbin.org',\n",
       "  'User-Agent': 'python-requests/2.12.4'},\n",
       " 'json': None,\n",
       " 'origin': '12.228.204.18',\n",
       " 'url': 'http://httpbin.org/post'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Another Web API Call Example\n",
    "Here is an example of another web API call to a website that gives position information on the internal space station and returns it in json format.  It does not need parameters, so we don't need to pass a payload, so we will use the get() method.\n",
    "\n",
    "You may want to check out their website for more API calls to try:\n",
    "\n",
    "https://wheretheiss.at/w/developer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = requests.get('https://api.wheretheiss.at/v1/satellites/25544')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"name\":\"iss\",\"id\":25544,\"latitude\":49.516621094704,\"longitude\":120.59381781782,\"altitude\":412.17433179078,\"velocity\":27608.11542189,\"visibility\":\"daylight\",\"footprint\":4467.4337370703,\"timestamp\":1531417775,\"daynum\":2458312.2427662,\"solar_lat\":21.895378260365,\"solar_lon\":274.02987776115,\"units\":\"kilometers\"}'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'altitude': 412.17433179078,\n",
       " 'daynum': 2458312.2427662,\n",
       " 'footprint': 4467.4337370703,\n",
       " 'id': 25544,\n",
       " 'latitude': 49.516621094704,\n",
       " 'longitude': 120.59381781782,\n",
       " 'name': 'iss',\n",
       " 'solar_lat': 21.895378260365,\n",
       " 'solar_lon': 274.02987776115,\n",
       " 'timestamp': 1531417775,\n",
       " 'units': 'kilometers',\n",
       " 'velocity': 27608.11542189,\n",
       " 'visibility': 'daylight'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download an Excel File from the internet \n",
    "We will now download an excel file from the internet.  This same method works with any file you want to download, a text file, or any other binary file, such as Word, pdf, images such as jpeg, videos such as mp4, a zip file, a tarball file, etc.  Basically any file.\n",
    "\n",
    "We simply call the get() method and the r.content attribute will hold the data in binary format with encoding.  We can them open a file in wb mode (write binary), write the contents, and close the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = requests.get('http://kevincrook.com/ucb/data/coffee_chain.xlsx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'PK\\x03\\x04\\x14\\x00\\x06\\x00\\x08\\x00\\x00\\x00!\\x00\\x9c\\xd7\\xfc\\xa8^\\x01\\x00\\x00<\\x04\\x00\\x00\\x13\\x00\\xcf\\x01[Content_Types].xml \\xa2\\xcb\\x01(\\xa0\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.content[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = open(\"coffee_chain.xlsx\", \"wb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "423875"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.write(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download a Zip File from the Internet and Unzip it\n",
    "We will download a zip file from the internet using the previous method of downloading a binary file.\n",
    "\n",
    "Using the os module, we will make a temporary directory to hold the zip file.  We will also use this same directory to unzip the contents.\n",
    "\n",
    "We will use the zipfile module to unzip the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = requests.get('http://kevincrook.com/ucb/data/Hospital_Compare_Data_OpenRefine_Breakout.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\KRC\\\\Berkeley\\\\W205_Data_Engineering'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(\"temp_zip\"):\n",
    "    os.mkdir(\"temp_zip\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'temp_zip\\\\Hospital_Compare_Data_OpenRefine_Breakout.zip'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir_file = os.path.join(\"temp_zip\", \"Hospital_Compare_Data_OpenRefine_Breakout.zip\" )\n",
    "dir_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f = open(dir_file, \"wb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3149304"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.write(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = zipfile.ZipFile(dir_file, \"r\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.extractall(\"temp_zip\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}