codelabs/CodeLab0TwitterAPI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CodeLab 0: Twitter API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Questions? Message @daniel.zou on Slack! :>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTTP Requests Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's learn how to use the Twitter API. For simplicity, we will use the Twitter API directly. There are many Python API wrappers like Tweepy that give us convenient functions for using the Twitter API. In this notebook, we will not learn a specific API wrapper."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our objective to get send a GET request to Twitter's servers. Twitter will interpret our request, do some logic on its end, and send us a response."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For starters, let's actually send a GET request to a familiar server: ```https://google.com/```. We'll use the python `requests` package, which is a very common library used to send HTTP requests in Python. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "r = requests.get(\"https://google.com/\")\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We received a status code 200, which means success. For your own learning, you can check out all other possible status codes [here](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes).\n",
    "\n",
    "We can grab the data from our response using `r.text`. Let's do that now and see what it looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"en\"><head><meta content=\"Search the world\\'s information, including webpages, images, videos and more. Google has many special features to help you find exactly what you\\'re looking for.\" name=\"description\"><meta content=\"noo'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.text[:300] # Print only the first 300 characters for brevity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this particular response, Google sends us HTML which our browsers can read and render as a webpage. Let's render this HTML into our notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"en\"><head><meta content=\"Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for.\" name=\"description\"><meta content=\"noodp\" name=\"robots\"><meta content=\"text/html; charset=UTF-8\" http-equiv=\"Content-Type\"><meta content=\"/images/branding/googleg/1x/googleg_standard_color_128dp.png\" itemprop=\"image\"><title>Google</title><script nonce=\"0uEC1P3IUv71fTcXaRiIdA==\">(function(){window.google={kEI:'eu5kXL-5D4uz8AOv26DIDw',kEXPI:'0,201941,1151805,58,1957,1017,625,781,697,528,731,749,1049,30,696,397,135,805,1077,106,489,5,2334606,329531,1294,12383,4855,32692,15247,867,10761,1402,5281,1100,3335,2,2,4605,2196,363,1166,7,2147,5505,224,2212,266,3741,1366,575,835,284,2,578,728,2431,59,2,1,3,1297,4323,3390,8,302,1267,774,2250,1407,3337,1146,5,2,2,1963,2595,182,839,2580,669,1050,1808,1129,268,81,7,1,2,488,620,29,983,6,406,458,2616,536,696,3394,1209,876,412,2,554,2635,380,438,796,1220,38,363,557,718,155,1215,1366,484,47,1080,542,1538,656,1558,1245,258,2,631,217,2050,136,159,2,4,2,670,44,334,600,8,573,2,4,246,420,349,167,82,98,1420,355,125,492,668,1447,12,87,533,464,340,293,318,43,229,76,26,330,637,127,17,321,255,2,115,411,1242,1140,206,403,136,21,186,150,251,366,458,435,3,308,329,122,207,217,197,135,64,21,1,12,392,30,390,9,39,337,1,96,320,199,294,55,77,26,4,246,10,296,64,213,222,88,36,177,600,348,512,5960859,2554,233,21,8797532,4,1572,549,333,444,1,2,80,1,900,583,9,304,1,8,1,2,2132,1,1,1,1,1,414,1,748,141,59,726,3,7,563,1,599,39,66,3,30,1,7,2,3,1,1,1,2,1,1,1,1,2,3,4,16,3,15,14,15,18,9,1,1,1,1,3,1,1,1,1,1,1,2,1,4,4,4,29,16,5,20,22306182',authuser:0,kscs:'c9c918f0_eu5kXL-5D4uz8AOv26DIDw',kGL:'US'};google.kHL='en';})();google.time=function(){return(new Date).getTime()};(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute(\"eid\")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute(\"leid\")));)a=a.parentNode;return b};google.https=function(){return\"https:\"==window.location.protocol};google.ml=function(){return null};google.log=function(a,b,e,c,g){if(a=google.logUrl(a,b,e,c,g)){b=new Image;var d=google.lc,f=google.li;d[f]=b;b.onerror=b.onload=b.onabort=function(){delete d[f]};google.vel&&google.vel.lu&&google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,e,c,g){var d=\"\",f=google.ls||\"\";e||-1!=b.search(\"&ei=\")||(d=\"&ei=\"+google.getEI(c),-1==b.search(\"&lei=\")&&(c=google.getLEI(c))&&(d+=\"&lei=\"+c));c=\"\";!e&&google.cshid&&-1==b.search(\"&cshid=\")&&\"slh\"!=a&&(c=\"&cshid=\"+google.cshid);a=e||\"/\"+(g||\"gen_204\")+\"?atyp=i&ct=\"+a+\"&cad=\"+b+d+f+\"&zx=\"+google.time()+c;/^http:/i.test(a)&&google.https()&&(google.ml(Error(\"a\"),!1,{src:a,glmm:1}),a=\"\");return a};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};</script><script nonce=\"0uEC1P3IUv71fTcXaRiIdA==\">var a=window.location,b=a.href.indexOf(\"#\");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf(\"#\")&&a.replace(\"/search?\"+c.replace(/(^|&)fp=[^&]*/g,\"\")+\"&cad=h\")};</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}\n",
       "</style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/nav_logo229.png) 0 -261px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}</style><script nonce=\"0uEC1P3IUv71fTcXaRiIdA==\"></script></head><body bgcolor=\"#fff\"><script nonce=\"0uEC1P3IUv71fTcXaRiIdA==\">(function(){var src='/images/nav_logo229.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}\n",
       "if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}\n",
       "}\n",
       "})();</script><div id=\"mngb\"> <div id=gbar><nobr><b class=gb1>Search</b> <a class=gb1 href=\"https://www.google.com/imghp?hl=en&tab=wi\">Images</a> <a class=gb1 href=\"https://maps.google.com/maps?hl=en&tab=wl\">Maps</a> <a class=gb1 href=\"https://play.google.com/?hl=en&tab=w8\">Play</a> <a class=gb1 href=\"https://www.youtube.com/?gl=US&tab=w1\">YouTube</a> <a class=gb1 href=\"https://news.google.com/nwshp?hl=en&tab=wn\">News</a> <a class=gb1 href=\"https://mail.google.com/mail/?tab=wm\">Gmail</a> <a class=gb1 href=\"https://drive.google.com/?tab=wo\">Drive</a> <a class=gb1 style=\"text-decoration:none\" href=\"https://www.google.com/intl/en/about/products?tab=wh\"><u>More</u> &raquo;</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href=\"http://www.google.com/history/optout?hl=en\" class=gb4>Web History</a> | <a  href=\"/preferences?hl=en\" class=gb4>Settings</a> | <a target=_top id=gb_70 href=\"https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/\" class=gb4>Sign in</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div> </div><center><br clear=\"all\" id=\"lgpd\"><div id=\"lga\"><img alt=\"Google\" height=\"92\" src=\"/images/branding/googlelogo/1x/googlelogo_white_background_color_272x92dp.png\" style=\"padding:28px 0 14px\" width=\"272\" id=\"hplogo\" onload=\"window.lol&&lol()\"><br><br></div><form action=\"/search\" name=\"f\"><table cellpadding=\"0\" cellspacing=\"0\"><tr valign=\"top\"><td width=\"25%\">&nbsp;</td><td align=\"center\" nowrap=\"\"><input name=\"ie\" value=\"ISO-8859-1\" type=\"hidden\"><input value=\"en\" name=\"hl\" type=\"hidden\"><input name=\"source\" type=\"hidden\" value=\"hp\"><input name=\"biw\" type=\"hidden\"><input name=\"bih\" type=\"hidden\"><div class=\"ds\" style=\"height:32px;margin:4px 0\"><input style=\"color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top\" autocomplete=\"off\" class=\"lst\" value=\"\" title=\"Google Search\" maxlength=\"2048\" name=\"q\" size=\"57\"></div><br style=\"line-height:0\"><span class=\"ds\"><span class=\"lsbb\"><input class=\"lsb\" value=\"Google Search\" name=\"btnG\" type=\"submit\"></span></span><span class=\"ds\"><span class=\"lsbb\"><input class=\"lsb\" value=\"I'm Feeling Lucky\" name=\"btnI\" onclick=\"if(this.form.q.value)this.checked=1; else top.location='/doodles/'\" type=\"submit\"></span></span></td><td class=\"fl sblc\" align=\"left\" nowrap=\"\" width=\"25%\"><a href=\"/advanced_search?hl=en&amp;authuser=0\">Advanced search</a><a href=\"/language_tools?hl=en&amp;authuser=0\">Language tools</a></td></tr></table><input id=\"gbv\" name=\"gbv\" type=\"hidden\" value=\"1\"><script nonce=\"0uEC1P3IUv71fTcXaRiIdA==\">(function(){var a,b=\"1\";if(document&&document.getElementById)if(\"undefined\"!=typeof XMLHttpRequest)b=\"2\";else if(\"undefined\"!=typeof ActiveXObject){var c,d,e=[\"MSXML2.XMLHTTP.6.0\",\"MSXML2.XMLHTTP.3.0\",\"MSXML2.XMLHTTP\",\"Microsoft.XMLHTTP\"];for(c=0;d=e[c++];)try{new ActiveXObject(d),b=\"2\"}catch(h){}}a=b;if(\"2\"==a&&-1==location.search.indexOf(\"&gbv=2\")){var f=google.gbvu,g=document.getElementById(\"gbv\");g&&(g.value=a);f&&window.setTimeout(function(){location.href=f},0)};}).call(this);</script></form><div id=\"gac_scont\"></div><div style=\"font-size:83%;min-height:3.5em\"><br><div id=\"prm\"><style>.szppmdbYutt__middle-slot-promo{font-size:small;margin-bottom:32px}.szppmdbYutt__middle-slot-promo a.ZIeIlb{display:inline-block;text-decoration:none}.szppmdbYutt__middle-slot-promo img{border:none;margin-right:5px;vertical-align:middle}</style><div class=\"szppmdbYutt__middle-slot-promo\" data-ved=\"0ahUKEwi_3oO_sbrgAhWLGXwKHa8tCPkQnIcBCAQ\"><a class=\"NKcBbd\" href=\"https://www.google.com/url?q=https://earth.google.com/web/%4034.7325522,-94.20826205,312.21020567a,12000007.13021886d,35y,0h,0t,0r/data%3DCjASLhIgN2JiMTc5NDUyMzJhMTFlOTg4ZTMzYmJlMjQ5MGY5NmUiCnZveV9zcGxhc2g%3Futm_source%3Dreferral%26utm_campaign%3Dhpp%26utm_term%3Dblackhistory&amp;source=hpp&amp;id=19010948&amp;ct=3&amp;usg=AFQjCNGZojoukqnVvxGEwPdncjIl3yQsmw&amp;sa=X&amp;ved=0ahUKEwi_3oO_sbrgAhWLGXwKHa8tCPkQ8IcBCAU\" rel=\"nofollow\">Take a journey through Black history with Google Earth</a></div></div></div><span id=\"footer\"><div style=\"font-size:10pt\"><div style=\"margin:19px auto;text-align:center\" id=\"fll\"><a href=\"/intl/en/ads/\">Advertising Programs</a><a href=\"/services/\">Business Solutions</a><a href=\"https://plus.google.com/116899029375914044550\" rel=\"publisher\">+Google</a><a href=\"/intl/en/about.html\">About Google</a></div></div><p style=\"color:#767676;font-size:8pt\">&copy; 2019 - <a href=\"/intl/en/policies/privacy/\">Privacy</a> - <a href=\"/intl/en/policies/terms/\">Terms</a></p></span></center><script nonce=\"0uEC1P3IUv71fTcXaRiIdA==\">(function(){window.google.cdo={height:0,width:0};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b){var c=window.document,d=\"CSS1Compat\"==c.compatMode?c.documentElement:c.body;a=d.clientWidth;b=d.clientHeight}a&&b&&(a!=google.cdo.width||b!=google.cdo.height)&&google.log(\"\",\"\",\"/client_204?&atyp=i&biw=\"+a+\"&bih=\"+b+\"&ei=\"+google.kEI);}).call(this);})();(function(){var u='/xjs/_/js/k\\x3dxjs.hp.en_US.JrhqsPLSQXo.O/m\\x3dsb_he,d/am\\x3dYsAs/rt\\x3dj/d\\x3d1/rs\\x3dACT90oHZvWqSsLIwXzgtCUSZcUX3HykDQA';var b={gen204:\"xjsls\",clearcut:31};setTimeout(function(){var a=document.createElement(\"script\");a.src=u;google.timers&&google.timers.load&&google.tick&&google.tick(\"load\",b);document.body.appendChild(a)},0);})();(function(){window.google.xjsu='/xjs/_/js/k\\x3dxjs.hp.en_US.JrhqsPLSQXo.O/m\\x3dsb_he,d/am\\x3dYsAs/rt\\x3dj/d\\x3d1/rs\\x3dACT90oHZvWqSsLIwXzgtCUSZcUX3HykDQA';})();function _DumpException(e){throw e;}\n",
       "(function(){var pmc='{\\x22Qnk92g\\x22:{},\\x22U5B21g\\x22:{},\\x22YFCs/g\\x22:{},\\x22ZI/YVQ\\x22:{},\\x22d\\x22:{},\\x22sb_he\\x22:{\\x22agen\\x22:true,\\x22cgen\\x22:true,\\x22client\\x22:\\x22heirloom-hp\\x22,\\x22dh\\x22:true,\\x22dhqt\\x22:true,\\x22ds\\x22:\\x22\\x22,\\x22ffql\\x22:\\x22en\\x22,\\x22fl\\x22:true,\\x22host\\x22:\\x22google.com\\x22,\\x22isbh\\x22:28,\\x22jsonp\\x22:true,\\x22msgs\\x22:{\\x22cibl\\x22:\\x22Clear Search\\x22,\\x22dym\\x22:\\x22Did you mean:\\x22,\\x22lcky\\x22:\\x22I\\\\u0026#39;m Feeling Lucky\\x22,\\x22lml\\x22:\\x22Learn more\\x22,\\x22oskt\\x22:\\x22Input tools\\x22,\\x22psrc\\x22:\\x22This search was removed from your \\\\u003Ca href\\x3d\\\\\\x22/history\\\\\\x22\\\\u003EWeb History\\\\u003C/a\\\\u003E\\x22,\\x22psrl\\x22:\\x22Remove\\x22,\\x22sbit\\x22:\\x22Search by image\\x22,\\x22srch\\x22:\\x22Google Search\\x22},\\x22ovr\\x22:{},\\x22pq\\x22:\\x22\\x22,\\x22refpd\\x22:true,\\x22rfs\\x22:[],\\x22sbpl\\x22:24,\\x22sbpr\\x22:24,\\x22scd\\x22:10,\\x22sce\\x22:5,\\x22stok\\x22:\\x22rdv8Ay8aWjHWX_tGrsQGowlpp1k\\x22,\\x22uhde\\x22:false}}';google.pmc=JSON.parse(pmc);})();</script>        </body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sending Requests to the Twitter API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Twitter API can be accessed by a URL, just like `Google.com`. Specifically, we want to send a GET request to `https://api.twitter.com/1.1/ search/tweets.json?q=SFMTA`. We already know how to send GET requests, so let's just do that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [400]>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get(\"https://api.twitter.com/1.1/search/tweets.json?q=SFMTA&tweet_mode=extended\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, we got a 400 response, which means failure. This is because Twitter restricts their API to registered developers. So first, we have to get some developer credentials by following the instructions [here](https://developer.twitter.com/en/apply-for-access.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will place our credentials in a file called `creds.txt`, in the same directory as this notebook, in this format:\n",
    "<br></br>`[ConsumerAPIkeys.APIkey]`<br></br>\n",
    "`[ConsumerAPIkeys.APIsecretkey]`<br></br>\n",
    "`[AccessToken&AccessSecret.AccessToken]`<br></br>\n",
    "`[AccessToken&AccessSecret.AccessTokenSecret]`<br></br>\n",
    "Where we replace `[something]` with its value (including deleting the brackets)<br></br>\n",
    "Make sure you separate each value with a newline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"creds.txt\", \"r\") as f:\n",
    "    contents = f.read()\n",
    "    keys = contents.split(\"\\n\")\n",
    "    APIkey = keys[0]\n",
    "    APIsecretkey = keys[1]\n",
    "    AccessToken = keys[2]\n",
    "    AccessTokenSecret = keys[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Twitter uses something called OAuth1, which is an open standard for accessing restricted data. `requests` has some features to help us manage OAuth1. However, these are not installed by default, so let's install it if necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (2.21.0)\n",
      "Requirement already satisfied: requests_oauthlib in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (1.2.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (from requests) (2018.11.29)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (from requests) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (from requests) (1.24.1)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (from requests) (2.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /Users/dz/.pyenv/versions/3.6.0/envs/jupyter3/lib/python3.6/site-packages (from requests_oauthlib) (3.0.1)\n",
      "\u001b[33mYou are using pip version 18.1, however version 19.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install requests requests_oauthlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from requests_oauthlib import OAuth1\n",
    "auth = OAuth1(APIkey, APIsecretkey, AccessToken, AccessTokenSecret)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's make our GET request like before, but pass into our `Auth` object to the method as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"statuses\":[{\"created_at\":\"Thu Feb 14 04:56:55 +0000 2019\",\"id\":1095909730185240576,\"id_str\":\"1095909730185240576\",\"full_text\":\"Next bus is 42 minutes farther up the line. How is this acceptable @sfm'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(\"https://api.twitter.com/1.1/search/tweets.json?q=SFMTA&tweet_mode=extended\", auth=auth)\n",
    "r.text[:200] # Let's only print the first 200 characters for brevity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our response data actually is in a format called JSON, which we can convert easily into a Python dictionary using the json package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['statuses', 'search_metadata'])\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "response_dict = json.loads(r.text)\n",
    "print(response_dict.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By inspecting the raw response text above, it looks like our tweets are in the \"statuses\" key, which maps to an array value. Looking at the array, we see that each tweet is an object in the array. Let's just grab the first tweet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'created_at': 'Thu Feb 14 04:56:55 +0000 2019',\n",
       " 'id': 1095909730185240576,\n",
       " 'id_str': '1095909730185240576',\n",
       " 'full_text': 'Next bus is 42 minutes farther up the line. How is this acceptable @sfmta_muni? https://t.co/4bXOX5ctbT',\n",
       " 'truncated': False,\n",
       " 'display_text_range': [0, 79],\n",
       " 'entities': {'hashtags': [],\n",
       "  'symbols': [],\n",
       "  'user_mentions': [{'screen_name': 'sfmta_muni',\n",
       "    'name': 'SFMTA',\n",
       "    'id': 109702390,\n",
       "    'id_str': '109702390',\n",
       "    'indices': [67, 78]}],\n",
       "  'urls': [{'url': 'https://t.co/4bXOX5ctbT',\n",
       "    'expanded_url': 'https://twitter.com/mmmmurf/status/1095909359895306240',\n",
       "    'display_url': 'twitter.com/mmmmurf/status…',\n",
       "    'indices': [80, 103]}]},\n",
       " 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},\n",
       " 'source': '<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>',\n",
       " 'in_reply_to_status_id': None,\n",
       " 'in_reply_to_status_id_str': None,\n",
       " 'in_reply_to_user_id': None,\n",
       " 'in_reply_to_user_id_str': None,\n",
       " 'in_reply_to_screen_name': None,\n",
       " 'user': {'id': 518897779,\n",
       "  'id_str': '518897779',\n",
       "  'name': 'Mike Murphy',\n",
       "  'screen_name': 'mmmmurf',\n",
       "  'location': 'San Francisco',\n",
       "  'description': 'Night editor of MarketWatch. Giants fan. Lover of fog and burritos.',\n",
       "  'url': 'https://t.co/hzJHKqZmh2',\n",
       "  'entities': {'url': {'urls': [{'url': 'https://t.co/hzJHKqZmh2',\n",
       "      'expanded_url': 'http://www.marketwatch.com/',\n",
       "      'display_url': 'marketwatch.com',\n",
       "      'indices': [0, 23]}]},\n",
       "   'description': {'urls': []}},\n",
       "  'protected': False,\n",
       "  'followers_count': 549,\n",
       "  'friends_count': 348,\n",
       "  'listed_count': 54,\n",
       "  'created_at': 'Thu Mar 08 21:32:06 +0000 2012',\n",
       "  'favourites_count': 5259,\n",
       "  'utc_offset': None,\n",
       "  'time_zone': None,\n",
       "  'geo_enabled': False,\n",
       "  'verified': False,\n",
       "  'statuses_count': 13194,\n",
       "  'lang': 'en',\n",
       "  'contributors_enabled': False,\n",
       "  'is_translator': False,\n",
       "  'is_translation_enabled': False,\n",
       "  'profile_background_color': 'C0DEED',\n",
       "  'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "  'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "  'profile_background_tile': False,\n",
       "  'profile_image_url': 'http://pbs.twimg.com/profile_images/3207831659/a95aad260953eeebfc35ecbcb3442416_normal.jpeg',\n",
       "  'profile_image_url_https': 'https://pbs.twimg.com/profile_images/3207831659/a95aad260953eeebfc35ecbcb3442416_normal.jpeg',\n",
       "  'profile_banner_url': 'https://pbs.twimg.com/profile_banners/518897779/1398231383',\n",
       "  'profile_link_color': '0084B4',\n",
       "  'profile_sidebar_border_color': 'C0DEED',\n",
       "  'profile_sidebar_fill_color': 'DDEEF6',\n",
       "  'profile_text_color': '333333',\n",
       "  'profile_use_background_image': True,\n",
       "  'has_extended_profile': True,\n",
       "  'default_profile': False,\n",
       "  'default_profile_image': False,\n",
       "  'following': False,\n",
       "  'follow_request_sent': False,\n",
       "  'notifications': False,\n",
       "  'translator_type': 'none'},\n",
       " 'geo': None,\n",
       " 'coordinates': None,\n",
       " 'place': None,\n",
       " 'contributors': None,\n",
       " 'is_quote_status': True,\n",
       " 'quoted_status_id': 1095909359895306240,\n",
       " 'quoted_status_id_str': '1095909359895306240',\n",
       " 'quoted_status': {'created_at': 'Thu Feb 14 04:55:27 +0000 2019',\n",
       "  'id': 1095909359895306240,\n",
       "  'id_str': '1095909359895306240',\n",
       "  'full_text': '30-minute wait for the next bus, which means 3 drivers in a row called in sick and were not replaced. Thanks for nothing @sfmta_muni!',\n",
       "  'truncated': False,\n",
       "  'display_text_range': [0, 133],\n",
       "  'entities': {'hashtags': [],\n",
       "   'symbols': [],\n",
       "   'user_mentions': [{'screen_name': 'sfmta_muni',\n",
       "     'name': 'SFMTA',\n",
       "     'id': 109702390,\n",
       "     'id_str': '109702390',\n",
       "     'indices': [121, 132]}],\n",
       "   'urls': []},\n",
       "  'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},\n",
       "  'source': '<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>',\n",
       "  'in_reply_to_status_id': None,\n",
       "  'in_reply_to_status_id_str': None,\n",
       "  'in_reply_to_user_id': None,\n",
       "  'in_reply_to_user_id_str': None,\n",
       "  'in_reply_to_screen_name': None,\n",
       "  'user': {'id': 518897779,\n",
       "   'id_str': '518897779',\n",
       "   'name': 'Mike Murphy',\n",
       "   'screen_name': 'mmmmurf',\n",
       "   'location': 'San Francisco',\n",
       "   'description': 'Night editor of MarketWatch. Giants fan. Lover of fog and burritos.',\n",
       "   'url': 'https://t.co/hzJHKqZmh2',\n",
       "   'entities': {'url': {'urls': [{'url': 'https://t.co/hzJHKqZmh2',\n",
       "       'expanded_url': 'http://www.marketwatch.com/',\n",
       "       'display_url': 'marketwatch.com',\n",
       "       'indices': [0, 23]}]},\n",
       "    'description': {'urls': []}},\n",
       "   'protected': False,\n",
       "   'followers_count': 549,\n",
       "   'friends_count': 348,\n",
       "   'listed_count': 54,\n",
       "   'created_at': 'Thu Mar 08 21:32:06 +0000 2012',\n",
       "   'favourites_count': 5259,\n",
       "   'utc_offset': None,\n",
       "   'time_zone': None,\n",
       "   'geo_enabled': False,\n",
       "   'verified': False,\n",
       "   'statuses_count': 13194,\n",
       "   'lang': 'en',\n",
       "   'contributors_enabled': False,\n",
       "   'is_translator': False,\n",
       "   'is_translation_enabled': False,\n",
       "   'profile_background_color': 'C0DEED',\n",
       "   'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "   'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "   'profile_background_tile': False,\n",
       "   'profile_image_url': 'http://pbs.twimg.com/profile_images/3207831659/a95aad260953eeebfc35ecbcb3442416_normal.jpeg',\n",
       "   'profile_image_url_https': 'https://pbs.twimg.com/profile_images/3207831659/a95aad260953eeebfc35ecbcb3442416_normal.jpeg',\n",
       "   'profile_banner_url': 'https://pbs.twimg.com/profile_banners/518897779/1398231383',\n",
       "   'profile_link_color': '0084B4',\n",
       "   'profile_sidebar_border_color': 'C0DEED',\n",
       "   'profile_sidebar_fill_color': 'DDEEF6',\n",
       "   'profile_text_color': '333333',\n",
       "   'profile_use_background_image': True,\n",
       "   'has_extended_profile': True,\n",
       "   'default_profile': False,\n",
       "   'default_profile_image': False,\n",
       "   'following': False,\n",
       "   'follow_request_sent': False,\n",
       "   'notifications': False,\n",
       "   'translator_type': 'none'},\n",
       "  'geo': None,\n",
       "  'coordinates': None,\n",
       "  'place': None,\n",
       "  'contributors': None,\n",
       "  'is_quote_status': False,\n",
       "  'retweet_count': 0,\n",
       "  'favorite_count': 0,\n",
       "  'favorited': False,\n",
       "  'retweeted': False,\n",
       "  'lang': 'en'},\n",
       " 'retweet_count': 0,\n",
       " 'favorite_count': 0,\n",
       " 'favorited': False,\n",
       " 'retweeted': False,\n",
       " 'possibly_sensitive': False,\n",
       " 'lang': 'en'}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet = response_dict[\"statuses\"][0]\n",
    "tweet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! Twitter gives us a lot of interesting data to work with, some of which might be handy later. For now, let's just extract the message out of this object using the \"full_text\" field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Next bus is 42 minutes farther up the line. How is this acceptable @sfmta_muni? https://t.co/4bXOX5ctbT'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet[\"full_text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay fantastic! We've successfully made a request to the Twitter API search endpoint directly supplying the \"SFMTA\" query string. With this foundation, we can start pulling and saving tweets so that we can do analysis on them later. There's also tons of interesting and useful metadata that Twitter supplies us that may help us do a better job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks for sticking with it and reaching the end of the CodeLab. I encourage you to continue working through the CodeLabs so that you can get up to speed and start contributing to #nltweets as soon as possible! :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}