Twitter_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Twitter API\n",
    "\n",
    "This tutorial presents an overview of how to use the Python programming language to interact with the Twitter API, both for acquiring data and for posting it. We're using the Twitter API because it's useful in its own right but also presents an interesting \"case study\" of how to work with APIs offered by commercial entities and social media sites.\n",
    "\n",
    "## About the API\n",
    "\n",
    "The Twitter API allows you to programmatically enact many of the same actions that you can perform with the Twitter app and the Twitter website, such as searching for Tweets, following users, reading your timeline, posting tweets and direct messages, etc, though there are some parts of the Twitter user experience, like polls, that are (as of this writing) unavailable through the API. You can use the API to do things like collect data from tweets and write automated agents that post to Twitter.\n",
    "\n",
    "In particular, we're going to be talking about Twitter's [REST API](https://dev.twitter.com/rest/public). (\"REST\" stands for [Representational State Transfer](https://en.wikipedia.org/wiki/Representational_state_transfer), a popular style of API design). For the kind of work we'll be doing, the [streaming API](https://dev.twitter.com/streaming/overview) is also worth a look, but left as an exercise for the reader.\n",
    "\n",
    "## Authorization\n",
    "\n",
    "All requests to the REST API—making a post, running a search, following or unfollowing an account—must be made *on behalf of a user*. Your code must be authorized to submit requests to the API on that user's behalf. A user can authorize code to act on their behalf through the medium of something called an *application*. You can see which applications you've authorized by logging into Twitter and [following this link](https://twitter.com/settings/applications).\n",
    "\n",
    "When making requests to the Twitter API, you don't use your username and password: instead you use four different unique identifiers: the Consumer (Application) Key, the Consumer (Application) Secret, an Access Token, and a Token Secret. The Consumer Key/Secret identify the application, and the Token/Token Secret identify a particular account as having access through a particular application. You don't choose these values; they're strings of random numbers automatically generated by Twitter. These strings, together, act as a sort of \"password\" for the Twitter API.\n",
    "\n",
    "In order to obtain these four magical strings, we need to...\n",
    "\n",
    "* Create a Twitter \"application,\" which is associated with a Twitter account;\n",
    "the \"API Key\" and \"API Secret\" are created with this application;\n",
    "* Create an access token and token secret;\n",
    "* Copy all four strings to use when creating Python programs that access Twitter.\n",
    "\n",
    "[This site has a good overview of the steps you need to\n",
    "perform](http://iag.me/socialmedia/how-to-create-a-twitter-app-in-8-easy-steps/)\n",
    "in order to create a Twitter application. I'll demonstrate the process in\n",
    "class. You'll need to have already signed up for a Twitter account!\n",
    "\n",
    "When you're done creating your application and getting your token for the application, assign them to the variables below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "api_key = \"\"\n",
    "api_secret = \"\"\n",
    "access_token = \"\"\n",
    "token_secret = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Twython\n",
    "\n",
    "Twitter's API operates over HTTP and returns JSON objects, and technically you could use the `requests` library (or any other HTTP client) to make requests to and receive responses from the API. However, the Twitter API uses a somewhat complicated authentication process called [Oauth](http://oauth.net/), which requires the generation of cryptographic signatures of requests in order to ensure their security. This process is a bit complicated, and not worth implementing from scratch. For this reason, most programmers making use of the Twitter API use a third-party library to do so. These libraries wrap up and abstract away the particulars of working with Oauth authentication so that programmers don't have to worry about them. As a happy side-effect, the libraries provide particular abstractions for API calls which make it slightly easier to use the API—you can just call methods with parameters instead of constructing URLs in your code \"by hand\".\n",
    "\n",
    "There are a number of different libraries for accessing the Twitter API. We're going to use one called [Twython](https://twython.readthedocs.io/en/latest/). You can install Twython with `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting twython\n",
      "  Downloading twython-3.4.0.tar.gz\n",
      "Requirement already satisfied (use --upgrade to upgrade): requests>=2.1.0 in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from twython)\n",
      "Collecting requests-oauthlib>=0.4.0 (from twython)\n",
      "  Downloading requests_oauthlib-0.6.1-py2.py3-none-any.whl\n",
      "Collecting oauthlib>=0.6.2 (from requests-oauthlib>=0.4.0->twython)\n",
      "  Downloading oauthlib-1.1.2.tar.gz (111kB)\n",
      "\u001b[K    100% |████████████████████████████████| 114kB 2.0MB/s \n",
      "\u001b[?25hInstalling collected packages: oauthlib, requests-oauthlib, twython\n",
      "  Running setup.py install for oauthlib\n",
      "  Running setup.py install for twython\n",
      "Successfully installed oauthlib-1.1.2 requests-oauthlib-0.6.1 twython-3.4.0\n",
      "\u001b[33mYou are using pip version 7.1.2, however version 8.1.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip3 install twython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching Twitter\n",
    "\n",
    "Here's our first example Twython snippet, which uses the `search` resource to find tweets that match a particular search term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['#BigData Triggers #Predictive #Journalism https://t.co/Jyjno5Rcvy #journalists',\n",
       " 'Data Journalism: A Simple Guide for Marketers https://t.co/w9uHCXtAZ9 https://t.co/DMImBSLTfE',\n",
       " 'The latest Data Journalism! https://t.co/moNhOKnSPs Thanks to @TheInsightBee @vikesverkko #journalism #datajournalism',\n",
       " \"Biggest leak in the history of data journalism just went live, and it's about Cards Against Humanity.\",\n",
       " 'Newspapers And Data: Good For Outreach, Bad For Journalism? https://t.co/SkhxpPt4q1 #DigiAsia #DigitalPublishing https://t.co/leYrvl1XUS',\n",
       " 'RT @dinocitraro: This Digital Humanities + Data Journalism Symposium looks fantastic: https://t.co/uKVElmNdhW More info here: https://t.co/…',\n",
       " 'Data Journalism Awards celebrate evidence-based questioning in our society https://t.co/jToiOpgKxq @euroscientist https://t.co/YIIEf3vKd5',\n",
       " 'RT @josephwillits: Red dots of bloodshed &amp; death. Harrowing map of every car bomb in Baghdad since 2003 https://t.co/t8TcgWqMI5 https://t.c…',\n",
       " 'RT @a_m_papagiotis: How one Mexican data team uncovered the story of 4,000 missing women https://t.co/vwALimBe7q via @wordpressdotcom',\n",
       " 'Data Generation Gap: Younger IT Workers Believe The Hype - InformationWeek https://t.co/E7AaFEj123 https://t.co/DEVrvZwjtZ',\n",
       " 'Journalism in the Age of Data: A Video Report on Data Visualization by Geoff McGhee... https://t.co/nEiOoaSXs3',\n",
       " 'Data Journalism: A Simple Guide for Marketers https://t.co/5Ju2NpNrCd',\n",
       " 'Data Journalism: A Simple Guide for Marketers https://t.co/dUDT2qEWWV',\n",
       " 'RT @GrowthMob: Data Journalism: A Simple Guide for Marketers https://t.co/jOtB1WCDgX via @hubspot @DholakiyaPratik #Startup #Marketing #Gro…',\n",
       " 'RT @GrowthMob: Data Journalism: A Simple Guide for Marketers https://t.co/jOtB1WCDgX via @hubspot @DholakiyaPratik #Startup #Marketing #Gro…',\n",
       " 'RT @GrowthMob: Data Journalism: A Simple Guide for Marketers https://t.co/jOtB1WCDgX via @hubspot @DholakiyaPratik #Startup #Marketing #Gro…',\n",
       " 'Data Journalism: A Simple Guide for Marketers https://t.co/jOtB1WCDgX via @hubspot @DholakiyaPratik #Startup #Marketing #GrowthHacking',\n",
       " 'RT @HamdiCheb: @NassiraELM https://t.co/RRBfbS6ZXE',\n",
       " 'RT @a_m_papagiotis: How one Mexican data team uncovered the story of 4,000 missing women https://t.co/vwALimBe7q via @wordpressdotcom',\n",
       " 'How one Mexican data team uncovered the story of 4,000 missing women https://t.co/vwALimBe7q via @wordpressdotcom']"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import twython\n",
    "\n",
    "# create a Twython object by passing the necessary secret passwords\n",
    "twitter = twython.Twython(api_key, api_secret, access_token, token_secret)\n",
    "\n",
    "response = twitter.search(q=\"data journalism\", result_type=\"recent\", count=20)\n",
    "[r['text'] for r in response['statuses']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.search()` method performs a Twitter search, just as though you'd gone to [the Twitter search page](https://twitter.com/search-home) and typed in your query. The method returns a JSON object, which Twython converts to a dictionary for us. This dictionary contains a number of items; importantly, the value for the key `statuses` is a list of tweets that match the search term. Let's look at the underlying response in detail. I'm going to run the search again, this time asking for only two results instead of twenty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'search_metadata': {'completed_in': 0.05,\n",
       "  'count': 2,\n",
       "  'max_id': 750060071153958912,\n",
       "  'max_id_str': '750060071153958912',\n",
       "  'next_results': '?max_id=750057453123960831&q=data%20journalism&count=2&include_entities=1&result_type=recent',\n",
       "  'query': 'data+journalism',\n",
       "  'refresh_url': '?since_id=750060071153958912&q=data%20journalism&result_type=recent&include_entities=1',\n",
       "  'since_id': 0,\n",
       "  'since_id_str': '0'},\n",
       " 'statuses': [{'contributors': None,\n",
       "   'coordinates': None,\n",
       "   'created_at': 'Mon Jul 04 20:13:51 +0000 2016',\n",
       "   'entities': {'hashtags': [],\n",
       "    'media': [{'display_url': 'pic.twitter.com/V44EbUvvVe',\n",
       "      'expanded_url': 'http://twitter.com/OrenKessler/status/749729760293511168/photo/1',\n",
       "      'id': 749729753117044736,\n",
       "      'id_str': '749729753117044736',\n",
       "      'indices': [111, 134],\n",
       "      'media_url': 'http://pbs.twimg.com/media/CmeTPPwVMAA8Tgn.jpg',\n",
       "      'media_url_https': 'https://pbs.twimg.com/media/CmeTPPwVMAA8Tgn.jpg',\n",
       "      'sizes': {'large': {'h': 1334, 'resize': 'fit', 'w': 750},\n",
       "       'medium': {'h': 1200, 'resize': 'fit', 'w': 675},\n",
       "       'small': {'h': 680, 'resize': 'fit', 'w': 382},\n",
       "       'thumb': {'h': 150, 'resize': 'crop', 'w': 150}},\n",
       "      'source_status_id': 749729760293511168,\n",
       "      'source_status_id_str': '749729760293511168',\n",
       "      'source_user_id': 18101696,\n",
       "      'source_user_id_str': '18101696',\n",
       "      'type': 'photo',\n",
       "      'url': 'https://t.co/V44EbUvvVe'}],\n",
       "    'symbols': [],\n",
       "    'urls': [{'display_url': 'theguardian.com/news/datablog/…',\n",
       "      'expanded_url': 'https://www.theguardian.com/news/datablog/2010/oct/23/wikileaks-iraq-data-journalism',\n",
       "      'indices': [87, 110],\n",
       "      'url': 'https://t.co/lr3wD1YVGe'}],\n",
       "    'user_mentions': [{'id': 18101696,\n",
       "      'id_str': '18101696',\n",
       "      'indices': [3, 15],\n",
       "      'name': 'Oren Kessler',\n",
       "      'screen_name': 'OrenKessler'}]},\n",
       "   'favorite_count': 0,\n",
       "   'favorited': False,\n",
       "   'geo': None,\n",
       "   'id': 750060071153958912,\n",
       "   'id_str': '750060071153958912',\n",
       "   'in_reply_to_screen_name': None,\n",
       "   'in_reply_to_status_id': None,\n",
       "   'in_reply_to_status_id_str': None,\n",
       "   'in_reply_to_user_id': None,\n",
       "   'in_reply_to_user_id_str': None,\n",
       "   'is_quote_status': False,\n",
       "   'lang': 'en',\n",
       "   'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},\n",
       "   'place': None,\n",
       "   'possibly_sensitive': False,\n",
       "   'retweet_count': 33,\n",
       "   'retweeted': False,\n",
       "   'retweeted_status': {'contributors': None,\n",
       "    'coordinates': None,\n",
       "    'created_at': 'Sun Jul 03 22:21:18 +0000 2016',\n",
       "    'entities': {'hashtags': [],\n",
       "     'media': [{'display_url': 'pic.twitter.com/V44EbUvvVe',\n",
       "       'expanded_url': 'http://twitter.com/OrenKessler/status/749729760293511168/photo/1',\n",
       "       'id': 749729753117044736,\n",
       "       'id_str': '749729753117044736',\n",
       "       'indices': [94, 117],\n",
       "       'media_url': 'http://pbs.twimg.com/media/CmeTPPwVMAA8Tgn.jpg',\n",
       "       'media_url_https': 'https://pbs.twimg.com/media/CmeTPPwVMAA8Tgn.jpg',\n",
       "       'sizes': {'large': {'h': 1334, 'resize': 'fit', 'w': 750},\n",
       "        'medium': {'h': 1200, 'resize': 'fit', 'w': 675},\n",
       "        'small': {'h': 680, 'resize': 'fit', 'w': 382},\n",
       "        'thumb': {'h': 150, 'resize': 'crop', 'w': 150}},\n",
       "       'type': 'photo',\n",
       "       'url': 'https://t.co/V44EbUvvVe'}],\n",
       "     'symbols': [],\n",
       "     'urls': [{'display_url': 'theguardian.com/news/datablog/…',\n",
       "       'expanded_url': 'https://www.theguardian.com/news/datablog/2010/oct/23/wikileaks-iraq-data-journalism',\n",
       "       'indices': [70, 93],\n",
       "       'url': 'https://t.co/lr3wD1YVGe'}],\n",
       "     'user_mentions': []},\n",
       "    'favorite_count': 14,\n",
       "    'favorited': False,\n",
       "    'geo': None,\n",
       "    'id': 749729760293511168,\n",
       "    'id_str': '749729760293511168',\n",
       "    'in_reply_to_screen_name': None,\n",
       "    'in_reply_to_status_id': None,\n",
       "    'in_reply_to_status_id_str': None,\n",
       "    'in_reply_to_user_id': None,\n",
       "    'in_reply_to_user_id_str': None,\n",
       "    'is_quote_status': False,\n",
       "    'lang': 'en',\n",
       "    'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},\n",
       "    'place': {'attributes': {},\n",
       "     'bounding_box': {'coordinates': [[[-77.119401, 38.801826],\n",
       "        [-76.909396, 38.801826],\n",
       "        [-76.909396, 38.9953797],\n",
       "        [-77.119401, 38.9953797]]],\n",
       "      'type': 'Polygon'},\n",
       "     'contained_within': [],\n",
       "     'country': 'United States',\n",
       "     'country_code': 'US',\n",
       "     'full_name': 'Washington, DC',\n",
       "     'id': '01fbe706f872cb32',\n",
       "     'name': 'Washington',\n",
       "     'place_type': 'city',\n",
       "     'url': 'https://api.twitter.com/1.1/geo/id/01fbe706f872cb32.json'},\n",
       "    'possibly_sensitive': False,\n",
       "    'retweet_count': 33,\n",
       "    'retweeted': False,\n",
       "    'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>',\n",
       "    'text': 'These maps showing every attack in Baghdad since 2003 are just insane https://t.co/lr3wD1YVGe https://t.co/V44EbUvvVe',\n",
       "    'truncated': False,\n",
       "    'user': {'contributors_enabled': False,\n",
       "     'created_at': 'Sat Dec 13 17:41:53 +0000 2008',\n",
       "     'default_profile': False,\n",
       "     'default_profile_image': False,\n",
       "     'description': 'Deputy Director for Research and Research Fellow at Foundation for Defense of Democracies. Lots of Egypt. RTs may be the opposite of endorsements.',\n",
       "     'entities': {'description': {'urls': []},\n",
       "      'url': {'urls': [{'display_url': 'defenddemocracy.org/about-fdd/team…',\n",
       "         'expanded_url': 'http://www.defenddemocracy.org/about-fdd/team-overview/oren-kessler/',\n",
       "         'indices': [0, 23],\n",
       "         'url': 'https://t.co/pncCL0oUws'}]}},\n",
       "     'favourites_count': 4867,\n",
       "     'follow_request_sent': False,\n",
       "     'followers_count': 11927,\n",
       "     'following': False,\n",
       "     'friends_count': 5112,\n",
       "     'geo_enabled': True,\n",
       "     'has_extended_profile': False,\n",
       "     'id': 18101696,\n",
       "     'id_str': '18101696',\n",
       "     'is_translation_enabled': False,\n",
       "     'is_translator': False,\n",
       "     'lang': 'en',\n",
       "     'listed_count': 486,\n",
       "     'location': 'Washington DC',\n",
       "     'name': 'Oren Kessler',\n",
       "     'notifications': False,\n",
       "     'profile_background_color': 'C0DEED',\n",
       "     'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "     'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "     'profile_background_tile': False,\n",
       "     'profile_banner_url': 'https://pbs.twimg.com/profile_banners/18101696/1459049086',\n",
       "     'profile_image_url': 'http://pbs.twimg.com/profile_images/713926571204218880/UY_SCAqx_normal.jpg',\n",
       "     'profile_image_url_https': 'https://pbs.twimg.com/profile_images/713926571204218880/UY_SCAqx_normal.jpg',\n",
       "     'profile_link_color': '1B95E0',\n",
       "     'profile_sidebar_border_color': 'C0DEED',\n",
       "     'profile_sidebar_fill_color': 'DDEEF6',\n",
       "     'profile_text_color': '333333',\n",
       "     'profile_use_background_image': True,\n",
       "     'protected': False,\n",
       "     'screen_name': 'OrenKessler',\n",
       "     'statuses_count': 37486,\n",
       "     'time_zone': 'Eastern Time (US & Canada)',\n",
       "     'url': 'https://t.co/pncCL0oUws',\n",
       "     'utc_offset': -14400,\n",
       "     'verified': False}},\n",
       "   'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>',\n",
       "   'text': 'RT @OrenKessler: These maps showing every attack in Baghdad since 2003 are just insane https://t.co/lr3wD1YVGe https://t.co/V44EbUvvVe',\n",
       "   'truncated': False,\n",
       "   'user': {'contributors_enabled': False,\n",
       "    'created_at': 'Thu Oct 28 16:29:17 +0000 2010',\n",
       "    'default_profile': True,\n",
       "    'default_profile_image': False,\n",
       "    'description': 'Business communications pro, frmr #potus speechwriter & defender of free enterprise. Dad w/daughters. Happiest when I run more than I drive.',\n",
       "    'entities': {'description': {'urls': []}},\n",
       "    'favourites_count': 500,\n",
       "    'follow_request_sent': False,\n",
       "    'followers_count': 506,\n",
       "    'following': False,\n",
       "    'friends_count': 525,\n",
       "    'geo_enabled': False,\n",
       "    'has_extended_profile': False,\n",
       "    'id': 209153772,\n",
       "    'id_str': '209153772',\n",
       "    'is_translation_enabled': False,\n",
       "    'is_translator': False,\n",
       "    'lang': 'en',\n",
       "    'listed_count': 15,\n",
       "    'location': 'DC/MD',\n",
       "    'name': 'Noam Neusner',\n",
       "    'notifications': False,\n",
       "    'profile_background_color': 'C0DEED',\n",
       "    'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "    'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "    'profile_background_tile': False,\n",
       "    'profile_banner_url': 'https://pbs.twimg.com/profile_banners/209153772/1353293142',\n",
       "    'profile_image_url': 'http://pbs.twimg.com/profile_images/2987636674/0dfa99cd800590566ae8c1ab3ca34d2e_normal.jpeg',\n",
       "    'profile_image_url_https': 'https://pbs.twimg.com/profile_images/2987636674/0dfa99cd800590566ae8c1ab3ca34d2e_normal.jpeg',\n",
       "    'profile_link_color': '0084B4',\n",
       "    'profile_sidebar_border_color': 'C0DEED',\n",
       "    'profile_sidebar_fill_color': 'DDEEF6',\n",
       "    'profile_text_color': '333333',\n",
       "    'profile_use_background_image': True,\n",
       "    'protected': False,\n",
       "    'screen_name': 'NoamNeusner',\n",
       "    'statuses_count': 3208,\n",
       "    'time_zone': 'Eastern Time (US & Canada)',\n",
       "    'url': None,\n",
       "    'utc_offset': -14400,\n",
       "    'verified': False}},\n",
       "  {'contributors': None,\n",
       "   'coordinates': None,\n",
       "   'created_at': 'Mon Jul 04 20:03:26 +0000 2016',\n",
       "   'entities': {'hashtags': [{'indices': [98, 106], 'text': 'twitter'},\n",
       "     {'indices': [107, 116], 'text': 'opendata'}],\n",
       "    'symbols': [],\n",
       "    'urls': [{'display_url': 'paper.li/postoditacco/1…',\n",
       "      'expanded_url': 'http://paper.li/postoditacco/1381694616?edition_id=5d93afe0-4222-11e6-bfc9-0cc47a0d1609',\n",
       "      'indices': [34, 57],\n",
       "      'url': 'https://t.co/dkM1nYl3aQ'}],\n",
       "    'user_mentions': [{'id': 239362738,\n",
       "      'id_str': '239362738',\n",
       "      'indices': [68, 79],\n",
       "      'name': 'numeroteca',\n",
       "      'screen_name': 'numeroteca'},\n",
       "     {'id': 2234881,\n",
       "      'id_str': '2234881',\n",
       "      'indices': [80, 87],\n",
       "      'name': 'Mindy McAdams',\n",
       "      'screen_name': 'macloo'},\n",
       "     {'id': 16911276,\n",
       "      'id_str': '16911276',\n",
       "      'indices': [88, 97],\n",
       "      'name': 'Timetric',\n",
       "      'screen_name': 'timetric'}]},\n",
       "   'favorite_count': 0,\n",
       "   'favorited': False,\n",
       "   'geo': None,\n",
       "   'id': 750057453123960832,\n",
       "   'id_str': '750057453123960832',\n",
       "   'in_reply_to_screen_name': None,\n",
       "   'in_reply_to_status_id': None,\n",
       "   'in_reply_to_status_id_str': None,\n",
       "   'in_reply_to_user_id': None,\n",
       "   'in_reply_to_user_id_str': None,\n",
       "   'is_quote_status': False,\n",
       "   'lang': 'en',\n",
       "   'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},\n",
       "   'place': None,\n",
       "   'possibly_sensitive': False,\n",
       "   'retweet_count': 0,\n",
       "   'retweeted': False,\n",
       "   'source': '<a href=\"http://paper.li\" rel=\"nofollow\">Paper.li</a>',\n",
       "   'text': 'The latest Data Journalism Daily! https://t.co/dkM1nYl3aQ Thanks to @numeroteca @macloo @timetric #twitter #opendata',\n",
       "   'truncated': False,\n",
       "   'user': {'contributors_enabled': False,\n",
       "    'created_at': 'Wed Nov 07 00:44:46 +0000 2007',\n",
       "    'default_profile': False,\n",
       "    'default_profile_image': False,\n",
       "    'description': 'Webaholic. Fact-checker. Storyteller. Data lover. #socialmedia & #communication inside (tm).\\nAlso working on BI and ADV for an international publisher',\n",
       "    'entities': {'description': {'urls': []},\n",
       "     'url': {'urls': [{'display_url': 'myweb20.it',\n",
       "        'expanded_url': 'http://www.myweb20.it',\n",
       "        'indices': [0, 22],\n",
       "        'url': 'http://t.co/8mvOl9RvlK'}]}},\n",
       "    'favourites_count': 2648,\n",
       "    'follow_request_sent': False,\n",
       "    'followers_count': 2515,\n",
       "    'following': False,\n",
       "    'friends_count': 1942,\n",
       "    'geo_enabled': True,\n",
       "    'has_extended_profile': False,\n",
       "    'id': 10016342,\n",
       "    'id_str': '10016342',\n",
       "    'is_translation_enabled': False,\n",
       "    'is_translator': False,\n",
       "    'lang': 'it',\n",
       "    'listed_count': 304,\n",
       "    'location': 'Damascus, Syria',\n",
       "    'name': 'Roberto ＼(＾o＾)／',\n",
       "    'notifications': False,\n",
       "    'profile_background_color': 'FFFFFF',\n",
       "    'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/771220795/d8f23f4028338de980af394548ba6953.jpeg',\n",
       "    'profile_background_image_url_https': 'https://pbs.twimg.com/profile_background_images/771220795/d8f23f4028338de980af394548ba6953.jpeg',\n",
       "    'profile_background_tile': True,\n",
       "    'profile_banner_url': 'https://pbs.twimg.com/profile_banners/10016342/1405752496',\n",
       "    'profile_image_url': 'http://pbs.twimg.com/profile_images/378800000703172715/8d3d40317dc63a86093683cfd650aa9e_normal.jpeg',\n",
       "    'profile_image_url_https': 'https://pbs.twimg.com/profile_images/378800000703172715/8d3d40317dc63a86093683cfd650aa9e_normal.jpeg',\n",
       "    'profile_link_color': '1146E8',\n",
       "    'profile_sidebar_border_color': 'FFFFFF',\n",
       "    'profile_sidebar_fill_color': 'FFFFFF',\n",
       "    'profile_text_color': '342135',\n",
       "    'profile_use_background_image': True,\n",
       "    'protected': False,\n",
       "    'screen_name': 'postoditacco',\n",
       "    'statuses_count': 28727,\n",
       "    'time_zone': 'Rome',\n",
       "    'url': 'http://t.co/8mvOl9RvlK',\n",
       "    'utc_offset': 7200,\n",
       "    'verified': False}}]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = twitter.search(q=\"data journalism\", result_type=\"recent\", count=2)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there's a lot of stuff in here, even for just two tweets. In the top-level dictionary, there's a key `search_metadata` whose value is a dictionary with, well, metadata about the search: how many results it returned, what the query was, and what URL to use to get the next page of results. The value for the `statuses` key is a list of dictionaries, each of which contains information about the matching tweets. Tweets are limited to 140 characters, but have *much more* than 140 characters of metadata. [Twitter has a good guide to what each of the fields mean here](https://dev.twitter.com/overview/api/tweets), but here are the most interesting key/value pairs from our perspective:\n",
    "\n",
    "* `id_str`: the unique numerical ID of the tweet\n",
    "* `in_reply_to_status_id_str`: the ID of the tweet that this tweet is a reply to, if applicable\n",
    "* `retweet_count`: number of times this tweet has been retweeted\n",
    "* `retweet_status`: the tweet that this tweet is a retweet of, if applicable\n",
    "* `favorite_count`: the number of times that this tweet has been favorited\n",
    "* `text`: the actual text of the tweet\n",
    "* `user`: a dictionary with information on the user who wrote the tweet, including the `screen_name` key which has the Twitter screen name of the user\n",
    "\n",
    "> NOTE: You can do much more with the query than just search for raw strings. [The \"Query operators\" section on this page](https://dev.twitter.com/rest/public/search) shows the different bits of syntax you can use to make your query more expressive.\n",
    "\n",
    "You can form the URL of a particular tweet by combining the tweet's ID and the user's screen name using the following function. (This is helpful if you want to view the tweet in a web browser.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tweet_url(tweet):\n",
    "    return 'https://twitter.com/' + \\\n",
    "        tweet['user']['screen_name'] + \\\n",
    "        \"/statuses/\" + \\\n",
    "        tweet['id_str']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://twitter.com/postoditacco/statuses/750057453123960832'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet_url(response['statuses'][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twython, the REST API, and method parameters\n",
    "\n",
    "In general, Twython has one method for every \"endpoint\" in the Twitter REST API. Usually the Twython method has a name that resembles or is identical to the corresponding URL in the REST API. [The Twython API documentation](https://twython.readthedocs.io/en/latest/api.html) lists the available methods and which parts of the Twitter API they map to.\n",
    "\n",
    "As a means of becoming more familiar with this, let's dig a bit deeper into search. The `.search()` method of Twython takes a number of different parameters, which match up with the query string parameters of the REST API's `search/tweets` endpoint [as documented here](https://dev.twitter.com/rest/reference/get/search/tweets). Every parameter that can be specified on the query string in a REST API call can also be included as a named parameter in the call to Twython's `.search()` method. The preceding examples already show some examples of this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "response = twitter.search(q=\"data journalism\", result_type=\"recent\", count=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This call to `.search()` includes the parameters `q` (which specifies the search query), `result_type` (which can be set to either `popular`, `recent` or `mixed`, depending on how you want results to be returned) and `count` (which specifies how many tweets you want returned in the response, with an upper limit of 100). Looking at the documentation, it appears that there's another interesting parameter we could play with: the `geocode` parameter, which will make our search respond with tweets only within a given radius of a particular latitude/longitude. Let's use this to find the screen names of people tweeting about data journalism within a few miles of Columbia University:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sash_Marguerite\n",
      "RealRakhmetov\n",
      "pminozzo\n",
      "hscott61\n",
      "DickensOlewe\n",
      "cunyjschool\n"
     ]
    }
   ],
   "source": [
    "response = twitter.search(q=\"data journalism\",\n",
    "                          result_type=\"recent\",\n",
    "                          count=100,\n",
    "                          geocode=\"40.807511,-73.963265,4mi\")\n",
    "for resp in response['statuses']:\n",
    "    print(resp['user']['screen_name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting a user's timeline\n",
    "\n",
    "The Twitter API provides an endpoint for fetching the tweets of a particular user. The endpoint in the API for this is [statuses/user_timeline](https://dev.twitter.com/rest/reference/get/statuses/user_timeline) and the function in Twython is [.get_user_timeline()](https://twython.readthedocs.io/en/latest/api.html#twython.Twython.get_user_timeline). This function looks a lot like `.search()` in the shape of its response. Let's try it out, fetching the last few tweets of the [Twitter account of the Columbia Journalism School](https://twitter.com/columbiajourn)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final hour of giving for the Annual Fund: https://t.co/txFzXZ7rHI\n",
      "You're on deadline for the Annual Fund, 14 hours (EST) remain: https://t.co/txFzXZ7rHI https://t.co/PVbCl5ve9E\n",
      "It's not too late to contribute to the future of journalism through the Annual Fund. https://t.co/txFzXZ7rHI https://t.co/YsHB34WQpz\n",
      "#cjs59 Paul Zimmerman, aka \"Dr. Z,\" writes about his Jschool experience in this memoir excerpt: https://t.co/d1PT4m0f36 @SInow\n",
      "For #WVAfloods coverage, see @NBCNews #cjs12 @MorganRadford https://t.co/nKEtr8KsXn\n",
      "Congrats to @MagnumPhotos Nominee &amp; #cjs10 @dianamarkosian. Her work explores her Armenian heritage &amp; memory https://t.co/Rug8kUh0TC\n",
      "2/2 #canalampliado included grad contributors \n",
      "Matteo Lonardi, @aglorios, @SarahLJorgensen &amp; @simko_bednarski  \n",
      "https://t.co/uVXWsQ3z28\n",
      "1/2 Adj Prof Walt Bogdanich, #cjs15 @jacwil28 &amp; @agMendezPty co-authored @nytimes article on #canalampliado\n",
      " https://t.co/uVXWsQ3z28\n"
     ]
    }
   ],
   "source": [
    "response = twitter.get_user_timeline(screen_name='columbiajourn',\n",
    "                                     count=20,\n",
    "                                     include_rts=False,\n",
    "                                     exclude_replies=True)\n",
    "for item in response:\n",
    "    print(item['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `screen_name` parameter specifies whose timeline we want; the `count` parameter specifies how many tweets we want from that account. The `include_rts` and `exclude_replies` parameters control whether or not particular kinds of tweets are included in the response: setting `include_rts` to `False` means we don't see any retweets in our results, while the `exclude_replies` parameter set to `True` means we don't see any tweets that are replies to other users. (According to the API documentation, \"Using `exclude_replies` with the `count` parameter will mean you will receive up-to count tweets — this is because the count parameter retrieves that many tweets before filtering out retweets and replies,\" which is why asking for 20 tweets doesn't necessarily *return* 20 tweets in this case.)\n",
    "\n",
    "Note that the `.get_user_timeline()` function returns not a dictionary with a key whose value is a list of tweets, like the `.search()` function. Instead, it simply returns a JSON list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'contributors': None,\n",
       "  'coordinates': None,\n",
       "  'created_at': 'Sat Jul 02 12:46:34 +0000 2016',\n",
       "  'entities': {'hashtags': [{'indices': [43, 50], 'text': 'serial'}],\n",
       "   'symbols': [],\n",
       "   'urls': [{'display_url': 'twitter.com/vanityfair/sta…',\n",
       "     'expanded_url': 'https://twitter.com/vanityfair/status/749196140084690944',\n",
       "     'indices': [139, 140],\n",
       "     'url': 'https://t.co/tf8e3J3mVS'}],\n",
       "   'user_mentions': [{'id': 295652602,\n",
       "     'id_str': '295652602',\n",
       "     'indices': [3, 14],\n",
       "     'name': 'Mirta Ojito',\n",
       "     'screen_name': 'MirtaOjito'},\n",
       "    {'id': 16369150,\n",
       "     'id_str': '16369150',\n",
       "     'indices': [85, 99],\n",
       "     'name': 'Columbia Journalism',\n",
       "     'screen_name': 'columbiajourn'},\n",
       "    {'id': 15704275,\n",
       "     'id_str': '15704275',\n",
       "     'indices': [110, 122],\n",
       "     'name': 'danachivvis',\n",
       "     'screen_name': 'danachivvis'}]},\n",
       "  'favorite_count': 0,\n",
       "  'favorited': False,\n",
       "  'geo': None,\n",
       "  'id': 749222732689092608,\n",
       "  'id_str': '749222732689092608',\n",
       "  'in_reply_to_screen_name': None,\n",
       "  'in_reply_to_status_id': None,\n",
       "  'in_reply_to_status_id_str': None,\n",
       "  'in_reply_to_user_id': None,\n",
       "  'in_reply_to_user_id_str': None,\n",
       "  'is_quote_status': True,\n",
       "  'lang': 'en',\n",
       "  'place': None,\n",
       "  'possibly_sensitive': False,\n",
       "  'quoted_status_id': 749196140084690944,\n",
       "  'quoted_status_id_str': '749196140084690944',\n",
       "  'retweet_count': 2,\n",
       "  'retweeted': False,\n",
       "  'retweeted_status': {'contributors': None,\n",
       "   'coordinates': None,\n",
       "   'created_at': 'Sat Jul 02 12:24:01 +0000 2016',\n",
       "   'entities': {'hashtags': [{'indices': [27, 34], 'text': 'serial'}],\n",
       "    'symbols': [],\n",
       "    'urls': [{'display_url': 'twitter.com/vanityfair/sta…',\n",
       "      'expanded_url': 'https://twitter.com/vanityfair/status/749196140084690944',\n",
       "      'indices': [117, 140],\n",
       "      'url': 'https://t.co/tf8e3J3mVS'}],\n",
       "    'user_mentions': [{'id': 16369150,\n",
       "      'id_str': '16369150',\n",
       "      'indices': [69, 83],\n",
       "      'name': 'Columbia Journalism',\n",
       "      'screen_name': 'columbiajourn'},\n",
       "     {'id': 15704275,\n",
       "      'id_str': '15704275',\n",
       "      'indices': [94, 106],\n",
       "      'name': 'danachivvis',\n",
       "      'screen_name': 'danachivvis'}]},\n",
       "   'favorite_count': 7,\n",
       "   'favorited': False,\n",
       "   'geo': None,\n",
       "   'id': 749217061264560128,\n",
       "   'id_str': '749217061264560128',\n",
       "   'in_reply_to_screen_name': None,\n",
       "   'in_reply_to_status_id': None,\n",
       "   'in_reply_to_status_id_str': None,\n",
       "   'in_reply_to_user_id': None,\n",
       "   'in_reply_to_user_id_str': None,\n",
       "   'is_quote_status': True,\n",
       "   'lang': 'en',\n",
       "   'place': None,\n",
       "   'possibly_sensitive': False,\n",
       "   'quoted_status': {'contributors': None,\n",
       "    'coordinates': None,\n",
       "    'created_at': 'Sat Jul 02 11:00:53 +0000 2016',\n",
       "    'entities': {'hashtags': [{'indices': [0, 7], 'text': 'Serial'}],\n",
       "     'media': [{'display_url': 'pic.twitter.com/UdNROUbTWa',\n",
       "       'expanded_url': 'http://twitter.com/VanityFair/status/749196140084690944/photo/1',\n",
       "       'id': 749196137656180737,\n",
       "       'id_str': '749196137656180737',\n",
       "       'indices': [81, 104],\n",
       "       'media_url': 'http://pbs.twimg.com/media/CmWt6vQXYAE5EsB.jpg',\n",
       "       'media_url_https': 'https://pbs.twimg.com/media/CmWt6vQXYAE5EsB.jpg',\n",
       "       'sizes': {'large': {'h': 960, 'resize': 'fit', 'w': 1440},\n",
       "        'medium': {'h': 800, 'resize': 'fit', 'w': 1200},\n",
       "        'small': {'h': 453, 'resize': 'fit', 'w': 680},\n",
       "        'thumb': {'h': 150, 'resize': 'crop', 'w': 150}},\n",
       "       'type': 'photo',\n",
       "       'url': 'https://t.co/UdNROUbTWa'}],\n",
       "     'symbols': [],\n",
       "     'urls': [{'display_url': 'vntyfr.com/43mFxcm',\n",
       "       'expanded_url': 'http://vntyfr.com/43mFxcm',\n",
       "       'indices': [57, 80],\n",
       "       'url': 'https://t.co/cZiTmFiyXV'}],\n",
       "     'user_mentions': []},\n",
       "    'extended_entities': {'media': [{'display_url': 'pic.twitter.com/UdNROUbTWa',\n",
       "       'expanded_url': 'http://twitter.com/VanityFair/status/749196140084690944/photo/1',\n",
       "       'id': 749196137656180737,\n",
       "       'id_str': '749196137656180737',\n",
       "       'indices': [81, 104],\n",
       "       'media_url': 'http://pbs.twimg.com/media/CmWt6vQXYAE5EsB.jpg',\n",
       "       'media_url_https': 'https://pbs.twimg.com/media/CmWt6vQXYAE5EsB.jpg',\n",
       "       'sizes': {'large': {'h': 960, 'resize': 'fit', 'w': 1440},\n",
       "        'medium': {'h': 800, 'resize': 'fit', 'w': 1200},\n",
       "        'small': {'h': 453, 'resize': 'fit', 'w': 680},\n",
       "        'thumb': {'h': 150, 'resize': 'crop', 'w': 150}},\n",
       "       'type': 'photo',\n",
       "       'url': 'https://t.co/UdNROUbTWa'}]},\n",
       "    'favorite_count': 43,\n",
       "    'favorited': False,\n",
       "    'geo': None,\n",
       "    'id': 749196140084690944,\n",
       "    'id_str': '749196140084690944',\n",
       "    'in_reply_to_screen_name': None,\n",
       "    'in_reply_to_status_id': None,\n",
       "    'in_reply_to_status_id_str': None,\n",
       "    'in_reply_to_user_id': None,\n",
       "    'in_reply_to_user_id_str': None,\n",
       "    'is_quote_status': False,\n",
       "    'lang': 'en',\n",
       "    'place': None,\n",
       "    'possibly_sensitive': False,\n",
       "    'retweet_count': 24,\n",
       "    'retweeted': False,\n",
       "    'source': '<a href=\"http://www.socialflow.com\" rel=\"nofollow\">SocialFlow</a>',\n",
       "    'text': '#Serial star Adnan Syed is getting a new trial after all https://t.co/cZiTmFiyXV https://t.co/UdNROUbTWa',\n",
       "    'truncated': False,\n",
       "    'user': {'contributors_enabled': False,\n",
       "     'created_at': 'Mon Jun 30 14:17:35 +0000 2008',\n",
       "     'default_profile': False,\n",
       "     'default_profile_image': False,\n",
       "     'description': 'In-depth reporting, gripping narratives, and world-class photography, plus heaping doses of Oscar-blogging, royal-watching, and assorted guilty pleasures.',\n",
       "     'entities': {'description': {'urls': []},\n",
       "      'url': {'urls': [{'display_url': 'vanityfair.com',\n",
       "         'expanded_url': 'http://www.vanityfair.com',\n",
       "         'indices': [0, 22],\n",
       "         'url': 'http://t.co/THwtnNR9Wy'}]}},\n",
       "     'favourites_count': 3355,\n",
       "     'follow_request_sent': False,\n",
       "     'followers_count': 4030067,\n",
       "     'following': False,\n",
       "     'friends_count': 1066,\n",
       "     'geo_enabled': False,\n",
       "     'has_extended_profile': False,\n",
       "     'id': 15279429,\n",
       "     'id_str': '15279429',\n",
       "     'is_translation_enabled': False,\n",
       "     'is_translator': False,\n",
       "     'lang': 'en',\n",
       "     'listed_count': 18366,\n",
       "     'location': 'New York, NY',\n",
       "     'name': 'VANITY FAIR',\n",
       "     'notifications': False,\n",
       "     'profile_background_color': '000000',\n",
       "     'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/611295825776357376/bqC8Op6g.jpg',\n",
       "     'profile_background_image_url_https': 'https://pbs.twimg.com/profile_background_images/611295825776357376/bqC8Op6g.jpg',\n",
       "     'profile_background_tile': False,\n",
       "     'profile_banner_url': 'https://pbs.twimg.com/profile_banners/15279429/1434576586',\n",
       "     'profile_image_url': 'http://pbs.twimg.com/profile_images/694922349238308864/wVeLVf86_normal.jpg',\n",
       "     'profile_image_url_https': 'https://pbs.twimg.com/profile_images/694922349238308864/wVeLVf86_normal.jpg',\n",
       "     'profile_link_color': '990000',\n",
       "     'profile_sidebar_border_color': '000000',\n",
       "     'profile_sidebar_fill_color': 'DDFFCC',\n",
       "     'profile_text_color': '333333',\n",
       "     'profile_use_background_image': True,\n",
       "     'protected': False,\n",
       "     'screen_name': 'VanityFair',\n",
       "     'statuses_count': 61255,\n",
       "     'time_zone': 'Eastern Time (US & Canada)',\n",
       "     'url': 'http://t.co/THwtnNR9Wy',\n",
       "     'utc_offset': -14400,\n",
       "     'verified': True}},\n",
       "   'quoted_status_id': 749196140084690944,\n",
       "   'quoted_status_id_str': '749196140084690944',\n",
       "   'retweet_count': 2,\n",
       "   'retweeted': False,\n",
       "   'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>',\n",
       "   'text': \"If you haven't listened to #serial, do. Produced by one of my former @columbiajourn students, @danachivvis.So proud. https://t.co/tf8e3J3mVS\",\n",
       "   'truncated': False,\n",
       "   'user': {'contributors_enabled': False,\n",
       "    'created_at': 'Mon May 09 12:43:21 +0000 2011',\n",
       "    'default_profile': True,\n",
       "    'default_profile_image': False,\n",
       "    'description': 'Passionate about the news. Author, Hunting Season: Immigration and Murder in an All-American Town. Director, News Standards @Telemundo.',\n",
       "    'entities': {'description': {'urls': []}},\n",
       "    'favourites_count': 412,\n",
       "    'follow_request_sent': False,\n",
       "    'followers_count': 1249,\n",
       "    'following': False,\n",
       "    'friends_count': 503,\n",
       "    'geo_enabled': False,\n",
       "    'has_extended_profile': False,\n",
       "    'id': 295652602,\n",
       "    'id_str': '295652602',\n",
       "    'is_translation_enabled': False,\n",
       "    'is_translator': False,\n",
       "    'lang': 'en',\n",
       "    'listed_count': 65,\n",
       "    'location': 'New York, Miami',\n",
       "    'name': 'Mirta Ojito',\n",
       "    'notifications': False,\n",
       "    'profile_background_color': 'C0DEED',\n",
       "    'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "    'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "    'profile_background_tile': False,\n",
       "    'profile_banner_url': 'https://pbs.twimg.com/profile_banners/295652602/1456203979',\n",
       "    'profile_image_url': 'http://pbs.twimg.com/profile_images/1345286862/Mirta_Ojito_normal.jpg',\n",
       "    'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1345286862/Mirta_Ojito_normal.jpg',\n",
       "    'profile_link_color': '0084B4',\n",
       "    'profile_sidebar_border_color': 'C0DEED',\n",
       "    'profile_sidebar_fill_color': 'DDEEF6',\n",
       "    'profile_text_color': '333333',\n",
       "    'profile_use_background_image': True,\n",
       "    'protected': False,\n",
       "    'screen_name': 'MirtaOjito',\n",
       "    'statuses_count': 1529,\n",
       "    'time_zone': 'Eastern Time (US & Canada)',\n",
       "    'url': None,\n",
       "    'utc_offset': -14400,\n",
       "    'verified': False}},\n",
       "  'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>',\n",
       "  'text': \"RT @MirtaOjito: If you haven't listened to #serial, do. Produced by one of my former @columbiajourn students, @danachivvis.So proud. https:…\",\n",
       "  'truncated': False,\n",
       "  'user': {'contributors_enabled': False,\n",
       "   'created_at': 'Fri Sep 19 19:49:35 +0000 2008',\n",
       "   'default_profile': False,\n",
       "   'default_profile_image': False,\n",
       "   'description': 'We train students to become high-impact journalists. https://t.co/JSRYKFLGsg',\n",
       "   'entities': {'description': {'urls': [{'display_url': 'bitly.com/cjs_adm',\n",
       "       'expanded_url': 'http://bitly.com/cjs_adm',\n",
       "       'indices': [53, 76],\n",
       "       'url': 'https://t.co/JSRYKFLGsg'}]},\n",
       "    'url': {'urls': [{'display_url': 'journalism.columbia.edu',\n",
       "       'expanded_url': 'http://www.journalism.columbia.edu',\n",
       "       'indices': [0, 23],\n",
       "       'url': 'https://t.co/G5PSZbcV3Q'}]}},\n",
       "   'favourites_count': 968,\n",
       "   'follow_request_sent': False,\n",
       "   'followers_count': 41146,\n",
       "   'following': False,\n",
       "   'friends_count': 1112,\n",
       "   'geo_enabled': False,\n",
       "   'has_extended_profile': False,\n",
       "   'id': 16369150,\n",
       "   'id_str': '16369150',\n",
       "   'is_translation_enabled': False,\n",
       "   'is_translator': False,\n",
       "   'lang': 'en',\n",
       "   'listed_count': 1981,\n",
       "   'location': 'New York, NY',\n",
       "   'name': 'Columbia Journalism',\n",
       "   'notifications': False,\n",
       "   'profile_background_color': '002664',\n",
       "   'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/570414669/xokdu8996gc5omt2fa0l.jpeg',\n",
       "   'profile_background_image_url_https': 'https://pbs.twimg.com/profile_background_images/570414669/xokdu8996gc5omt2fa0l.jpeg',\n",
       "   'profile_background_tile': True,\n",
       "   'profile_banner_url': 'https://pbs.twimg.com/profile_banners/16369150/1466177743',\n",
       "   'profile_image_url': 'http://pbs.twimg.com/profile_images/378800000120489911/fd97e4435944df1284661f5f044c5574_normal.jpeg',\n",
       "   'profile_image_url_https': 'https://pbs.twimg.com/profile_images/378800000120489911/fd97e4435944df1284661f5f044c5574_normal.jpeg',\n",
       "   'profile_link_color': '0039A6',\n",
       "   'profile_sidebar_border_color': '051DF5',\n",
       "   'profile_sidebar_fill_color': 'D2D3D9',\n",
       "   'profile_text_color': '080808',\n",
       "   'profile_use_background_image': True,\n",
       "   'protected': False,\n",
       "   'screen_name': 'columbiajourn',\n",
       "   'statuses_count': 8790,\n",
       "   'time_zone': 'Eastern Time (US & Canada)',\n",
       "   'url': 'https://t.co/G5PSZbcV3Q',\n",
       "   'utc_offset': -14400,\n",
       "   'verified': False}}]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = twitter.get_user_timeline(screen_name='columbiajourn', count=1)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using cursors to fetch more results\n",
    "\n",
    "The `.search()` and `.get_user_timeline()` functions by default only return the most recent results, up to the number specified with `count` (and sometimes even fewer than that). In order to find older tweets, you need to *page through* the results. If you were doing this \"by hand,\" you would use the `max_id` or `since_id` parameters to find tweets older than the last tweet in the current result, repeating that process until you'd exhausted the results (or found as many tweets as you need). This is delicate work and thankfully Twython includes pre-built functionality to make this easier: the `.cursor()` function.\n",
    "\n",
    "The `.cursor()` function takes the function you want to page through as the first parameter, and after that the keyword parameters that you would normally pass to that function. Given this information, it can repeatedly call the given function on your behalf, going back as far as it can. The object returned from the `.cursor()` function can be used as the iterable object in a `for` loop, allowing you to iterate over all of the results. Here's an example using `.search()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cursor = twitter.cursor(twitter.search, q='\"data journalism\" -filter:retweets', count=100)\n",
    "all_text = list()\n",
    "for tweet in cursor:\n",
    "    all_text.append(tweet['text'])\n",
    "    if len(all_text) > 500: # stop after 1000 tweets\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This snippet finds 500 tweets containing the phrase `data journalism` (excluding retweets) and stores the text of those tweets in a list. We can then use this text for data analysis, like a simple word count:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['guide',\n",
       " 'simple',\n",
       " 'marketers',\n",
       " 'every',\n",
       " 'maps',\n",
       " 'death',\n",
       " \"it's\",\n",
       " 'wikileaks',\n",
       " 'just',\n",
       " 'iraq:',\n",
       " 'deaths',\n",
       " 'about',\n",
       " 'went',\n",
       " 'this',\n",
       " 'history',\n",
       " 'live,',\n",
       " 'awards',\n",
       " 'biggest',\n",
       " 'leak',\n",
       " 'gana',\n",
       " 'papers',\n",
       " '@rezahakbari',\n",
       " 'https://t.co/ipjp4v3g67',\n",
       " '2016”',\n",
       " 'from']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "import re\n",
    "c = Counter()\n",
    "for text in all_text:\n",
    "    c.update([t.lower() for t in text.split()])\n",
    "# most common ten words that have a length greater than three and aren't\n",
    "# \"data\" or \"journalism\"\n",
    "[k for k, v in c.most_common() \\\n",
    " if len(k) > 3 and not(re.search(r\"data|journalism\", k))][:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rate limits\n",
    "\n",
    "TK!\n",
    "\n",
    "## Entities\n",
    "\n",
    "TK!\n",
    "\n",
    "## Posting tweets\n",
    "\n",
    "You can also use the Twitter API to *post tweets* on behalf of a user. In this tutorial, we're going to use this ability of the API to make a simple bot.\n",
    "\n",
    "### Simple example\n",
    "\n",
    "When you first create a Twitter application, the credentials you have by default (i.e., the ones you get when you click \"Create my access token\") are for *your own user*. This means that you can post tweets to your own account using these credentials, and to your own account only. This isn't normally very desirable, but let's give it a shot, just to see how to update your status (i.e., post a tweet) with Twython. Here we go:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'contributors': None,\n",
       " 'coordinates': None,\n",
       " 'created_at': 'Tue Jul 05 03:27:25 +0000 2016',\n",
       " 'entities': {'hashtags': [], 'symbols': [], 'urls': [], 'user_mentions': []},\n",
       " 'favorite_count': 0,\n",
       " 'favorited': False,\n",
       " 'geo': None,\n",
       " 'id': 750169184496037888,\n",
       " 'id_str': '750169184496037888',\n",
       " 'in_reply_to_screen_name': None,\n",
       " 'in_reply_to_status_id': None,\n",
       " 'in_reply_to_status_id_str': None,\n",
       " 'in_reply_to_user_id': None,\n",
       " 'in_reply_to_user_id_str': None,\n",
       " 'is_quote_status': False,\n",
       " 'lang': 'en',\n",
       " 'place': None,\n",
       " 'retweet_count': 0,\n",
       " 'retweeted': False,\n",
       " 'source': '<a href=\"http://www.example.com\" rel=\"nofollow\">aparrish test</a>',\n",
       " 'text': \"This is a test tweet for a tutorial I'm going through, please ignore\",\n",
       " 'truncated': False,\n",
       " 'user': {'contributors_enabled': False,\n",
       "  'created_at': 'Fri May 23 08:27:48 +0000 2008',\n",
       "  'default_profile': True,\n",
       "  'default_profile_image': False,\n",
       "  'description': \"For all the times you want to test aparrish, there's aparrishtest.\",\n",
       "  'entities': {'description': {'urls': []}},\n",
       "  'favourites_count': 0,\n",
       "  'follow_request_sent': False,\n",
       "  'followers_count': 13,\n",
       "  'following': False,\n",
       "  'friends_count': 5,\n",
       "  'geo_enabled': False,\n",
       "  'has_extended_profile': False,\n",
       "  'id': 14879231,\n",
       "  'id_str': '14879231',\n",
       "  'is_translation_enabled': False,\n",
       "  'is_translator': False,\n",
       "  'lang': 'en',\n",
       "  'listed_count': 0,\n",
       "  'location': 'Brooklyntest',\n",
       "  'name': 'aparrishtest',\n",
       "  'notifications': False,\n",
       "  'profile_background_color': 'C0DEED',\n",
       "  'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "  'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png',\n",
       "  'profile_background_tile': False,\n",
       "  'profile_image_url': 'http://pbs.twimg.com/profile_images/420488681/tagnic8yaaWp_normal',\n",
       "  'profile_image_url_https': 'https://pbs.twimg.com/profile_images/420488681/tagnic8yaaWp_normal',\n",
       "  'profile_link_color': '0084B4',\n",
       "  'profile_sidebar_border_color': 'C0DEED',\n",
       "  'profile_sidebar_fill_color': 'DDEEF6',\n",
       "  'profile_text_color': '333333',\n",
       "  'profile_use_background_image': True,\n",
       "  'protected': False,\n",
       "  'screen_name': 'aparrishtest',\n",
       "  'statuses_count': 300,\n",
       "  'time_zone': 'Central Time (US & Canada)',\n",
       "  'url': None,\n",
       "  'utc_offset': -18000,\n",
       "  'verified': False}}"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twitter.update_status(status=\"This is a test tweet for a tutorial I'm going through, please ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your account, and you'll see that your status has been updated! (You can safely delete this tweet if you'd like.) As you can see, the `.update_status()` function takes a single named parameter, `status`, which should have a string as its value. Twitter will update your status with the given string. The function returns a dictionary with information about the tweet that was just created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authorizing another user\n",
    "\n",
    "Of course, you generally don't want to update *your own status*. You want to write a program that updates *someone else's status*, even if that someone else is a bot user of your own creation.\n",
    "\n",
    "Before you proceed, create a new Twitter account. You'll need to log out of your current account and then open up the Twitter website, or (preferably) use your browser's \"private\" or \"incognito\" functionality. Every Twitter account requires a unique e-mail address, and you'll need to have access to the e-mail address to \"verify\" your account, so make sure you have an e-mail address you can use (and check) other than the one you used for your primary Twitter account. (We'll go over this process in class.)\n",
    "\n",
    "Once you've created a new Twitter account, you'll need to have that user *authorize* the Twitter application we created earlier to tweet on its behalf. Doing this is a two-step process. Run the cell below (making sure that the `api_key` and `api_secret` variables have been set to the consumer key and consumer secret of your application, respectively), and then open the URL it prints out *while you are logged into your bot's account*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "twitter = twython.Twython(api_key, api_secret)\n",
    "auth = twitter.get_authentication_tokens()\n",
    "print(\"Log into Twitter as the user you want to authorize and visit this URL:\")\n",
    "print(\"\\t\" + auth['auth_url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the page that appears, confirm that you want to authorize the application. A PIN will appear. Paste this PIN into the cell below, as the value assigned to the variable `pin`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pin = \"\"\n",
    "\n",
    "twitter = twython.Twython(api_key, api_secret, auth['oauth_token'], auth['oauth_token_secret'])\n",
    "tokens = twitter.get_authorized_tokens(pin)\n",
    "\n",
    "new_access_token = tokens['oauth_token']\n",
    "new_token_secret = tokens['oauth_token_secret']\n",
    "print(\"your access token:\", new_access_token)\n",
    "print(\"your token secret:\", new_token_secret)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now you have an access token and token secret for your bot's account. Run the cell below to create a new Twython object authorized with these credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "twitter = twython.Twython(api_key, api_secret, new_access_token, new_token_secret)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And run the following cell to post a test tweet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'contributors': None,\n",
       " 'coordinates': None,\n",
       " 'created_at': 'Tue Jul 05 03:41:20 +0000 2016',\n",
       " 'entities': {'hashtags': [], 'symbols': [], 'urls': [], 'user_mentions': []},\n",
       " 'favorite_count': 0,\n",
       " 'favorited': False,\n",
       " 'geo': None,\n",
       " 'id': 750172685100023808,\n",
       " 'id_str': '750172685100023808',\n",
       " 'in_reply_to_screen_name': None,\n",
       " 'in_reply_to_status_id': None,\n",
       " 'in_reply_to_status_id_str': None,\n",
       " 'in_reply_to_user_id': None,\n",
       " 'in_reply_to_user_id_str': None,\n",
       " 'is_quote_status': False,\n",
       " 'lang': 'en',\n",
       " 'place': None,\n",
       " 'retweet_count': 0,\n",
       " 'retweeted': False,\n",
       " 'source': '<a href=\"http://www.example.com\" rel=\"nofollow\">aparrish test</a>',\n",
       " 'text': 'hello, world!',\n",
       " 'truncated': False,\n",
       " 'user': {'contributors_enabled': False,\n",
       "  'created_at': 'Sat Jun 16 19:54:12 +0000 2007',\n",
       "  'default_profile': False,\n",
       "  'default_profile_image': False,\n",
       "  'description': 'I write computer programs that write poetry (uh, generously construed). Recently resident @FordhamEnglish; adjunct @NYU_ITP. My bots: https://t.co/ZhhCXX5tdQ',\n",
       "  'entities': {'description': {'urls': [{'display_url': 'twitter.com/aparrish/lists…',\n",
       "      'expanded_url': 'https://twitter.com/aparrish/lists/my-bots',\n",
       "      'indices': [134, 157],\n",
       "      'url': 'https://t.co/ZhhCXX5tdQ'}]},\n",
       "   'url': {'urls': [{'display_url': 'decontextualize.com',\n",
       "      'expanded_url': 'http://www.decontextualize.com/',\n",
       "      'indices': [0, 23],\n",
       "      'url': 'https://t.co/NvexPAokK8'}]}},\n",
       "  'favourites_count': 23715,\n",
       "  'follow_request_sent': False,\n",
       "  'followers_count': 3435,\n",
       "  'following': False,\n",
       "  'friends_count': 1013,\n",
       "  'geo_enabled': False,\n",
       "  'has_extended_profile': False,\n",
       "  'id': 6857962,\n",
       "  'id_str': '6857962',\n",
       "  'is_translation_enabled': False,\n",
       "  'is_translator': False,\n",
       "  'lang': 'en',\n",
       "  'listed_count': 204,\n",
       "  'location': 'Brooklyn, NY',\n",
       "  'name': 'Allison Parrish',\n",
       "  'notifications': False,\n",
       "  'profile_background_color': '9AE4E8',\n",
       "  'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/4483699/phelpslovecraft.png',\n",
       "  'profile_background_image_url_https': 'https://pbs.twimg.com/profile_background_images/4483699/phelpslovecraft.png',\n",
       "  'profile_background_tile': True,\n",
       "  'profile_banner_url': 'https://pbs.twimg.com/profile_banners/6857962/1430626711',\n",
       "  'profile_image_url': 'http://pbs.twimg.com/profile_images/730146910892331008/V4IsS80w_normal.jpg',\n",
       "  'profile_image_url_https': 'https://pbs.twimg.com/profile_images/730146910892331008/V4IsS80w_normal.jpg',\n",
       "  'profile_link_color': '0000FF',\n",
       "  'profile_sidebar_border_color': '87BC44',\n",
       "  'profile_sidebar_fill_color': 'E0FF92',\n",
       "  'profile_text_color': '000000',\n",
       "  'profile_use_background_image': False,\n",
       "  'protected': False,\n",
       "  'screen_name': 'aparrish',\n",
       "  'statuses_count': 11771,\n",
       "  'time_zone': 'Eastern Time (US & Canada)',\n",
       "  'url': 'https://t.co/NvexPAokK8',\n",
       "  'utc_offset': -14400,\n",
       "  'verified': False}}"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twitter.update_status(status=\"hello, world!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple bot example\n",
    "\n",
    "We're now going to make a simple bot that posts tweets with information about a randomly selected lake from the MONDIAL database. First, we'll make a big list of dictionaries with all of the information from the table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "143"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pg8000\n",
    "\n",
    "lakes = list()\n",
    "\n",
    "conn = pg8000.connect(database=\"mondial\")\n",
    "cursor = conn.cursor()\n",
    "cursor.execute(\"SELECT name, area, depth, elevation, type, river FROM lake\")\n",
    "for row in cursor.fetchall():\n",
    "    lakes.append({'name': row[0],\n",
    "                 'area': row[1],\n",
    "                 'depth': row[2],\n",
    "                 'elevation': row[3],\n",
    "                 'type': row[4],\n",
    "                 'river': row[5]})\n",
    "len(lakes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following dictionary maps each column to a sentence frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sentences = {\n",
    "    'area': 'The area of {} is {} square kilometers.',\n",
    "    'depth': 'The depth of {} is {} meters.',\n",
    "    'elevation': 'The elevation of {} is {} meters.',\n",
    "    'type': 'The type of {} is \"{}.\"',\n",
    "    'river': '{} empties into a river named {}.'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell selects a random lake from the list, and a random sentence frame from the `sentences` dictionary, and attempts to fill in the frame with relevant information from the lake."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The area of Lago Maggiore is 216 square kilometers.\n",
      "The type of Lake Eyre is \"salt.\"\n",
      "Lake Burley Griffin empties into a river named Murrumbidgee River.\n",
      "The depth of Lake Eyre is 4 meters.\n",
      "The elevation of Lake Tana is 1830 meters.\n",
      "The area of Kuybyshev Reservoir is 6450 square kilometers.\n",
      "Lago di Como empties into a river named Adda.\n",
      "The depth of Lake Bosumtwi is 81 meters.\n",
      "The area of Lake Winnipesaukee is 186 square kilometers.\n",
      "The area of Caspian Sea is 386400 square kilometers.\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "def random_lake_sentence(lakes, sentences):\n",
    "    rlake = random.choice(lakes)\n",
    "    # get the keys in the dictionary whose value is not None; we'll only try to\n",
    "    # make sentences for these\n",
    "    possible_keys = [k for k, v in rlake.items() if v is not None and k != 'name']\n",
    "    rframe = random.choice(possible_keys)\n",
    "    return sentences[rframe].format(rlake['name'], rlake[rframe])\n",
    "for i in range(10):\n",
    "    print(random_lake_sentence(lakes, sentences))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now call the `.update_status()` function with the result of the random text generation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'contributors': None,\n",
       " 'coordinates': None,\n",
       " 'created_at': 'Tue Jul 05 04:02:25 +0000 2016',\n",
       " 'entities': {'hashtags': [], 'symbols': [], 'urls': [], 'user_mentions': []},\n",
       " 'favorite_count': 0,\n",
       " 'favorited': False,\n",
       " 'geo': None,\n",
       " 'id': 750177992240947200,\n",
       " 'id_str': '750177992240947200',\n",
       " 'in_reply_to_screen_name': None,\n",
       " 'in_reply_to_status_id': None,\n",
       " 'in_reply_to_status_id_str': None,\n",
       " 'in_reply_to_user_id': None,\n",
       " 'in_reply_to_user_id_str': None,\n",
       " 'is_quote_status': False,\n",
       " 'lang': 'es',\n",
       " 'place': None,\n",
       " 'retweet_count': 0,\n",
       " 'retweeted': False,\n",
       " 'source': '<a href=\"http://www.example.com\" rel=\"nofollow\">aparrish test</a>',\n",
       " 'text': 'Lago de Chapala empties into a river named Rio Lerma.',\n",
       " 'truncated': False,\n",
       " 'user': {'contributors_enabled': False,\n",
       "  'created_at': 'Sat Jun 16 19:54:12 +0000 2007',\n",
       "  'default_profile': False,\n",
       "  'default_profile_image': False,\n",
       "  'description': 'I write computer programs that write poetry (uh, generously construed). Recently resident @FordhamEnglish; adjunct @NYU_ITP. My bots: https://t.co/ZhhCXX5tdQ',\n",
       "  'entities': {'description': {'urls': [{'display_url': 'twitter.com/aparrish/lists…',\n",
       "      'expanded_url': 'https://twitter.com/aparrish/lists/my-bots',\n",
       "      'indices': [134, 157],\n",
       "      'url': 'https://t.co/ZhhCXX5tdQ'}]},\n",
       "   'url': {'urls': [{'display_url': 'decontextualize.com',\n",
       "      'expanded_url': 'http://www.decontextualize.com/',\n",
       "      'indices': [0, 23],\n",
       "      'url': 'https://t.co/NvexPAokK8'}]}},\n",
       "  'favourites_count': 23715,\n",
       "  'follow_request_sent': False,\n",
       "  'followers_count': 3435,\n",
       "  'following': False,\n",
       "  'friends_count': 1013,\n",
       "  'geo_enabled': False,\n",
       "  'has_extended_profile': False,\n",
       "  'id': 6857962,\n",
       "  'id_str': '6857962',\n",
       "  'is_translation_enabled': False,\n",
       "  'is_translator': False,\n",
       "  'lang': 'en',\n",
       "  'listed_count': 203,\n",
       "  'location': 'Brooklyn, NY',\n",
       "  'name': 'Allison Parrish',\n",
       "  'notifications': False,\n",
       "  'profile_background_color': '9AE4E8',\n",
       "  'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/4483699/phelpslovecraft.png',\n",
       "  'profile_background_image_url_https': 'https://pbs.twimg.com/profile_background_images/4483699/phelpslovecraft.png',\n",
       "  'profile_background_tile': True,\n",
       "  'profile_banner_url': 'https://pbs.twimg.com/profile_banners/6857962/1430626711',\n",
       "  'profile_image_url': 'http://pbs.twimg.com/profile_images/730146910892331008/V4IsS80w_normal.jpg',\n",
       "  'profile_image_url_https': 'https://pbs.twimg.com/profile_images/730146910892331008/V4IsS80w_normal.jpg',\n",
       "  'profile_link_color': '0000FF',\n",
       "  'profile_sidebar_border_color': '87BC44',\n",
       "  'profile_sidebar_fill_color': 'E0FF92',\n",
       "  'profile_text_color': '000000',\n",
       "  'profile_use_background_image': False,\n",
       "  'protected': False,\n",
       "  'screen_name': 'aparrish',\n",
       "  'statuses_count': 11773,\n",
       "  'time_zone': 'Eastern Time (US & Canada)',\n",
       "  'url': 'https://t.co/NvexPAokK8',\n",
       "  'utc_offset': -14400,\n",
       "  'verified': False}}"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twitter.update_status(status=random_lake_sentence(lakes, sentences))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make this into a \"bot,\" we'd need to go an extra step: move all of this code into a standalone Python script, and set a cron job to run it every so often (maybe every few hours).\n",
    "\n",
    "## Further reading\n",
    "\n",
    "TK"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}