snscrape/cli-with-python/snscrape-python-cli.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Article Notebook for Scraping Twitter Using snscrape's CLI Commands With Python\n",
    "<br>Package Github: https://github.com/JustAnotherArchivist/snscrape\n",
    "<br>This notebook will be using the development version of snscrape\n",
    "\n",
    "Article Read-Along: https://medium.com/better-programming/how-to-scrape-tweets-with-snscrape-90124ed006af\n",
    "\n",
    "### Notebook Author: Martin Beck\n",
    "<b>Information current as of November, 26th 2020</b><br>\n",
    "\n",
    "This notebook contains materials for scraping tweets from Twitter using snscrape's CLI commands with Python\n",
    "\n",
    "<b>Dependencies: </b> \n",
    "- Your <b>Python</b> version must be <b>3.8</b> or higher. The development version of snscrape will not work with Python 3.7 or lower. You can download the latest Python version [here](https://www.python.org/downloads/).\n",
    "- <b>Development version of snscrape</b>, uncomment the pip install line in the below cell to pip install in the notebook if you don't already have it.\n",
    "- <b>Pandas</b>, the dataframes allows easy manipulation and indexing of data, this is more of a preference but is what I follow in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the pip install command below if you don't already have the library\n",
    "# !pip install git+https://github.com/JustAnotherArchivist/snscrape.git\n",
    "\n",
    "# Run the below command if you don't already have Pandas\n",
    "# !pip install pandas\n",
    "\n",
    "# Imports\n",
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Query by Username\n",
    "The code below will scrape for 100 tweets by a username then provide a CSV file with Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Setting variables to be used in format string command below\n",
    "tweet_count = 100\n",
    "username = \"jack\"\n",
    "\n",
    "# Using OS library to call CLI commands in Python\n",
    "os.system(\"snscrape --jsonl --max-results {} twitter-search 'from:{}'> user-tweets.json\".format(tweet_count, username))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>outlinks</th>\n",
       "      <th>tcooutlinks</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>quoteCount</th>\n",
       "      <th>conversationId</th>\n",
       "      <th>lang</th>\n",
       "      <th>source</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>mentionedUsers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://twitter.com/jack/status/13324354308016...</td>\n",
       "      <td>2020-11-27 21:25:36+00:00</td>\n",
       "      <td>@JesseDorogusker @Square ❤️</td>\n",
       "      <td>@JesseDorogusker @Square ❤️</td>\n",
       "      <td>1332435430801690624</td>\n",
       "      <td>{'username': 'jack', 'displayname': 'jack', 'i...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>54</td>\n",
       "      <td>8</td>\n",
       "      <td>226</td>\n",
       "      <td>1</td>\n",
       "      <td>1332428871891775488</td>\n",
       "      <td>und</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'JesseDorogusker', 'displayname'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://twitter.com/jack/status/13291496370060...</td>\n",
       "      <td>2020-11-18 19:49:02+00:00</td>\n",
       "      <td>@NeerajKA Welcome!</td>\n",
       "      <td>@NeerajKA Welcome!</td>\n",
       "      <td>1329149637006041088</td>\n",
       "      <td>{'username': 'jack', 'displayname': 'jack', 'i...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>72</td>\n",
       "      <td>14</td>\n",
       "      <td>800</td>\n",
       "      <td>8</td>\n",
       "      <td>1329140522565439490</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'NeerajKA', 'displayname': 'Neer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://twitter.com/jack/status/13291372550263...</td>\n",
       "      <td>2020-11-18 18:59:50+00:00</td>\n",
       "      <td>Join @CashApp! #Bitcoin https://t.co/SbYANIZyix</td>\n",
       "      <td>Join @CashApp! #Bitcoin twitter.com/owenbjenni...</td>\n",
       "      <td>1329137255026311168</td>\n",
       "      <td>{'username': 'jack', 'displayname': 'jack', 'i...</td>\n",
       "      <td>[https://twitter.com/owenbjennings/status/1329...</td>\n",
       "      <td>[https://t.co/SbYANIZyix]</td>\n",
       "      <td>585</td>\n",
       "      <td>277</td>\n",
       "      <td>2507</td>\n",
       "      <td>132</td>\n",
       "      <td>1329137255026311168</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'url': 'https://twitter.com/owenbjennings/sta...</td>\n",
       "      <td>[{'username': 'CashApp', 'displayname': 'Cash ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://twitter.com/jack/status/13291366656847...</td>\n",
       "      <td>2020-11-18 18:57:29+00:00</td>\n",
       "      <td>@kateconger @sarahintampa Nah</td>\n",
       "      <td>@kateconger @sarahintampa Nah</td>\n",
       "      <td>1329136665684705280</td>\n",
       "      <td>{'username': 'jack', 'displayname': 'jack', 'i...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>38</td>\n",
       "      <td>5</td>\n",
       "      <td>176</td>\n",
       "      <td>10</td>\n",
       "      <td>1329126492731699203</td>\n",
       "      <td>und</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'kateconger', 'displayname': 'o....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://twitter.com/jack/status/13291358061921...</td>\n",
       "      <td>2020-11-18 18:54:05+00:00</td>\n",
       "      <td>@mmasnick Terrible idea! And terribly false.</td>\n",
       "      <td>@mmasnick Terrible idea! And terribly false.</td>\n",
       "      <td>1329135806192107521</td>\n",
       "      <td>{'username': 'jack', 'displayname': 'jack', 'i...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>51</td>\n",
       "      <td>13</td>\n",
       "      <td>222</td>\n",
       "      <td>16</td>\n",
       "      <td>1329128773845860352</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'mmasnick', 'displayname': 'Mike...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 url  \\\n",
       "0  https://twitter.com/jack/status/13324354308016...   \n",
       "1  https://twitter.com/jack/status/13291496370060...   \n",
       "2  https://twitter.com/jack/status/13291372550263...   \n",
       "3  https://twitter.com/jack/status/13291366656847...   \n",
       "4  https://twitter.com/jack/status/13291358061921...   \n",
       "\n",
       "                       date                                          content  \\\n",
       "0 2020-11-27 21:25:36+00:00                      @JesseDorogusker @Square ❤️   \n",
       "1 2020-11-18 19:49:02+00:00                               @NeerajKA Welcome!   \n",
       "2 2020-11-18 18:59:50+00:00  Join @CashApp! #Bitcoin https://t.co/SbYANIZyix   \n",
       "3 2020-11-18 18:57:29+00:00                    @kateconger @sarahintampa Nah   \n",
       "4 2020-11-18 18:54:05+00:00     @mmasnick Terrible idea! And terribly false.   \n",
       "\n",
       "                                     renderedContent                   id  \\\n",
       "0                        @JesseDorogusker @Square ❤️  1332435430801690624   \n",
       "1                                 @NeerajKA Welcome!  1329149637006041088   \n",
       "2  Join @CashApp! #Bitcoin twitter.com/owenbjenni...  1329137255026311168   \n",
       "3                      @kateconger @sarahintampa Nah  1329136665684705280   \n",
       "4       @mmasnick Terrible idea! And terribly false.  1329135806192107521   \n",
       "\n",
       "                                                user  \\\n",
       "0  {'username': 'jack', 'displayname': 'jack', 'i...   \n",
       "1  {'username': 'jack', 'displayname': 'jack', 'i...   \n",
       "2  {'username': 'jack', 'displayname': 'jack', 'i...   \n",
       "3  {'username': 'jack', 'displayname': 'jack', 'i...   \n",
       "4  {'username': 'jack', 'displayname': 'jack', 'i...   \n",
       "\n",
       "                                            outlinks  \\\n",
       "0                                                 []   \n",
       "1                                                 []   \n",
       "2  [https://twitter.com/owenbjennings/status/1329...   \n",
       "3                                                 []   \n",
       "4                                                 []   \n",
       "\n",
       "                 tcooutlinks  replyCount  retweetCount  likeCount  quoteCount  \\\n",
       "0                         []          54             8        226           1   \n",
       "1                         []          72            14        800           8   \n",
       "2  [https://t.co/SbYANIZyix]         585           277       2507         132   \n",
       "3                         []          38             5        176          10   \n",
       "4                         []          51            13        222          16   \n",
       "\n",
       "        conversationId lang  \\\n",
       "0  1332428871891775488  und   \n",
       "1  1329140522565439490   en   \n",
       "2  1329137255026311168   en   \n",
       "3  1329126492731699203  und   \n",
       "4  1329128773845860352   en   \n",
       "\n",
       "                                              source  media  retweetedTweet  \\\n",
       "0  <a href=\"http://twitter.com/download/iphone\" r...    NaN             NaN   \n",
       "1  <a href=\"http://twitter.com/download/iphone\" r...    NaN             NaN   \n",
       "2  <a href=\"http://twitter.com/download/iphone\" r...    NaN             NaN   \n",
       "3  <a href=\"http://twitter.com/download/iphone\" r...    NaN             NaN   \n",
       "4  <a href=\"http://twitter.com/download/iphone\" r...    NaN             NaN   \n",
       "\n",
       "                                         quotedTweet  \\\n",
       "0                                               None   \n",
       "1                                               None   \n",
       "2  {'url': 'https://twitter.com/owenbjennings/sta...   \n",
       "3                                               None   \n",
       "4                                               None   \n",
       "\n",
       "                                      mentionedUsers  \n",
       "0  [{'username': 'JesseDorogusker', 'displayname'...  \n",
       "1  [{'username': 'NeerajKA', 'displayname': 'Neer...  \n",
       "2  [{'username': 'CashApp', 'displayname': 'Cash ...  \n",
       "3  [{'username': 'kateconger', 'displayname': 'o....  \n",
       "4  [{'username': 'mmasnick', 'displayname': 'Mike...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reads the json generated from the CLI command above and creates a pandas dataframe\n",
    "tweets_df1 = pd.read_json('user-tweets.json', lines=True)\n",
    "\n",
    "# Displays first 5 entries from dataframe\n",
    "tweets_df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export dataframe into a CSV\n",
    "tweets_df1.to_csv('user-tweets.csv', sep=',', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Query by Text Search\n",
    "The code below will scrape for 500 tweets between June 1st, 2020 and July 31st, 2020, by a text search then provide a CSV file with Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting variables to be used in format string command below\n",
    "tweet_count = 500\n",
    "text_query = \"its the elephant\"\n",
    "since_date = \"2020-06-01\"\n",
    "until_date = \"2020-07-31\"\n",
    "\n",
    "# Using OS library to call CLI commands in Python\n",
    "os.system('snscrape --jsonl --max-results {} --since {} twitter-search \"{} until:{}\"> text-query-tweets.json'.format(tweet_count, since_date, text_query, until_date))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>outlinks</th>\n",
       "      <th>tcooutlinks</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>quoteCount</th>\n",
       "      <th>conversationId</th>\n",
       "      <th>lang</th>\n",
       "      <th>source</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>mentionedUsers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://twitter.com/TylerPaulUtt1/status/12889...</td>\n",
       "      <td>2020-07-30 23:57:02+00:00</td>\n",
       "      <td>@SiBuduh @langoinstitute do you know the Ko wo...</td>\n",
       "      <td>@SiBuduh @langoinstitute do you know the Ko wo...</td>\n",
       "      <td>1288986997143601152</td>\n",
       "      <td>{'username': 'TylerPaulUtt1', 'displayname': '...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1288307058928947204</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/#!/download/ipad\" ...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'SiBuduh', 'displayname': 'Ed Lu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://twitter.com/EndlessSynthwav/status/128...</td>\n",
       "      <td>2020-07-30 23:44:04+00:00</td>\n",
       "      <td>@RockstarGames Any idea if the elephant rifle ...</td>\n",
       "      <td>@RockstarGames Any idea if the elephant rifle ...</td>\n",
       "      <td>1288983731122966534</td>\n",
       "      <td>{'username': 'EndlessSynthwav', 'displayname':...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1288983731122966534</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/android\" ...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'RockstarGames', 'displayname': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://twitter.com/aanalyst50/status/12889677...</td>\n",
       "      <td>2020-07-30 22:40:40+00:00</td>\n",
       "      <td>@realDonaldTrump Trump just keeps ignoring the...</td>\n",
       "      <td>@realDonaldTrump Trump just keeps ignoring the...</td>\n",
       "      <td>1288967774795116550</td>\n",
       "      <td>{'username': 'aanalyst50', 'displayname': 'Don...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1288966119676616704</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'realDonaldTrump', 'displayname'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://twitter.com/RozeyBozzy/status/12889669...</td>\n",
       "      <td>2020-07-30 22:37:18+00:00</td>\n",
       "      <td>@cslogan88 Famous 19th century song from Engla...</td>\n",
       "      <td>@cslogan88 Famous 19th century song from Engla...</td>\n",
       "      <td>1288966929236066309</td>\n",
       "      <td>{'username': 'RozeyBozzy', 'displayname': 'Roz...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1288965246602838017</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"https://mobile.twitter.com\" rel=\"nofo...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'cslogan88', 'displayname': 'Chr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://twitter.com/alfred_hanan/status/128896...</td>\n",
       "      <td>2020-07-30 22:32:44+00:00</td>\n",
       "      <td>@realDonaldTrump #RepublicanTrumpVirus.\\nLets ...</td>\n",
       "      <td>@realDonaldTrump #RepublicanTrumpVirus.\\nLets ...</td>\n",
       "      <td>1288965780030144512</td>\n",
       "      <td>{'username': 'alfred_hanan', 'displayname': 'A...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1288947487911419905</td>\n",
       "      <td>en</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/android\" ...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'username': 'realDonaldTrump', 'displayname'...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 url  \\\n",
       "0  https://twitter.com/TylerPaulUtt1/status/12889...   \n",
       "1  https://twitter.com/EndlessSynthwav/status/128...   \n",
       "2  https://twitter.com/aanalyst50/status/12889677...   \n",
       "3  https://twitter.com/RozeyBozzy/status/12889669...   \n",
       "4  https://twitter.com/alfred_hanan/status/128896...   \n",
       "\n",
       "                       date  \\\n",
       "0 2020-07-30 23:57:02+00:00   \n",
       "1 2020-07-30 23:44:04+00:00   \n",
       "2 2020-07-30 22:40:40+00:00   \n",
       "3 2020-07-30 22:37:18+00:00   \n",
       "4 2020-07-30 22:32:44+00:00   \n",
       "\n",
       "                                             content  \\\n",
       "0  @SiBuduh @langoinstitute do you know the Ko wo...   \n",
       "1  @RockstarGames Any idea if the elephant rifle ...   \n",
       "2  @realDonaldTrump Trump just keeps ignoring the...   \n",
       "3  @cslogan88 Famous 19th century song from Engla...   \n",
       "4  @realDonaldTrump #RepublicanTrumpVirus.\\nLets ...   \n",
       "\n",
       "                                     renderedContent                   id  \\\n",
       "0  @SiBuduh @langoinstitute do you know the Ko wo...  1288986997143601152   \n",
       "1  @RockstarGames Any idea if the elephant rifle ...  1288983731122966534   \n",
       "2  @realDonaldTrump Trump just keeps ignoring the...  1288967774795116550   \n",
       "3  @cslogan88 Famous 19th century song from Engla...  1288966929236066309   \n",
       "4  @realDonaldTrump #RepublicanTrumpVirus.\\nLets ...  1288965780030144512   \n",
       "\n",
       "                                                user outlinks tcooutlinks  \\\n",
       "0  {'username': 'TylerPaulUtt1', 'displayname': '...       []          []   \n",
       "1  {'username': 'EndlessSynthwav', 'displayname':...       []          []   \n",
       "2  {'username': 'aanalyst50', 'displayname': 'Don...       []          []   \n",
       "3  {'username': 'RozeyBozzy', 'displayname': 'Roz...       []          []   \n",
       "4  {'username': 'alfred_hanan', 'displayname': 'A...       []          []   \n",
       "\n",
       "   replyCount  retweetCount  likeCount  quoteCount       conversationId lang  \\\n",
       "0           1             0          0           0  1288307058928947204   en   \n",
       "1           0             0          0           0  1288983731122966534   en   \n",
       "2           0             0          1           0  1288966119676616704   en   \n",
       "3           0             0          0           0  1288965246602838017   en   \n",
       "4           0             0          0           0  1288947487911419905   en   \n",
       "\n",
       "                                              source media  retweetedTweet  \\\n",
       "0  <a href=\"http://twitter.com/#!/download/ipad\" ...  None             NaN   \n",
       "1  <a href=\"http://twitter.com/download/android\" ...  None             NaN   \n",
       "2  <a href=\"http://twitter.com/download/iphone\" r...  None             NaN   \n",
       "3  <a href=\"https://mobile.twitter.com\" rel=\"nofo...  None             NaN   \n",
       "4  <a href=\"http://twitter.com/download/android\" ...  None             NaN   \n",
       "\n",
       "  quotedTweet                                     mentionedUsers  \n",
       "0        None  [{'username': 'SiBuduh', 'displayname': 'Ed Lu...  \n",
       "1        None  [{'username': 'RockstarGames', 'displayname': ...  \n",
       "2        None  [{'username': 'realDonaldTrump', 'displayname'...  \n",
       "3        None  [{'username': 'cslogan88', 'displayname': 'Chr...  \n",
       "4        None  [{'username': 'realDonaldTrump', 'displayname'...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reads the json generated from the CLI command above and creates a pandas dataframe\n",
    "tweets_df2 = pd.read_json('text-query-tweets.json', lines=True)\n",
    "\n",
    "# Displays first 5 entries from dataframe\n",
    "tweets_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export dataframe into a CSV\n",
    "tweets_df2.to_csv('text-query-tweets.csv', sep=',', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}