examples/DataConnector_Twitter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part of this file can't be rendered in GitHub. Refer to the following link for a properly rendered version of this file: https://nbviewer.jupyter.org/github/sfu-db/dataprep/blob/develop/examples/DataConnector_Twitter.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Connector for Twitter \n",
    "\n",
    "In this example, we will be going over how to use Connector with Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Connector is a component in the DataPrep library that aims to simplify data access by providing a standard API set. The goal is to help users skip the complex API configuration. In this tutorial, we demonstrate how to use the connector component with Twitter.\n",
    "\n",
    "If you haven't installed DataPrep, run command `!pip install dataprep` or execute the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run me if you'd like to install\n",
    "!pip install dataprep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download and store the configuration files in DataPrep. \n",
    "\n",
    "The configuration files are used to configure the parameters and initial setup for the API. The available configuration files can be manually downloaded here: [Configuration Files](https://github.com/sfu-db/DataConnectorConfigs) or automatically downloaded at usage. \n",
    "\n",
    "Store the configuration file in the dataprep folder. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Obtaining access token from Twitter\n",
    "\n",
    "In order for Twitter API to work, you need four keys - the **Consumer**, **Consumer Secret**, **Access Token** and **Access Token Secret**. These keys are unique identifiers of an application requesting access to the Twitter API. The access token will be your credential when making an API request. These keys and tokens can be obtained with the following three simple steps.\n",
    "\n",
    "##### 1. Create a Developer Account\n",
    "In order to access Twitter data, you will need to create a server-side application on Twitter and to do so, you will need to create a developer account in order to create an application. You can sign up for the developer account on the [Twitter Developer Account](https://developer.twitter.com/en). An email confirming your approval of the developer account will be sent to you shortly after signing up.\n",
    "\n",
    "##### 2. Create an Application\n",
    "After gaining access to your developer account, log in and create an app in the [Apps section of your profile](https://developer.twitter.com/en/apps). Fill in the name, description and URL of your website. You can skip the callback URL and other URL sections.\n",
    "\n",
    "##### 3. Generate Auth Keys and Tokens\n",
    "\n",
    "After you successfully create an application, you can find your access keys and token keys in the **Keys and Tokens** section of Application Details. Generate the Access Token and Access Token Secret. Store them in a secure location as they will provide you access to the Twitter's data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize connector\n",
    "\n",
    "Copy and paste the **Consumer Key** into the **client_id** variable. Copy and paste the **Consumer Secret Key** into the **client_secret** variable. Ensure the **API name** argument is correct. This establishes a connection with Twitter and returns an object. Once you run the code you can use the built in functions available from connector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataprep.connector import connect, info\n",
    "\n",
    "client_id = '<insert Consumer Key>'\n",
    "client_secret = '<insert Consumer Secret Key>'\n",
    "dc = connect('twitter', _auth={'client_id':client_id, 'client_secret':client_secret})\n",
    "\n",
    "dc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Functionalities\n",
    "\n",
    "Connector has several functions you can perform to gain insight on the data downloaded from Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connector.info\n",
    "The info method gives information and guidelines on using the connector. There are 4 sections in the response and they are table, parameters, example and schema.\n",
    "\n",
    ">1. Table - The table(s) being accessed.\n",
    ">2. Parameters - Identifies which parameters can be used to call the method.\n",
    ">3. Examples - Shows how you can call the methods in the Connector class.\n",
    ">4. Schema - Names and data types of attributes in the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html>\n",
       "    <head>\n",
       "        <title> DataPrep.Connector Info </title>\n",
       "        <script>\n",
       "            function switchTab637(e) {\n",
       "                const selectedAreaSuffix = e.parentElement.parentElement.className.split('-')[1];\n",
       "                const selectedTabId = e.id.split('-')[2];\n",
       "                const selectedTabContent = document.querySelector(`.contents-${selectedAreaSuffix}>div:nth-of-type(${selectedTabId})`);\n",
       "                const contentArray = document.getElementsByClassName('info-637')\n",
       "                for (let i of contentArray) {\n",
       "                    i.style.display = 'none';\n",
       "                }\n",
       "                selectedTabContent.style.display = 'block';\n",
       "            }\n",
       "        </script>\n",
       "        <style>\n",
       "            .container-637 {\n",
       "            }\n",
       "            .container-637 input[type=radio] {\n",
       "            display: none;\n",
       "            }\n",
       "            .tabs-637 {\n",
       "            font-family: -apple-system, 'Helvetica Neue', 'Helvetica', 'Arial', 'Lucida Grande', sans-serif;\n",
       "            -webkit-font-smoothing: antialiased;\n",
       "            display: grid;\n",
       "            flex-wrap: wrap;\n",
       "            gap: 5px;\n",
       "            grid-template-columns: repeat(auto-fill, minmax(110px, 1fr));\n",
       "            justify-content: wrap;\n",
       "            }\n",
       "            .tabs-637 label {\n",
       "            word-wrap: break-word;\n",
       "            text-align: center;\n",
       "            font-size: 12px;\n",
       "            border-bottom: 2px solid #9edae5;\n",
       "            transition: border 0.3s ease;\n",
       "            }\n",
       "            .tabs-637 label:hover {\n",
       "            background: rgba(0, 0, 0, 0.1);\n",
       "            }\n",
       "            .tabs-637 input:checked+label {\n",
       "            border-bottom: 2px solid #1f77b4;\n",
       "            }\n",
       "            .contents-637>div:nth-of-type(n+2) {\n",
       "            display: none;\n",
       "            }\n",
       "            h4 {\n",
       "            font-size: 14px;\n",
       "            font-family: Arial;\n",
       "            }\n",
       "            p {\n",
       "            font-size: 13px;\n",
       "            }\n",
       "        </style>\n",
       "    </head>\n",
       "    <body>\n",
       "        <div class=\"container-637\">\n",
       "            <div class=\"tabs-637\">\n",
       "                <input type=\"radio\" name=\"select\" id='tab-637-1' checked onclick=\"switchTab637(this)\">\n",
       "                <label for=\"tab-637-1\">tweets</label>\n",
       "                \n",
       "                \n",
       "                \n",
       "                    \t\t\n",
       "            </div>\n",
       "            <div class=\"contents-637\">\n",
       "                \n",
       "                <div class=\"info-637\">\n",
       "                    <div>\n",
       "                        <h4><u>Parameters</u></h4>\n",
       "                        \n",
       "                        \n",
       "                        <p style=\"display:wrap\"><code>q (required)</code></p>\n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        <p style=\"display:wrap\"><code>count</code></p>\n",
       "                        \n",
       "                        <p style=\"display:wrap\"><code>max_id</code></p>\n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                    </div>\n",
       "                    <div>\n",
       "                        <h4><u>Example</u></h4>\n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        \n",
       "                        <p><code>dc = connect('twitter', _auth={'client_id':'QNf0IeGSeMq3K*********NIU9mfHFMfX3cYe' , 'client_secret':'eeNspLqiRoVfX*********3V2ntIiXKui9A6X'}, _concurrency=3)<br>df = await dc.query('tweets', q='celebrity', _count=20)</code></p>\n",
       "                    </div>\n",
       "                    <div>\n",
       "                    \t<h4><u>Schema</u></h4>\n",
       "                    \t<style  type=\"text/css\" >\n",
       "    #T_890d309c_35b1_11eb_8225_c82a14415709 th {\n",
       "          background: white;\n",
       "          font-weight: bold;\n",
       "          text-align: right;\n",
       "          font-family: arial;\n",
       "          font-size: 13;\n",
       "    }    #T_890d309c_35b1_11eb_8225_c82a14415709 td {\n",
       "          font-family: arial;\n",
       "    }    #T_890d309c_35b1_11eb_8225_c82a14415709 tr:nth-of-type(odd) {\n",
       "          background: #f5f5f5;\n",
       "          font-size: 13;\n",
       "          text-align: right;\n",
       "    }    #T_890d309c_35b1_11eb_8225_c82a14415709 tr:nth-of-type(even) {\n",
       "          background: #white;\n",
       "          font-size: 13;\n",
       "          text-align: right;\n",
       "    }    #T_890d309c_35b1_11eb_8225_c82a14415709 tr:hover {\n",
       "          background-color: e0f1ff;\n",
       "    }</style><table id=\"T_890d309c_35b1_11eb_8225_c82a14415709\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >column_name</th>        <th class=\"col_heading level0 col1\" >data_type</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_890d309c_35b1_11eb_8225_c82a14415709level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row0_col0\" class=\"data row0 col0\" >created_at</td>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row0_col1\" class=\"data row0 col1\" >string</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_890d309c_35b1_11eb_8225_c82a14415709level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row1_col0\" class=\"data row1 col0\" >text</td>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row1_col1\" class=\"data row1 col1\" >string</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_890d309c_35b1_11eb_8225_c82a14415709level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row2_col0\" class=\"data row2 col0\" >truncated</td>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row2_col1\" class=\"data row2 col1\" >boolean</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_890d309c_35b1_11eb_8225_c82a14415709level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row3_col0\" class=\"data row3 col0\" >iso_language_code</td>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row3_col1\" class=\"data row3 col1\" >string</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_890d309c_35b1_11eb_8225_c82a14415709level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row4_col0\" class=\"data row4 col0\" >hashtags</td>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row4_col1\" class=\"data row4 col1\" >string</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_890d309c_35b1_11eb_8225_c82a14415709level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row5_col0\" class=\"data row5 col0\" >id</td>\n",
       "                        <td id=\"T_890d309c_35b1_11eb_8225_c82a14415709row5_col1\" class=\"data row5 col1\" >int</td>\n",
       "            </tr>\n",
       "    </tbody></table>\n",
       "                    </div>\n",
       "                </div>\n",
       "                \n",
       "            </div>\n",
       "        </div>\n",
       "    </body>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "info('twitter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connector.query\n",
    "The query method downloads the website data and displays it in a Dataframe. The parameters must meet the requirements as indicated in connector.info for the operation to run. You can use the **_count** parameter to specify the number of tweets to be fetched. Each request can currently fetch a maximum of 100 requests.\n",
    "\n",
    "When the data is received from the server, it will either be in a JSON or XML format. The connector reformats the data in pandas Dataframe for the convenience of downstream operations.\n",
    "\n",
    "As an example, let's try to get 50 tweets related to COVID-19 from Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Searching for tweets related to COVID-19"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "      <th>truncated</th>\n",
       "      <th>iso_language_code</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>RT @camposdecas: Segunda fase del intento de d...</td>\n",
       "      <td>False</td>\n",
       "      <td>es</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621572565528583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>This week during a Covid-19 pandemic, I’m work...</td>\n",
       "      <td>True</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621572372434945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>RT @devisridhar: The Pfizer vaccine is importa...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621572146008065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>RT @Atksufi: Covid-19 would have been over if ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621572129284096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>RT @kasapoglu: Eşimle birlikte yaptırmış olduğ...</td>\n",
       "      <td>False</td>\n",
       "      <td>tr</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621572028624896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>Números que marcam esta quinta-feira, 3 de Dez...</td>\n",
       "      <td>True</td>\n",
       "      <td>pt</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621570464063488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Thu Dec 03 22:12:33 +0000 2020</td>\n",
       "      <td>Hi @JimInhofe @LindseyGrahamSC @SenatorRisch a...</td>\n",
       "      <td>True</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621570124353540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>Michigan football dealing with at least 12 pos...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621569440595970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>Trump has given America too much grief to be c...</td>\n",
       "      <td>True</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621569193132032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>RT @LiveSquawk: $MRNA | Moderna: Phase 1 Covid...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621568870330368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>@olderChrisBrown @Keihndeth @ConceptualJames S...</td>\n",
       "      <td>True</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621568115253249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>Facebook will ban fakes about COVID-19 vaccine...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621568090116104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>RT @Atksufi: Covid-19 would have been over if ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621567905529856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>RT @Mippcivzla: #NotiMippCI 📰🗞| Venezuela plan...</td>\n",
       "      <td>False</td>\n",
       "      <td>es</td>\n",
       "      <td>[{'text': 'NotiMippCI', 'indices': [16, 27]}]</td>\n",
       "      <td>1334621567855321090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>RT @PeruNewsAlerta: El mundo supera los 1.5 mi...</td>\n",
       "      <td>False</td>\n",
       "      <td>es</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621567821594624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>Moderna is bullish on COVID-19 vaccine availab...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621567268032517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>COVID-19: NIMR Trains Health Personnel On Usag...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621567146360834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>RT @YorkRegionGovt: Today, York Region is repo...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621566907330561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Thu Dec 03 22:12:32 +0000 2020</td>\n",
       "      <td>RT @eldiarioes: La vacuna de Pfizer podría est...</td>\n",
       "      <td>False</td>\n",
       "      <td>es</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621566311788549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @UNMHSC: New Mexico 🗣\\n\\nWe all have the po...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621565426626560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @CDCgov: A person with #COVID19 can spread ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'text': 'COVID19', 'indices': [26, 34]}]</td>\n",
       "      <td>1334621564520816640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @RepUnderwood: This is heartbreaking. I'm a...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621564495683588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>@DHSCgovuk The modeller who made the predictio...</td>\n",
       "      <td>True</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621564462112770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>Malo #Gusto a été testé positif au COVID-19 et...</td>\n",
       "      <td>False</td>\n",
       "      <td>fr</td>\n",
       "      <td>[{'text': 'Gusto', 'indices': [5, 11]}]</td>\n",
       "      <td>1334621564147535875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @BreitbartNews: Director of National Intell...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621563862257666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @politico: President Trump approved request...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621563371577344</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @ForexLive: Moderna is bullish on COVID-19 ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621563346366465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>Testes para Covid-19 vendidos em Joinville pod...</td>\n",
       "      <td>False</td>\n",
       "      <td>pt</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621563266736131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @Atksufi: Covid-19 would have been over if ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621563266723844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @g1: Facebook e Instagram dizem que vão rem...</td>\n",
       "      <td>False</td>\n",
       "      <td>pt</td>\n",
       "      <td>[{'text': 'G1', 'indices': [124, 127]}]</td>\n",
       "      <td>1334621562746658819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Thu Dec 03 22:12:31 +0000 2020</td>\n",
       "      <td>RT @cnnadam: NEW: Three former US presidents –...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621562394324996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>Moderna: Phase 1 Covid-19 Vaccine Participants...</td>\n",
       "      <td>True</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621561513504769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>@CEOofpee @No_Context_PvZ It's been closed due...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621561236631552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>RT @USATODAY: As COVID-19 cases, hospitalizati...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621559475085316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>@oedsonline @araqueiro @NetoSiva2 @UOL https:/...</td>\n",
       "      <td>False</td>\n",
       "      <td>und</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621559357640706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>RT @EdwardHolfman: Luis Lacalle Pou, Uruguay’s...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621559151943680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>RT @nicolarsouil: @franceonu C'est exactement ...</td>\n",
       "      <td>False</td>\n",
       "      <td>fr</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621559122780160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Thu Dec 03 22:12:30 +0000 2020</td>\n",
       "      <td>RT @MeeBee23: @kurteichenwald @govkristinoem E...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621558661283841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @Complex: California reported a record 28,0...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621557168074752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @telegramdotcom: 'Phenomenal' effort: Gov. ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621557080137729</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @mattlargey: Just get @HEB to manage the wh...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621556404875265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @UtahDepOfHealth: This afternoon we placed ...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621555494510593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>@TheRickWilson I thought COVID-19 wasn't a thi...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621555452764164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @cnnadam: NEW: Three former US presidents –...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621555167412225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @wjw26: Throughout the COVID-19 crisis in F...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621554920083457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @TarvariousWWMT: An angry business owner in...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621554160918530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @ValeriaBnews: Pergunta: Pq a mídia, a \"siê...</td>\n",
       "      <td>False</td>\n",
       "      <td>pt</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621554160832513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @AnnCoulter: \"Victor Davis Hanson...advance...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621554076839936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Thu Dec 03 22:12:29 +0000 2020</td>\n",
       "      <td>RT @DrEricDing: BREAKING— Pfizer is slashing i...</td>\n",
       "      <td>False</td>\n",
       "      <td>en</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621553649016832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Thu Dec 03 22:12:28 +0000 2020</td>\n",
       "      <td>não dê mole pra covid 19 nem 20</td>\n",
       "      <td>False</td>\n",
       "      <td>pt</td>\n",
       "      <td>[]</td>\n",
       "      <td>1334621552156008453</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        created_at  \\\n",
       "0   Thu Dec 03 22:12:33 +0000 2020   \n",
       "1   Thu Dec 03 22:12:33 +0000 2020   \n",
       "2   Thu Dec 03 22:12:33 +0000 2020   \n",
       "3   Thu Dec 03 22:12:33 +0000 2020   \n",
       "4   Thu Dec 03 22:12:33 +0000 2020   \n",
       "5   Thu Dec 03 22:12:33 +0000 2020   \n",
       "6   Thu Dec 03 22:12:33 +0000 2020   \n",
       "7   Thu Dec 03 22:12:32 +0000 2020   \n",
       "8   Thu Dec 03 22:12:32 +0000 2020   \n",
       "9   Thu Dec 03 22:12:32 +0000 2020   \n",
       "10  Thu Dec 03 22:12:32 +0000 2020   \n",
       "11  Thu Dec 03 22:12:32 +0000 2020   \n",
       "12  Thu Dec 03 22:12:32 +0000 2020   \n",
       "13  Thu Dec 03 22:12:32 +0000 2020   \n",
       "14  Thu Dec 03 22:12:32 +0000 2020   \n",
       "15  Thu Dec 03 22:12:32 +0000 2020   \n",
       "16  Thu Dec 03 22:12:32 +0000 2020   \n",
       "17  Thu Dec 03 22:12:32 +0000 2020   \n",
       "18  Thu Dec 03 22:12:32 +0000 2020   \n",
       "19  Thu Dec 03 22:12:31 +0000 2020   \n",
       "20  Thu Dec 03 22:12:31 +0000 2020   \n",
       "21  Thu Dec 03 22:12:31 +0000 2020   \n",
       "22  Thu Dec 03 22:12:31 +0000 2020   \n",
       "23  Thu Dec 03 22:12:31 +0000 2020   \n",
       "24  Thu Dec 03 22:12:31 +0000 2020   \n",
       "25  Thu Dec 03 22:12:31 +0000 2020   \n",
       "26  Thu Dec 03 22:12:31 +0000 2020   \n",
       "27  Thu Dec 03 22:12:31 +0000 2020   \n",
       "28  Thu Dec 03 22:12:31 +0000 2020   \n",
       "29  Thu Dec 03 22:12:31 +0000 2020   \n",
       "30  Thu Dec 03 22:12:31 +0000 2020   \n",
       "31  Thu Dec 03 22:12:30 +0000 2020   \n",
       "32  Thu Dec 03 22:12:30 +0000 2020   \n",
       "33  Thu Dec 03 22:12:30 +0000 2020   \n",
       "34  Thu Dec 03 22:12:30 +0000 2020   \n",
       "35  Thu Dec 03 22:12:30 +0000 2020   \n",
       "36  Thu Dec 03 22:12:30 +0000 2020   \n",
       "37  Thu Dec 03 22:12:30 +0000 2020   \n",
       "38  Thu Dec 03 22:12:29 +0000 2020   \n",
       "39  Thu Dec 03 22:12:29 +0000 2020   \n",
       "40  Thu Dec 03 22:12:29 +0000 2020   \n",
       "41  Thu Dec 03 22:12:29 +0000 2020   \n",
       "42  Thu Dec 03 22:12:29 +0000 2020   \n",
       "43  Thu Dec 03 22:12:29 +0000 2020   \n",
       "44  Thu Dec 03 22:12:29 +0000 2020   \n",
       "45  Thu Dec 03 22:12:29 +0000 2020   \n",
       "46  Thu Dec 03 22:12:29 +0000 2020   \n",
       "47  Thu Dec 03 22:12:29 +0000 2020   \n",
       "48  Thu Dec 03 22:12:29 +0000 2020   \n",
       "49  Thu Dec 03 22:12:28 +0000 2020   \n",
       "\n",
       "                                                 text  truncated  \\\n",
       "0   RT @camposdecas: Segunda fase del intento de d...      False   \n",
       "1   This week during a Covid-19 pandemic, I’m work...       True   \n",
       "2   RT @devisridhar: The Pfizer vaccine is importa...      False   \n",
       "3   RT @Atksufi: Covid-19 would have been over if ...      False   \n",
       "4   RT @kasapoglu: Eşimle birlikte yaptırmış olduğ...      False   \n",
       "5   Números que marcam esta quinta-feira, 3 de Dez...       True   \n",
       "6   Hi @JimInhofe @LindseyGrahamSC @SenatorRisch a...       True   \n",
       "7   Michigan football dealing with at least 12 pos...      False   \n",
       "8   Trump has given America too much grief to be c...       True   \n",
       "9   RT @LiveSquawk: $MRNA | Moderna: Phase 1 Covid...      False   \n",
       "10  @olderChrisBrown @Keihndeth @ConceptualJames S...       True   \n",
       "11  Facebook will ban fakes about COVID-19 vaccine...      False   \n",
       "12  RT @Atksufi: Covid-19 would have been over if ...      False   \n",
       "13  RT @Mippcivzla: #NotiMippCI 📰🗞| Venezuela plan...      False   \n",
       "14  RT @PeruNewsAlerta: El mundo supera los 1.5 mi...      False   \n",
       "15  Moderna is bullish on COVID-19 vaccine availab...      False   \n",
       "16  COVID-19: NIMR Trains Health Personnel On Usag...      False   \n",
       "17  RT @YorkRegionGovt: Today, York Region is repo...      False   \n",
       "18  RT @eldiarioes: La vacuna de Pfizer podría est...      False   \n",
       "19  RT @UNMHSC: New Mexico 🗣\\n\\nWe all have the po...      False   \n",
       "20  RT @CDCgov: A person with #COVID19 can spread ...      False   \n",
       "21  RT @RepUnderwood: This is heartbreaking. I'm a...      False   \n",
       "22  @DHSCgovuk The modeller who made the predictio...       True   \n",
       "23  Malo #Gusto a été testé positif au COVID-19 et...      False   \n",
       "24  RT @BreitbartNews: Director of National Intell...      False   \n",
       "25  RT @politico: President Trump approved request...      False   \n",
       "26  RT @ForexLive: Moderna is bullish on COVID-19 ...      False   \n",
       "27  Testes para Covid-19 vendidos em Joinville pod...      False   \n",
       "28  RT @Atksufi: Covid-19 would have been over if ...      False   \n",
       "29  RT @g1: Facebook e Instagram dizem que vão rem...      False   \n",
       "30  RT @cnnadam: NEW: Three former US presidents –...      False   \n",
       "31  Moderna: Phase 1 Covid-19 Vaccine Participants...       True   \n",
       "32  @CEOofpee @No_Context_PvZ It's been closed due...      False   \n",
       "33  RT @USATODAY: As COVID-19 cases, hospitalizati...      False   \n",
       "34  @oedsonline @araqueiro @NetoSiva2 @UOL https:/...      False   \n",
       "35  RT @EdwardHolfman: Luis Lacalle Pou, Uruguay’s...      False   \n",
       "36  RT @nicolarsouil: @franceonu C'est exactement ...      False   \n",
       "37  RT @MeeBee23: @kurteichenwald @govkristinoem E...      False   \n",
       "38  RT @Complex: California reported a record 28,0...      False   \n",
       "39  RT @telegramdotcom: 'Phenomenal' effort: Gov. ...      False   \n",
       "40  RT @mattlargey: Just get @HEB to manage the wh...      False   \n",
       "41  RT @UtahDepOfHealth: This afternoon we placed ...      False   \n",
       "42  @TheRickWilson I thought COVID-19 wasn't a thi...      False   \n",
       "43  RT @cnnadam: NEW: Three former US presidents –...      False   \n",
       "44  RT @wjw26: Throughout the COVID-19 crisis in F...      False   \n",
       "45  RT @TarvariousWWMT: An angry business owner in...      False   \n",
       "46  RT @ValeriaBnews: Pergunta: Pq a mídia, a \"siê...      False   \n",
       "47  RT @AnnCoulter: \"Victor Davis Hanson...advance...      False   \n",
       "48  RT @DrEricDing: BREAKING— Pfizer is slashing i...      False   \n",
       "49                    não dê mole pra covid 19 nem 20      False   \n",
       "\n",
       "   iso_language_code                                       hashtags  \\\n",
       "0                 es                                             []   \n",
       "1                 en                                             []   \n",
       "2                 en                                             []   \n",
       "3                 en                                             []   \n",
       "4                 tr                                             []   \n",
       "5                 pt                                             []   \n",
       "6                 en                                             []   \n",
       "7                 en                                             []   \n",
       "8                 en                                             []   \n",
       "9                 en                                             []   \n",
       "10                en                                             []   \n",
       "11                en                                             []   \n",
       "12                en                                             []   \n",
       "13                es  [{'text': 'NotiMippCI', 'indices': [16, 27]}]   \n",
       "14                es                                             []   \n",
       "15                en                                             []   \n",
       "16                en                                             []   \n",
       "17                en                                             []   \n",
       "18                es                                             []   \n",
       "19                en                                             []   \n",
       "20                en     [{'text': 'COVID19', 'indices': [26, 34]}]   \n",
       "21                en                                             []   \n",
       "22                en                                             []   \n",
       "23                fr        [{'text': 'Gusto', 'indices': [5, 11]}]   \n",
       "24                en                                             []   \n",
       "25                en                                             []   \n",
       "26                en                                             []   \n",
       "27                pt                                             []   \n",
       "28                en                                             []   \n",
       "29                pt        [{'text': 'G1', 'indices': [124, 127]}]   \n",
       "30                en                                             []   \n",
       "31                en                                             []   \n",
       "32                en                                             []   \n",
       "33                en                                             []   \n",
       "34               und                                             []   \n",
       "35                en                                             []   \n",
       "36                fr                                             []   \n",
       "37                en                                             []   \n",
       "38                en                                             []   \n",
       "39                en                                             []   \n",
       "40                en                                             []   \n",
       "41                en                                             []   \n",
       "42                en                                             []   \n",
       "43                en                                             []   \n",
       "44                en                                             []   \n",
       "45                en                                             []   \n",
       "46                pt                                             []   \n",
       "47                en                                             []   \n",
       "48                en                                             []   \n",
       "49                pt                                             []   \n",
       "\n",
       "                     id  \n",
       "0   1334621572565528583  \n",
       "1   1334621572372434945  \n",
       "2   1334621572146008065  \n",
       "3   1334621572129284096  \n",
       "4   1334621572028624896  \n",
       "5   1334621570464063488  \n",
       "6   1334621570124353540  \n",
       "7   1334621569440595970  \n",
       "8   1334621569193132032  \n",
       "9   1334621568870330368  \n",
       "10  1334621568115253249  \n",
       "11  1334621568090116104  \n",
       "12  1334621567905529856  \n",
       "13  1334621567855321090  \n",
       "14  1334621567821594624  \n",
       "15  1334621567268032517  \n",
       "16  1334621567146360834  \n",
       "17  1334621566907330561  \n",
       "18  1334621566311788549  \n",
       "19  1334621565426626560  \n",
       "20  1334621564520816640  \n",
       "21  1334621564495683588  \n",
       "22  1334621564462112770  \n",
       "23  1334621564147535875  \n",
       "24  1334621563862257666  \n",
       "25  1334621563371577344  \n",
       "26  1334621563346366465  \n",
       "27  1334621563266736131  \n",
       "28  1334621563266723844  \n",
       "29  1334621562746658819  \n",
       "30  1334621562394324996  \n",
       "31  1334621561513504769  \n",
       "32  1334621561236631552  \n",
       "33  1334621559475085316  \n",
       "34  1334621559357640706  \n",
       "35  1334621559151943680  \n",
       "36  1334621559122780160  \n",
       "37  1334621558661283841  \n",
       "38  1334621557168074752  \n",
       "39  1334621557080137729  \n",
       "40  1334621556404875265  \n",
       "41  1334621555494510593  \n",
       "42  1334621555452764164  \n",
       "43  1334621555167412225  \n",
       "44  1334621554920083457  \n",
       "45  1334621554160918530  \n",
       "46  1334621554160832513  \n",
       "47  1334621554076839936  \n",
       "48  1334621553649016832  \n",
       "49  1334621552156008453  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = await dc.query(\"tweets\", _q=\"covid-19\", _count=50)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# That's all for now. \n",
    "If you are interested in writing your own configuration file or modify an existing one, refer to the [Configuration Files](https://github.com/sfu-db/DataConnectorConfigs>)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}