1_network_hkprotests.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network Analysis of Twitter User on the 2019 Hong Kong Protest Movement "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the summer of 2019, lacking confidence in China’s judiciary system and human rights protection due to its history of suppressing political dissent, Hong Kong citizens demonstrated against the local government led by Chief Executive Carrier Lam to legalize the extradition of suspects to mainland China. Following an escalation in the severity of policing tactics on 12 June 2019, the protester’s objectives were becoming the five demands: fully withdraw the extradition bill; implement universal suffrage in Hong Kong, set up an inquiry to probe police brutality; withdraw a characterization of early protests as “riots and release those arrested at protest. \n",
    "\n",
    "Aligned with the decentralized nature of network-movement, Hong Kong 2019 protests are characterized as “formless, shapeless, like water”, as the hardcore protesters organizing in small cells with no formal hierarchy and crowdsourcing their tactics and slogans through social media in a highly dispersed way. This phenomenon is accordance with the theory developed by Swann (2020) in the book of Anarchist Cybernetics: Control and Communication in Radical Politics, which explores how large-scale coordination and communication can happen without central direction, and how social media platforms can facilitate interactive communication and participatory and democratic forms of organization. \n",
    "\n",
    "Given the mutation of global scenario of social mobilization towards higher intensity of digital network and higher degree of decentralization and autonomy, it’s critical to understand how the evolution spreads on social media and synchronize with the demonstration. \n",
    "\n",
    "This project argue that the decision-making process tend to be more decentralized, the identity can be formed from symbols, and synchrony between network and real-world activities. Using the Twitter data from 2019 Hong Kong Protests caused by the anti-extradition law amendment bill, this paper would use network analysis including K-core decomposition and community detection and sentimental analysis to examine the emotion cascade and propagate from one node to another, the user-hashtag interaction, the evolution of opinion groups and emergence of opinion leaders. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/x9/qwbg7g8502374fz76n_3_x1m0000gn/T/ipykernel_52082/695862002.py:16: DeprecationWarning: Importing display from IPython.core.display is deprecated since IPython 7.14, please import from IPython display\n",
      "  from IPython.core.display import display, HTML\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>.container { width:95% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generic ones\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "# Word processing libraries\n",
    "import re\n",
    "from nltk.corpus import wordnet\n",
    "import string\n",
    "from nltk import pos_tag\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import WhitespaceTokenizer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "# Widen the size of each cell\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:95% !important; }</style>\"))\n",
    "\n",
    "# import necessary packages for network analysis\n",
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import dataset \n",
    "\n",
    "The .csv files each contains 15k tweets and the search parameters used are stated below:\n",
    "1. search_words = \"#hongkong OR #hkprotests OR #freehongkong OR #hongkongprotests OR #hkpolicebrutality OR #antichinazi OR #standwithhongkong OR #hkpolicestate OR #HKpoliceterrorist OR #standwithhk OR #hkpoliceterrorism\"\n",
    "2. date_since = [\"2019-10-28\",\"2019-10-29\",\"2019-10-30\"]\n",
    "\n",
    "\n",
    "Source: \n",
    "- https://towardsdatascience.com/creating-the-twitter-sentiment-analysis-program-in-python-with-naive-bayes-classification-672e5589a7ed\n",
    "- https://towardsdatascience.com/extracting-twitter-data-pre-processing-and-sentiment-analysis-using-python-3-0-7192bd8b47cf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_1st = pd.read_csv(os.getcwd() + '/data/raw' + '/20191103_131218_sahkprotests_tweets.csv')\n",
    "tweets_2nd = pd.read_csv(os.getcwd() + '/data/raw' + '/20191103_153932_sahkprotests_tweets.csv')\n",
    "tweets_3rd = pd.read_csv(os.getcwd() + '/data/raw' + '/20191105_010011_sahkprotests_tweets.csv')\n",
    "tweets_4th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191105_222815_sahkprotests_tweets.csv')\n",
    "tweets_5th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191107_000333_sahkprotests_tweets.csv')\n",
    "tweets_6th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191108_001436_sahkprotests_tweets.csv')\n",
    "tweets_7th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191109_030106_sahkprotests_tweets.csv')\n",
    "tweets_8th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191109_120954_sahkprotests_tweets.csv')\n",
    "tweets_9th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191110_021422_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_10th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191110_134433_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_11th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191111_223912_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_12th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191112_231846_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_13th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191113_225544_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_14th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191115_000508_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_15th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191116_121136_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_16th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191117_001011_sahkprotests_tweets.csv', engine='python')\n",
    "tweets_17th = pd.read_csv(os.getcwd() + '/data/raw' + '/20191120_015710_sahkprotests_tweets.csv', engine='python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of 1st set is: (15000, 11)\n",
      "Size of 2nd set is: (15000, 11)\n",
      "Size of 3rd set is: (15000, 11)\n",
      "Size of 4th set is: (15001, 11)\n",
      "Size of 5th set is: (15000, 11)\n",
      "Size of 6th set is: (15000, 11)\n",
      "Size of 7th set is: (15000, 11)\n",
      "Size of 8th set is: (15000, 11)\n",
      "Size of 9th set is: (15001, 11)\n",
      "Size of 10th set is: (15001, 11)\n",
      "Size of 11th set is: (15000, 11)\n",
      "Size of 12th set is: (15000, 11)\n",
      "Size of 13th set is: (15000, 11)\n",
      "Size of 14th set is: (15000, 11)\n",
      "Size of 15th set is: (15000, 11)\n",
      "Size of 16th set is: (15000, 11)\n",
      "Size of 17th set is: (15000, 11)\n"
     ]
    }
   ],
   "source": [
    "# Shape of dataset\n",
    "print('Size of 1st set is:', tweets_1st.shape)\n",
    "print('Size of 2nd set is:', tweets_2nd.shape)\n",
    "print('Size of 3rd set is:', tweets_3rd.shape)\n",
    "print('Size of 4th set is:', tweets_4th.shape)\n",
    "print('Size of 5th set is:', tweets_5th.shape)\n",
    "print('Size of 6th set is:', tweets_6th.shape)\n",
    "print('Size of 7th set is:', tweets_7th.shape)\n",
    "print('Size of 8th set is:', tweets_8th.shape)\n",
    "print('Size of 9th set is:', tweets_9th.shape)\n",
    "print('Size of 10th set is:', tweets_10th.shape)\n",
    "print('Size of 11th set is:', tweets_11th.shape)\n",
    "print('Size of 12th set is:', tweets_12th.shape)\n",
    "print('Size of 13th set is:', tweets_13th.shape)\n",
    "print('Size of 14th set is:', tweets_14th.shape)\n",
    "print('Size of 15th set is:', tweets_15th.shape)\n",
    "print('Size of 16th set is:', tweets_16th.shape)\n",
    "print('Size of 17th set is:', tweets_17th.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10515 non-null  object\n",
      " 2   location        5694 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10566 non-null  object\n",
      " 2   location        5909 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10941 non-null  object\n",
      " 2   location        5770 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15001 entries, 0 to 15000\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   username        15001 non-null  object \n",
      " 1   acctdesc        11491 non-null  object \n",
      " 2   location        6577 non-null   object \n",
      " 3   following       15000 non-null  float64\n",
      " 4   followers       15000 non-null  float64\n",
      " 5   totaltweets     15000 non-null  object \n",
      " 6   usercreatedts   15000 non-null  object \n",
      " 7   tweetcreatedts  15000 non-null  object \n",
      " 8   retweetcount    15000 non-null  object \n",
      " 9   text            15000 non-null  object \n",
      " 10  hashtags        14999 non-null  object \n",
      "dtypes: float64(2), object(9)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10900 non-null  object\n",
      " 2   location        5730 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        11180 non-null  object\n",
      " 2   location        5663 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        11409 non-null  object\n",
      " 2   location        6077 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10803 non-null  object\n",
      " 2   location        5548 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15001 entries, 0 to 15000\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   username        15001 non-null  object \n",
      " 1   acctdesc        11594 non-null  object \n",
      " 2   location        5692 non-null   object \n",
      " 3   following       15000 non-null  float64\n",
      " 4   followers       15000 non-null  float64\n",
      " 5   totaltweets     15000 non-null  object \n",
      " 6   usercreatedts   15000 non-null  object \n",
      " 7   tweetcreatedts  15000 non-null  object \n",
      " 8   retweetcount    15000 non-null  object \n",
      " 9   text            15000 non-null  object \n",
      " 10  hashtags        14999 non-null  object \n",
      "dtypes: float64(2), object(9)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15001 entries, 0 to 15000\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   username        15001 non-null  object \n",
      " 1   acctdesc        10903 non-null  object \n",
      " 2   location        5903 non-null   object \n",
      " 3   following       15000 non-null  float64\n",
      " 4   followers       15000 non-null  float64\n",
      " 5   totaltweets     15000 non-null  object \n",
      " 6   usercreatedts   15000 non-null  object \n",
      " 7   tweetcreatedts  15000 non-null  object \n",
      " 8   retweetcount    15000 non-null  object \n",
      " 9   text            15000 non-null  object \n",
      " 10  hashtags        14999 non-null  object \n",
      "dtypes: float64(2), object(9)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10368 non-null  object\n",
      " 2   location        5799 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        9964 non-null   object\n",
      " 2   location        5297 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        9402 non-null   object\n",
      " 2   location        4419 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10628 non-null  object\n",
      " 2   location        4111 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        11536 non-null  object\n",
      " 2   location        5027 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        11995 non-null  object\n",
      " 2   location        5322 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n",
      "\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   username        15000 non-null  object\n",
      " 1   acctdesc        10879 non-null  object\n",
      " 2   location        5528 non-null   object\n",
      " 3   following       15000 non-null  int64 \n",
      " 4   followers       15000 non-null  int64 \n",
      " 5   totaltweets     15000 non-null  int64 \n",
      " 6   usercreatedts   15000 non-null  object\n",
      " 7   tweetcreatedts  15000 non-null  object\n",
      " 8   retweetcount    15000 non-null  int64 \n",
      " 9   text            15000 non-null  object\n",
      " 10  hashtags        15000 non-null  object\n",
      "dtypes: int64(4), object(7)\n",
      "memory usage: 1.3+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Summary statistics\n",
    "print(tweets_1st.info())\n",
    "print('\\n')\n",
    "print(tweets_2nd.info())\n",
    "print('\\n')\n",
    "print(tweets_3rd.info())\n",
    "print('\\n')\n",
    "print(tweets_4th.info())\n",
    "print('\\n')\n",
    "print(tweets_5th.info())\n",
    "print('\\n')\n",
    "print(tweets_6th.info())\n",
    "print('\\n')\n",
    "print(tweets_7th.info())\n",
    "print('\\n')\n",
    "print(tweets_8th.info())\n",
    "print('\\n')\n",
    "print(tweets_9th.info())\n",
    "print('\\n')\n",
    "print(tweets_10th.info())\n",
    "print('\\n')\n",
    "print(tweets_11th.info())\n",
    "print('\\n')\n",
    "print(tweets_12th.info())\n",
    "print('\\n')\n",
    "print(tweets_13th.info())\n",
    "print('\\n')\n",
    "print(tweets_14th.info())\n",
    "print('\\n')\n",
    "print(tweets_15th.info())\n",
    "print('\\n')\n",
    "print(tweets_16th.info())\n",
    "print('\\n')\n",
    "print(tweets_17th.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of concatenated dataset is: (255003, 11)\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 255003 entries, 0 to 255002\n",
      "Data columns (total 11 columns):\n",
      " #   Column          Non-Null Count   Dtype  \n",
      "---  ------          --------------   -----  \n",
      " 0   username        255003 non-null  object \n",
      " 1   acctdesc        185074 non-null  object \n",
      " 2   location        94066 non-null   object \n",
      " 3   following       255000 non-null  float64\n",
      " 4   followers       255000 non-null  float64\n",
      " 5   totaltweets     255000 non-null  object \n",
      " 6   usercreatedts   255000 non-null  object \n",
      " 7   tweetcreatedts  255000 non-null  object \n",
      " 8   retweetcount    255000 non-null  object \n",
      " 9   text            255000 non-null  object \n",
      " 10  hashtags        254997 non-null  object \n",
      "dtypes: float64(2), object(9)\n",
      "memory usage: 21.4+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Concat the two dataset together:\n",
    "data = pd.concat([tweets_1st, tweets_2nd, tweets_3rd, tweets_4th, tweets_5th, tweets_6th, tweets_7th,\n",
    "                  tweets_8th, tweets_9th, tweets_10th, tweets_11th, tweets_12th, tweets_13th, tweets_14th,\n",
    "                 tweets_15th, tweets_16th, tweets_17th], axis = 0)\n",
    "\n",
    "print('Size of concatenated dataset is:', data.shape)\n",
    "\n",
    "# Reset_index\n",
    "data.reset_index(inplace = True, drop = True)\n",
    "data.head()\n",
    "print(data.info())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking for Duplicated Entries and Removing Them\n",
    "Since we are performing the scraping close to each other, it is possible to scrape the same tweets as long as they fall within the search window of 7 days from the search_date.\n",
    "We will remove these duplicated rows from our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "username\n",
       "xzxzanalazy        651\n",
       "belleng324         540\n",
       "wdamidoinhere      399\n",
       "natalie_hoyin      286\n",
       "hky2147            264\n",
       "                  ... \n",
       "evangelion01241      1\n",
       "ST87699605           1\n",
       "Alice_Man_Man        1\n",
       "littletaekook01      1\n",
       "Candy96023268        1\n",
       "Name: count, Length: 51955, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out the number of unique users in the dataset\n",
    "# It can be seen that some users are more proactive on the social media platform than others.\n",
    "data['username'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10271</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>418</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-03 04:24:41</td>\n",
       "      <td>12</td>\n",
       "      <td>@JoachimWatson @Treerisu @SolomonYue @HawleyMO @amnesty @AmnestyUK @amnestyusa @BonnieGlaser @cafreeland \\n\\nAnyone can save these girls?\\n\\n#StandWithHongKong https://t.co/okzc6lTEyz</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10292</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>418</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-03 04:24:33</td>\n",
       "      <td>2</td>\n",
       "      <td>@JoachimWatson @SolomonYue @HawleyMO SOS! Please save our kids 🙏\\n#StandWithHongKong https://t.co/gdQdiTwBzD</td>\n",
       "      <td>[{'text': 'StandWithHongKong', 'indices': [86, 104]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10436</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>418</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-03 04:23:16</td>\n",
       "      <td>15</td>\n",
       "      <td>More &amp;amp; more cases like Hoi-lam Chan will happen if #hkpolice aren't held in control and international supervision.\\n\\n#HKPoliceState #HKPoliceTerrorism https://t.co/Dfi7rnd28T</td>\n",
       "      <td>[{'text': 'hkpolice', 'indices': [74, 83]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10463</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>418</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-03 04:23:04</td>\n",
       "      <td>34</td>\n",
       "      <td>This clip is a bit old but this happens on every protests. I sincerely don't want to believe what happens behind any cameras or press, in the detention centers/ police stations, but the answer is pretty straight-forward...\\n\\n#HKPolice #HKPoliceState https://t.co/7Uft3CNV7B</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10549</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>418</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-03 04:22:28</td>\n",
       "      <td>2818</td>\n",
       "      <td>Arrestees ALL GIRLS. \\nHard to believe they're not caught to serve the cops/PLAs in #HK especially when that's happening in #Xinjiang, and there're so many sexual abuse/ra_e allegations against #hkpolice.\\n\\n#HongKongProtests\\n#HKPoliceState\\n#HKPoliceTerrorism \\n@SolomonYue @HawleyMO https://t.co/2wX803P3lH</td>\n",
       "      <td>[{'text': 'HK', 'indices': [102, 105]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71978</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1112</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-06 14:53:28</td>\n",
       "      <td>2</td>\n",
       "      <td>@SenRickScott Other than interest, #ccp and its puppet #CarrieLam are restricting hongkongers to restore the human rights and universal suffrage, Senator Scott\\n#HongKongHumanRightsAndDemocracyAct \\n#HKHumanRightsandDemocracyAct \\n#StandWithHongKong</td>\n",
       "      <td>[{'text': 'ccp', 'indices': [56, 60]}, {'text': 'CarrieLam', 'indices': [76, 86]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72020</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1112</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-06 14:52:52</td>\n",
       "      <td>590</td>\n",
       "      <td>“The UK Government says that it is “fully committed to upholding Hong Kong’s high degree of autonomy and its rights and freedoms as enshrined in the ‘one country, two systems’ framework”. Where, though, is the evidence for this?” @agcolehamilton #standwithHK https://t.co/b0wClOORsC</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72062</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1112</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-06 14:52:07</td>\n",
       "      <td>378</td>\n",
       "      <td>Top Pro-CCP figure in #HongKong, Junius Ho was attacked and mildly injured.\\n\\nMy guess: Staged.\\nNicely filmed, ineffective stab, &amp;amp; very quick response to subdue.\\n\\nJust a pretext to cancel the upcoming District Council Election which they know they'll lose.\\nhttps://t.co/ZwsbFrqYQM</td>\n",
       "      <td>[{'text': 'HongKong', 'indices': [39, 48]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72153</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1112</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-06 14:50:59</td>\n",
       "      <td>132</td>\n",
       "      <td>Just like the pro-Beijing HK lawmaker Junius Ho #何君堯, my friend Jenital Ho was attacked today as well！\\nBut unlike Mr Junius，Mr Jenital was much more unfortunate. Because he has to wash off all the ketchup by himself.\\n#StandWithHongKong #FreedomHK #HKprotests https://t.co/dDJSILprx1</td>\n",
       "      <td>[{'text': '何君堯', 'indices': [62, 66]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72334</th>\n",
       "      <td>Phy32833861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1112</td>\n",
       "      <td>2019-09-28 03:09:49</td>\n",
       "      <td>2019-11-06 14:48:53</td>\n",
       "      <td>3506</td>\n",
       "      <td>#hongkong is this generation’s Berlin. It represents the struggle we face with China to preserve our economy, our jobs, and our security. From @tiktok_us to the @NBA, China is trying to export its values &amp;amp; its censorship to America. We have to stand up #StandWithHongKong https://t.co/iKEUMSfvsZ</td>\n",
       "      <td>[{'text': 'hongkong', 'indices': [14, 23]}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>166 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          username acctdesc location  following  followers totaltweets  \\\n",
       "10271  Phy32833861      NaN      NaN        7.0        7.0         418   \n",
       "10292  Phy32833861      NaN      NaN        7.0        7.0         418   \n",
       "10436  Phy32833861      NaN      NaN        7.0        7.0         418   \n",
       "10463  Phy32833861      NaN      NaN        7.0        7.0         418   \n",
       "10549  Phy32833861      NaN      NaN        7.0        7.0         418   \n",
       "...            ...      ...      ...        ...        ...         ...   \n",
       "71978  Phy32833861      NaN      NaN       11.0        9.0        1112   \n",
       "72020  Phy32833861      NaN      NaN       11.0        9.0        1112   \n",
       "72062  Phy32833861      NaN      NaN       11.0        9.0        1112   \n",
       "72153  Phy32833861      NaN      NaN       11.0        9.0        1112   \n",
       "72334  Phy32833861      NaN      NaN       11.0        9.0        1112   \n",
       "\n",
       "             usercreatedts       tweetcreatedts retweetcount  \\\n",
       "10271  2019-09-28 03:09:49  2019-11-03 04:24:41           12   \n",
       "10292  2019-09-28 03:09:49  2019-11-03 04:24:33            2   \n",
       "10436  2019-09-28 03:09:49  2019-11-03 04:23:16           15   \n",
       "10463  2019-09-28 03:09:49  2019-11-03 04:23:04           34   \n",
       "10549  2019-09-28 03:09:49  2019-11-03 04:22:28         2818   \n",
       "...                    ...                  ...          ...   \n",
       "71978  2019-09-28 03:09:49  2019-11-06 14:53:28            2   \n",
       "72020  2019-09-28 03:09:49  2019-11-06 14:52:52          590   \n",
       "72062  2019-09-28 03:09:49  2019-11-06 14:52:07          378   \n",
       "72153  2019-09-28 03:09:49  2019-11-06 14:50:59          132   \n",
       "72334  2019-09-28 03:09:49  2019-11-06 14:48:53         3506   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                        text  \\\n",
       "10271                                                                                                                                @JoachimWatson @Treerisu @SolomonYue @HawleyMO @amnesty @AmnestyUK @amnestyusa @BonnieGlaser @cafreeland \\n\\nAnyone can save these girls?\\n\\n#StandWithHongKong https://t.co/okzc6lTEyz   \n",
       "10292                                                                                                                                                                                                           @JoachimWatson @SolomonYue @HawleyMO SOS! Please save our kids 🙏\\n#StandWithHongKong https://t.co/gdQdiTwBzD   \n",
       "10436                                                                                                                                    More &amp; more cases like Hoi-lam Chan will happen if #hkpolice aren't held in control and international supervision.\\n\\n#HKPoliceState #HKPoliceTerrorism https://t.co/Dfi7rnd28T   \n",
       "10463                                     This clip is a bit old but this happens on every protests. I sincerely don't want to believe what happens behind any cameras or press, in the detention centers/ police stations, but the answer is pretty straight-forward...\\n\\n#HKPolice #HKPoliceState https://t.co/7Uft3CNV7B   \n",
       "10549  Arrestees ALL GIRLS. \\nHard to believe they're not caught to serve the cops/PLAs in #HK especially when that's happening in #Xinjiang, and there're so many sexual abuse/ra_e allegations against #hkpolice.\\n\\n#HongKongProtests\\n#HKPoliceState\\n#HKPoliceTerrorism \\n@SolomonYue @HawleyMO https://t.co/2wX803P3lH   \n",
       "...                                                                                                                                                                                                                                                                                                                      ...   \n",
       "71978                                                              @SenRickScott Other than interest, #ccp and its puppet #CarrieLam are restricting hongkongers to restore the human rights and universal suffrage, Senator Scott\\n#HongKongHumanRightsAndDemocracyAct \\n#HKHumanRightsandDemocracyAct \\n#StandWithHongKong   \n",
       "72020                             “The UK Government says that it is “fully committed to upholding Hong Kong’s high degree of autonomy and its rights and freedoms as enshrined in the ‘one country, two systems’ framework”. Where, though, is the evidence for this?” @agcolehamilton #standwithHK https://t.co/b0wClOORsC   \n",
       "72062                      Top Pro-CCP figure in #HongKong, Junius Ho was attacked and mildly injured.\\n\\nMy guess: Staged.\\nNicely filmed, ineffective stab, &amp; very quick response to subdue.\\n\\nJust a pretext to cancel the upcoming District Council Election which they know they'll lose.\\nhttps://t.co/ZwsbFrqYQM   \n",
       "72153                           Just like the pro-Beijing HK lawmaker Junius Ho #何君堯, my friend Jenital Ho was attacked today as well！\\nBut unlike Mr Junius，Mr Jenital was much more unfortunate. Because he has to wash off all the ketchup by himself.\\n#StandWithHongKong #FreedomHK #HKprotests https://t.co/dDJSILprx1   \n",
       "72334            #hongkong is this generation’s Berlin. It represents the struggle we face with China to preserve our economy, our jobs, and our security. From @tiktok_us to the @NBA, China is trying to export its values &amp; its censorship to America. We have to stand up #StandWithHongKong https://t.co/iKEUMSfvsZ   \n",
       "\n",
       "                                                                                 hashtags  \n",
       "10271                                                                                  []  \n",
       "10292                               [{'text': 'StandWithHongKong', 'indices': [86, 104]}]  \n",
       "10436                                         [{'text': 'hkpolice', 'indices': [74, 83]}]  \n",
       "10463                                                                                  []  \n",
       "10549                                             [{'text': 'HK', 'indices': [102, 105]}]  \n",
       "...                                                                                   ...  \n",
       "71978  [{'text': 'ccp', 'indices': [56, 60]}, {'text': 'CarrieLam', 'indices': [76, 86]}]  \n",
       "72020                                                                                  []  \n",
       "72062                                         [{'text': 'HongKong', 'indices': [39, 48]}]  \n",
       "72153                                              [{'text': '何君堯', 'indices': [62, 66]}]  \n",
       "72334                                         [{'text': 'hongkong', 'indices': [14, 23]}]  \n",
       "\n",
       "[166 rows x 11 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# It is acceptable to have repeated username since they can tweet multiple times throughout the day.\n",
    "data[data['username'] == 'Phy32833861']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial size of dataset before dropping duplicated rows: (255003, 11)\n"
     ]
    }
   ],
   "source": [
    "# Let's drop duplicated rows:\n",
    "print('Initial size of dataset before dropping duplicated rows:', data.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current size of dataset after dropping duplicated rows, if any, is: (248652, 11)\n"
     ]
    }
   ],
   "source": [
    "data.drop_duplicates(keep = False, inplace = True)\n",
    "print('Current size of dataset after dropping duplicated rows, if any, is:', data.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          username  \\\n",
      "0     five5demands   \n",
      "1   Dejavu53328974   \n",
      "2         tsksimon   \n",
      "3  JimmyWo67187904   \n",
      "4         ARCHI418   \n",
      "\n",
      "                                                                                        acctdesc  \\\n",
      "0                                            HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
      "1  #followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters   \n",
      "2                                                            #FightForFreedom #StandWithHongKong   \n",
      "3                                                                                            NaN   \n",
      "4                                                                                            NaN   \n",
      "\n",
      "    location  following  followers totaltweets        usercreatedts  \\\n",
      "0        NaN      437.0      260.0         466  2019-10-05 13:40:37   \n",
      "1  Hong Kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
      "2        NaN      825.0      456.0        6728  2014-10-05 15:57:07   \n",
      "3        NaN       22.0        1.0         656  2019-10-08 12:58:51   \n",
      "4        NaN       19.0       11.0         121  2014-06-19 16:19:36   \n",
      "\n",
      "        tweetcreatedts retweetcount  \\\n",
      "0  2019-11-03 02:57:49          292   \n",
      "1  2019-11-03 02:57:49         1315   \n",
      "2  2019-11-03 02:57:49          105   \n",
      "3  2019-11-03 02:57:48          660   \n",
      "4  2019-11-03 02:57:46            0   \n",
      "\n",
      "                                                                                                                                                                                                                                                                                                               text  \\\n",
      "0                                                                                                                                         Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
      "1                                                                                                    #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
      "2  This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
      "3       A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
      "4                                                                              #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
      "\n",
      "                                                                                                                                                                                                                                                                                                                   hashtags  \n",
      "0                                                                                                                                                                                                                                                                               [{'text': 'HongKong', 'indices': [36, 45]}]  \n",
      "1                                                                                                                                                                                                                                  [{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]  \n",
      "2                                                                                                                                                                                                                                                                      [{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]  \n",
      "3                                                                                                                                                                                                                                                                         [{'text': 'HongKongPolice', 'indices': [41, 56]}]  \n",
      "4  [{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]  \n"
     ]
    }
   ],
   "source": [
    "print(data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Non-English Words/Tokens\n",
    "\n",
    "Since it might be possible to remove non-english words that are used in daily english conversations such as names etc, it might be better to filter by the chinese language.\n",
    "\n",
    "##### Remove empty tweets first! \n",
    "If there are empty tweets or 'nan', the following codes will break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dropna(subset = ['text'], inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The unicode accounts for chinese characters and punctuations.\n",
    "def strip_chinese_words(string):\n",
    "    # list of english words\n",
    "    en_list = re.findall(u'[^\\u4E00-\\u9FA5\\u3000-\\u303F]', str(string))\n",
    "    \n",
    "    # Remove word from the list, if not english\n",
    "    for c in string:\n",
    "        if c not in en_list:\n",
    "            string = string.replace(c, '')\n",
    "    return string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>NaN</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[{'text': 'HongKong', 'indices': [36, 45]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#FightForFreedom #StandWithHongKong</td>\n",
       "      <td>NaN</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[{'text': 'HongKongPolice', 'indices': [41, 56]}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          username  \\\n",
       "0     five5demands   \n",
       "1   Dejavu53328974   \n",
       "2         tsksimon   \n",
       "3  JimmyWo67187904   \n",
       "4         ARCHI418   \n",
       "\n",
       "                                                                                        acctdesc  \\\n",
       "0                                            HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1  #followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                            #FightForFreedom #StandWithHongKong   \n",
       "3                                                                                            NaN   \n",
       "4                                                                                            NaN   \n",
       "\n",
       "    location  following  followers totaltweets        usercreatedts  \\\n",
       "0        NaN      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1  Hong Kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2        NaN      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3        NaN       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4        NaN       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "\n",
       "        tweetcreatedts retweetcount  \\\n",
       "0  2019-11-03 02:57:49          292   \n",
       "1  2019-11-03 02:57:49         1315   \n",
       "2  2019-11-03 02:57:49          105   \n",
       "3  2019-11-03 02:57:48          660   \n",
       "4  2019-11-03 02:57:46            0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                               text  \\\n",
       "0                                                                                                                                         Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                    #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2  This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3       A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                              #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                   hashtags  \n",
       "0                                                                                                                                                                                                                                                                               [{'text': 'HongKong', 'indices': [36, 45]}]  \n",
       "1                                                                                                                                                                                                                                  [{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]  \n",
       "2                                                                                                                                                                                                                                                                      [{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]  \n",
       "3                                                                                                                                                                                                                                                                         [{'text': 'HongKongPolice', 'indices': [41, 56]}]  \n",
       "4  [{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply strip_chinese_words(...) on the column 'text'\n",
    "data['text'] = data['text'].apply(lambda x: strip_chinese_words(x))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect @Users mentioned in each tweet\n",
    "\n",
    "We want to obtain this useful information from each tweet because it could allow us to analyse what are the popular figures in the protest movement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>NaN</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[{'text': 'HongKong', 'indices': [36, 45]}]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#FightForFreedom #StandWithHongKong</td>\n",
       "      <td>NaN</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[{'text': 'HongKongPolice', 'indices': [41, 56]}]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          username  \\\n",
       "0     five5demands   \n",
       "1   Dejavu53328974   \n",
       "2         tsksimon   \n",
       "3  JimmyWo67187904   \n",
       "4         ARCHI418   \n",
       "\n",
       "                                                                                        acctdesc  \\\n",
       "0                                            HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1  #followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                            #FightForFreedom #StandWithHongKong   \n",
       "3                                                                                            NaN   \n",
       "4                                                                                            NaN   \n",
       "\n",
       "    location  following  followers totaltweets        usercreatedts  \\\n",
       "0        NaN      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1  Hong Kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2        NaN      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3        NaN       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4        NaN       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "\n",
       "        tweetcreatedts retweetcount  \\\n",
       "0  2019-11-03 02:57:49          292   \n",
       "1  2019-11-03 02:57:49         1315   \n",
       "2  2019-11-03 02:57:49          105   \n",
       "3  2019-11-03 02:57:48          660   \n",
       "4  2019-11-03 02:57:46            0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                               text  \\\n",
       "0                                                                                                                                         Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                    #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2  This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3       A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                              #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                   hashtags  \\\n",
       "0                                                                                                                                                                                                                                                                               [{'text': 'HongKong', 'indices': [36, 45]}]   \n",
       "1                                                                                                                                                                                                                                  [{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]   \n",
       "2                                                                                                                                                                                                                                                                      [{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]   \n",
       "3                                                                                                                                                                                                                                                                         [{'text': 'HongKongPolice', 'indices': [41, 56]}]   \n",
       "4  [{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]   \n",
       "\n",
       "  mentioned_users  \n",
       "0              []  \n",
       "1       [@hkgetv]  \n",
       "2              []  \n",
       "3              []  \n",
       "4              []  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define function to sieve out @users in a tweet:\n",
    "def mentioned_users(string):\n",
    "    usernames = re.findall('@[^\\s]+', string)\n",
    "    return usernames\n",
    "\n",
    "\n",
    "# Create a new column and apply the function on the column 'text'\n",
    "data['mentioned_users'] = data['text'].apply(lambda x: mentioned_users(x))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main Text Cleaning\n",
    "We had to remove chinese words and @users from our tweets first before we do the usual text cleaning and processing. Otherwise, we will not be able to obtain these information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define Emoji_patterns\n",
    "emoji_pattern = re.compile(\"[\"\n",
    "         u\"\\U0001F600-\\U0001F64F\"  # emoticons\n",
    "         u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n",
    "         u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n",
    "         u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n",
    "         u\"\\U00002702-\\U000027B0\"\n",
    "         u\"\\U000024C2-\\U0001F251\"\n",
    "         \"]+\", flags=re.UNICODE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the function to implement POS tagging:\n",
    "def get_wordnet_pos(pos_tag):\n",
    "    if pos_tag.startswith('J'):\n",
    "        return wordnet.ADJ\n",
    "    elif pos_tag.startswith('V'):\n",
    "        return wordnet.VERB\n",
    "    elif pos_tag.startswith('N'):\n",
    "        return wordnet.NOUN\n",
    "    elif pos_tag.startswith('R'):\n",
    "        return wordnet.ADV\n",
    "    else:\n",
    "        return wordnet.NOUN\n",
    "\n",
    "\n",
    "# Define the main function to clean text in various ways:\n",
    "def clean_text(text):\n",
    "    \n",
    "    # Apply regex expressions first before converting string to list of tokens/words:\n",
    "    # 1. remove @usernames\n",
    "    text = re.sub('@[^\\s]+', '', text)\n",
    "    \n",
    "    # 2. remove URLs\n",
    "    text = re.sub('((www\\.[^\\s]+)|(https?://[^\\s]+))', '', text)\n",
    "    \n",
    "    # 3. remove hashtags entirely i.e. #hashtags\n",
    "    text = re.sub(r'#([^\\s]+)', '', text)\n",
    "    \n",
    "    # 4. remove emojis\n",
    "    text = emoji_pattern.sub(r'', text)\n",
    "    \n",
    "    # 5. Convert text to lowercase\n",
    "    text = text.lower()\n",
    "    \n",
    "    # 6. tokenize text and remove punctuation\n",
    "    text = [word.strip(string.punctuation) for word in text.split(\" \")]\n",
    "    \n",
    "    # 7. remove numbers\n",
    "    text = [word for word in text if not any(c.isdigit() for c in word)]\n",
    "    \n",
    "    # 8. remove stop words\n",
    "    stop = stopwords.words('english')\n",
    "    text = [x for x in text if x not in stop]\n",
    "    \n",
    "    # 9. remove empty tokens\n",
    "    text = [t for t in text if len(t) > 0]\n",
    "    \n",
    "    # 10. pos tag text and lemmatize text\n",
    "    pos_tags = pos_tag(text)\n",
    "    text = [WordNetLemmatizer().lemmatize(t[0], get_wordnet_pos(t[1])) for t in pos_tags]\n",
    "    \n",
    "    # 11. remove words with only one letter\n",
    "    text = [t for t in text if len(t) > 1]\n",
    "    \n",
    "    # join all\n",
    "    text = \" \".join(text)\n",
    "    \n",
    "    return(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/shiruizhou/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>NaN</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[{'text': 'HongKong', 'indices': [36, 45]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#FightForFreedom #StandWithHongKong</td>\n",
       "      <td>NaN</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[{'text': 'HongKongPolice', 'indices': [41, 56]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          username  \\\n",
       "0     five5demands   \n",
       "1   Dejavu53328974   \n",
       "2         tsksimon   \n",
       "3  JimmyWo67187904   \n",
       "4         ARCHI418   \n",
       "\n",
       "                                                                                        acctdesc  \\\n",
       "0                                            HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1  #followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                            #FightForFreedom #StandWithHongKong   \n",
       "3                                                                                            NaN   \n",
       "4                                                                                            NaN   \n",
       "\n",
       "    location  following  followers totaltweets        usercreatedts  \\\n",
       "0        NaN      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1  Hong Kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2        NaN      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3        NaN       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4        NaN       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "\n",
       "        tweetcreatedts retweetcount  \\\n",
       "0  2019-11-03 02:57:49          292   \n",
       "1  2019-11-03 02:57:49         1315   \n",
       "2  2019-11-03 02:57:49          105   \n",
       "3  2019-11-03 02:57:48          660   \n",
       "4  2019-11-03 02:57:46            0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                               text  \\\n",
       "0                                                                                                                                         Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                    #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2  This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3       A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                              #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                   hashtags  \\\n",
       "0                                                                                                                                                                                                                                                                               [{'text': 'HongKong', 'indices': [36, 45]}]   \n",
       "1                                                                                                                                                                                                                                  [{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]   \n",
       "2                                                                                                                                                                                                                                                                      [{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]   \n",
       "3                                                                                                                                                                                                                                                                         [{'text': 'HongKongPolice', 'indices': [41, 56]}]   \n",
       "4  [{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]   \n",
       "\n",
       "  mentioned_users  \\\n",
       "0              []   \n",
       "1       [@hkgetv]   \n",
       "2              []   \n",
       "3              []   \n",
       "4              []   \n",
       "\n",
       "                                                                                                                                                 cleaned_text  \n",
       "0                                                                                      disgust police mark protestors pen write number hand like nazi germany  \n",
       "1                                                                                                 teargas canister stick road citizen try pull failed\\ncredit  \n",
       "2       tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence  \n",
       "3  fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away  \n",
       "4                                                                        threw hand blast citizen’s back make large area burn tear gas read hand grenade kill  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply function on the column 'text':\n",
    "data['cleaned_text'] = data['text'].apply(lambda x: clean_text(x))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(248651, 13)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>Five demands, not one less 🇭🇰#FollowBackHongKong #科勞手足</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #StandWithHongKong</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>Hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>NaN</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>Hongkonger 🇭🇰 Stand with Hong Kong! Fight for freedom! 香港人加油!! #FollowBackHongKong #FiveDemandsNotOneLess #StandwithHongKong #科勞手足 #FightForFreedom</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[{'text': 'SOSHK', 'indices': [74, 80]}, {'text': 'StandWithHongKong', 'indices': [81, 99]}, {'text': 'HongKongProtests', 'indices': [101, 118]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>NaN</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "248646                                                                                               Five demands, not one less 🇭🇰#FollowBackHongKong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #StandWithHongKong   \n",
       "248648                                                                                        Hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  Hongkonger 🇭🇰 Stand with Hong Kong! Fight for freedom! 香港人加油!! #FollowBackHongKong #FiveDemandsNotOneLess #StandwithHongKong #科勞手足 #FightForFreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers totaltweets        usercreatedts  \\\n",
       "248646  Hong Kong      505.0      326.0        1301  2019-09-25 17:05:21   \n",
       "248647  Hong Kong       15.0      155.0         116  2019-09-12 06:32:09   \n",
       "248648        NaN      248.0      170.0         362  2019-09-28 14:29:13   \n",
       "248649  Hong Kong     4653.0     2490.0        4627  2015-03-09 06:57:20   \n",
       "248650        NaN      348.0       45.0        5042  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts retweetcount  \\\n",
       "248646  2019-11-19 17:19:37           72   \n",
       "248647  2019-11-19 17:19:36          796   \n",
       "248648  2019-11-19 17:19:36         1162   \n",
       "248649  2019-11-19 17:19:36            9   \n",
       "248650  2019-11-19 17:19:35           10   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                                                                                                 hashtags  \\\n",
       "248646                                                                                                                                                 []   \n",
       "248647                                                                                                                                                 []   \n",
       "248648                                                                                                                                                 []   \n",
       "248649  [{'text': 'SOSHK', 'indices': [74, 80]}, {'text': 'StandWithHongKong', 'indices': [81, 99]}, {'text': 'HongKongProtests', 'indices': [101, 118]}]   \n",
       "248650                                                                                                                                                 []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy  \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom  \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read  \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n  \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out the shape again and reset_index\n",
    "print(data.shape)\n",
    "data.reset_index(inplace = True, drop = True)\n",
    "\n",
    "# Check out data.tail() to validate index has been reset\n",
    "data.tail()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process the Column 'hashtags'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import ast to convert a string representation of list to list\n",
    "# The column 'hashtags' is affected\n",
    "import ast\n",
    "\n",
    "# Define a function to convert a string rep. of list to list\n",
    "## Function should also handle NaN values after conversion\n",
    "def strlist_to_list(text):\n",
    "    \n",
    "    # Remove NaN\n",
    "    if pd.isnull(text) == True: # if true\n",
    "        text = ''\n",
    "    else:\n",
    "        text = ast.literal_eval(text)\n",
    "    \n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>NaN</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[{'text': 'HongKong', 'indices': [36, 45]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>Hong Kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#FightForFreedom #StandWithHongKong</td>\n",
       "      <td>NaN</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[{'text': 'HongKongPolice', 'indices': [41, 56]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          username  \\\n",
       "0     five5demands   \n",
       "1   Dejavu53328974   \n",
       "2         tsksimon   \n",
       "3  JimmyWo67187904   \n",
       "4         ARCHI418   \n",
       "\n",
       "                                                                                        acctdesc  \\\n",
       "0                                            HKer Fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1  #followbackhongkong\\n#StandwithHK\\n#HKHumanRightsandDemocracyAct\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                            #FightForFreedom #StandWithHongKong   \n",
       "3                                                                                            NaN   \n",
       "4                                                                                            NaN   \n",
       "\n",
       "    location  following  followers totaltweets        usercreatedts  \\\n",
       "0        NaN      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1  Hong Kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2        NaN      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3        NaN       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4        NaN       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "\n",
       "        tweetcreatedts retweetcount  \\\n",
       "0  2019-11-03 02:57:49          292   \n",
       "1  2019-11-03 02:57:49         1315   \n",
       "2  2019-11-03 02:57:49          105   \n",
       "3  2019-11-03 02:57:48          660   \n",
       "4  2019-11-03 02:57:46            0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                               text  \\\n",
       "0                                                                                                                                         Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                    #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2  This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3       A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                              #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                   hashtags  \\\n",
       "0                                                                                                                                                                                                                                                                               [{'text': 'HongKong', 'indices': [36, 45]}]   \n",
       "1                                                                                                                                                                                                                                  [{'text': 'China', 'indices': [11, 17]}, {'text': 'PoliceState', 'indices': [118, 130]}]   \n",
       "2                                                                                                                                                                                                                                                                      [{'text': 'HKPoliceBrutality', 'indices': [42, 60]}]   \n",
       "3                                                                                                                                                                                                                                                                         [{'text': 'HongKongPolice', 'indices': [41, 56]}]   \n",
       "4  [{'text': 'police', 'indices': [0, 7]}, {'text': 'Teargas', 'indices': [16, 24]}, {'text': 'grenade', 'indices': [30, 38]}, {'text': 'hkger', 'indices': [155, 161]}, {'text': 'PoliceState', 'indices': [163, 175]}, {'text': 'PoliceTerrorism', 'indices': [176, 192]}, {'text': 'HKprotests', 'indices': [193, 204]}]   \n",
       "\n",
       "  mentioned_users  \\\n",
       "0              []   \n",
       "1       [@hkgetv]   \n",
       "2              []   \n",
       "3              []   \n",
       "4              []   \n",
       "\n",
       "                                                                                                                                                 cleaned_text  \n",
       "0                                                                                      disgust police mark protestors pen write number hand like nazi germany  \n",
       "1                                                                                                 teargas canister stick road citizen try pull failed\\ncredit  \n",
       "2       tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence  \n",
       "3  fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away  \n",
       "4                                                                        threw hand blast citizen’s back make large area burn tear gas read hand grenade kill  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply strlist_to_list(...) to the column 'hashtags'\n",
    "# Note that doing so will return a list of dictionaries, where there will be one dictionary for each hashtag in a single tweet.\n",
    "data['hashtags'] = data['hashtags'].apply(lambda x: strlist_to_list(x))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since each 'hashtag' contain a list of dictionaries, we need to loop through the list to extract the actual hashtags in the tweets.\n",
    "# Define a function to perform this extraction:\n",
    "def extract_hashtags(hashtag_list):\n",
    "    # argument:\n",
    "    # hashtag_list - a list of dictionary(ies), each containing a hashtag\n",
    "    \n",
    "    # Create a list to store the hashtags\n",
    "    hashtags = []\n",
    "    \n",
    "    # Loop through the list:\n",
    "    for i in range(0, len(hashtag_list)):\n",
    "        # extract the hashtag value using the key - 'text'\n",
    "        # For our purposes, we can ignore the indices, which tell us the position of the hashtags in the string of tweet\n",
    "        # lowercase the text as well\n",
    "        hashtags.append(hashtag_list[i]['text'].lower())\n",
    "        \n",
    "    return hashtags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                                                                     [hongkong]\n",
      "1                                                           [china, policestate]\n",
      "2                                                            [hkpolicebrutality]\n",
      "3                                                               [hongkongpolice]\n",
      "4    [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]\n",
      "Name: hashtags, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Apply function on the column - data['hashtags']\n",
    "data['hashtags'] = data['hashtags'].apply(lambda x: extract_hashtags(x))\n",
    "\n",
    "# Check out the updated column 'hashtags'\n",
    "print(data.head()['hashtags'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning up the Column 'location'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace NaN (empty) values with n.a to indicate that the user did not state his location\n",
    "# Define a function to handle this:\n",
    "def remove_nan(text):\n",
    "    if pd.isnull(text) == True: # entry is NaN\n",
    "        text = 'n.a'\n",
    "    else:\n",
    "        # lowercase text for possible easy handling\n",
    "        text = text.lower()\n",
    "        \n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0          n.a\n",
      "1    hong kong\n",
      "2          n.a\n",
      "3          n.a\n",
      "4          n.a\n",
      "Name: location, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Apply function on column - data['location']\n",
    "data['location'] = data['location'].apply(lambda x: remove_nan(x))\n",
    "\n",
    "# Check out the updated columns\n",
    "print(data.head()['location'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "location\n",
       "n.a                  157316\n",
       "hong kong             49093\n",
       "香港                     6829\n",
       "united states          1058\n",
       "hk                      912\n",
       "                      ...  \n",
       "standing rock             1\n",
       "setúbal, portugal         1\n",
       "tarragona                 1\n",
       "^.^===-~                  1\n",
       "rio de janeiro 🇧🇷         1\n",
       "Name: count, Length: 6218, dtype: int64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a quick look at the value_counts()\n",
    "data['location'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning up the Column 'acctdesc'\n",
    "\n",
    "Likewise, we will clean up this column by removing NaN values and replacing them with 'n.a'.\n",
    "For now, this is all we do to this column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                                              hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk\n",
      "1    #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters\n",
      "2                                                              #fightforfreedom #standwithhongkong\n",
      "3                                                                                              n.a\n",
      "4                                                                                              n.a\n",
      "Name: acctdesc, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Apply the function already defined above: remove_nan(...)\n",
    "# Apply function on column - data['location']\n",
    "data['acctdesc'] = data['acctdesc'].apply(lambda x: remove_nan(x))\n",
    "\n",
    "# Check out the updated columns\n",
    "print(data.head()['acctdesc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>n.a</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[china, policestate]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#fightforfreedom #standwithhongkong</td>\n",
       "      <td>n.a</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[hkpolicebrutality]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>five demands, not one less 🇭🇰#followbackhongkong #科勞手足</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #standwithhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>n.a</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>248651 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "0          five5demands   \n",
       "1        Dejavu53328974   \n",
       "2              tsksimon   \n",
       "3       JimmyWo67187904   \n",
       "4              ARCHI418   \n",
       "...                 ...   \n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "0                                                                                                       hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1                                                             #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                                                                                       #fightforfreedom #standwithhongkong   \n",
       "3                                                                                                                                                       n.a   \n",
       "4                                                                                                                                                       n.a   \n",
       "...                                                                                                                                                     ...   \n",
       "248646                                                                                               five demands, not one less 🇭🇰#followbackhongkong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #standwithhongkong   \n",
       "248648                                                                                        hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers totaltweets        usercreatedts  \\\n",
       "0             n.a      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1       hong kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2             n.a      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3             n.a       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4             n.a       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "...           ...        ...        ...         ...                  ...   \n",
       "248646  hong kong      505.0      326.0        1301  2019-09-25 17:05:21   \n",
       "248647  hong kong       15.0      155.0         116  2019-09-12 06:32:09   \n",
       "248648        n.a      248.0      170.0         362  2019-09-28 14:29:13   \n",
       "248649  hong kong     4653.0     2490.0        4627  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0        5042  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts retweetcount  \\\n",
       "0       2019-11-03 02:57:49          292   \n",
       "1       2019-11-03 02:57:49         1315   \n",
       "2       2019-11-03 02:57:49          105   \n",
       "3       2019-11-03 02:57:48          660   \n",
       "4       2019-11-03 02:57:46            0   \n",
       "...                     ...          ...   \n",
       "248646  2019-11-19 17:19:37           72   \n",
       "248647  2019-11-19 17:19:36          796   \n",
       "248648  2019-11-19 17:19:36         1162   \n",
       "248649  2019-11-19 17:19:36            9   \n",
       "248650  2019-11-19 17:19:35           10   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "0                                                                                                                                                           Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                                      #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2                    This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3                         A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                                                #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "...                                                                                                                                                                                                                                                                                                                               ...   \n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                           hashtags  \\\n",
       "0                                                                        [hongkong]   \n",
       "1                                                              [china, policestate]   \n",
       "2                                                               [hkpolicebrutality]   \n",
       "3                                                                  [hongkongpolice]   \n",
       "4       [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]   \n",
       "...                                                                             ...   \n",
       "248646                                                                           []   \n",
       "248647                                                                           []   \n",
       "248648                                                                           []   \n",
       "248649                                 [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                                                           []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "0                               []   \n",
       "1                        [@hkgetv]   \n",
       "2                               []   \n",
       "3                               []   \n",
       "4                               []   \n",
       "...                            ...   \n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \n",
       "0                                                                                                                         disgust police mark protestors pen write number hand like nazi germany  \n",
       "1                                                                                                                                    teargas canister stick road citizen try pull failed\\ncredit  \n",
       "2                                          tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence  \n",
       "3                                     fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away  \n",
       "4                                                                                                           threw hand blast citizen’s back make large area burn tear gas read hand grenade kill  \n",
       "...                                                                                                                                                                                          ...  \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy  \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom  \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read  \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n  \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please  \n",
       "\n",
       "[248651 rows x 13 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering - Rule-based Word Processing\n",
    "So far, we have removed duplicated rows, extract important information such as hashtags, mentioned users and users' locations, and also cleaned up the tweets in the previous section. In the coming section, we will focus on Rule-based word processing for our sentiment analysis. We will postpone some exploratory data visualization till later once we have all the ingredients.\n",
    "\n",
    "### Generating Sentiments from Tweets with NLTK Vader_Lexicon Library\n",
    "We will be using the Vader_lexicon library from NLTK to generate sentiment for each review. Vader uses lexicon of words to determine which words in the reviews are positive or negative. It will return a set of 4 scores on the positivity, negativity, neutrality of a text, and also an overall score whether a text is positive or negative.\n",
    "1. Positivity - 'pos'\n",
    "2. Negativity - 'neg'\n",
    "3. Neutrality - 'neu'\n",
    "4. Overall Score - 'compound'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing VADER from NLTK\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "# Create a sid object called SentimentIntensityAnalyzer()\n",
    "sid = SentimentIntensityAnalyzer()\n",
    "\n",
    "# Apply polarity_score method of SentimentIntensityAnalyzer()\n",
    "data['sentiment'] = data['cleaned_text'].apply(lambda x: sid.polarity_scores(x))\n",
    "\n",
    "# Keep only the compound scores under the column 'Sentiment'\n",
    "data = pd.concat([data.drop(['sentiment'], axis = 1), data['sentiment'].apply(pd.Series)], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>n.a</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.2732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[china, policestate]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.543</td>\n",
       "      <td>0.202</td>\n",
       "      <td>-0.1779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#fightforfreedom #standwithhongkong</td>\n",
       "      <td>n.a</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[hkpolicebrutality]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "      <td>0.339</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.070</td>\n",
       "      <td>-0.8074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.498</td>\n",
       "      <td>0.137</td>\n",
       "      <td>-0.7430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.519</td>\n",
       "      <td>0.277</td>\n",
       "      <td>0.1779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>five demands, not one less 🇭🇰#followbackhongkong #科勞手足</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.5574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #standwithhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.305</td>\n",
       "      <td>-0.1265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>n.a</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.785</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.0772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "      <td>0.378</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.8176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>248651 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "0          five5demands   \n",
       "1        Dejavu53328974   \n",
       "2              tsksimon   \n",
       "3       JimmyWo67187904   \n",
       "4              ARCHI418   \n",
       "...                 ...   \n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "0                                                                                                       hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1                                                             #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                                                                                       #fightforfreedom #standwithhongkong   \n",
       "3                                                                                                                                                       n.a   \n",
       "4                                                                                                                                                       n.a   \n",
       "...                                                                                                                                                     ...   \n",
       "248646                                                                                               five demands, not one less 🇭🇰#followbackhongkong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #standwithhongkong   \n",
       "248648                                                                                        hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers totaltweets        usercreatedts  \\\n",
       "0             n.a      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1       hong kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2             n.a      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3             n.a       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4             n.a       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "...           ...        ...        ...         ...                  ...   \n",
       "248646  hong kong      505.0      326.0        1301  2019-09-25 17:05:21   \n",
       "248647  hong kong       15.0      155.0         116  2019-09-12 06:32:09   \n",
       "248648        n.a      248.0      170.0         362  2019-09-28 14:29:13   \n",
       "248649  hong kong     4653.0     2490.0        4627  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0        5042  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts retweetcount  \\\n",
       "0       2019-11-03 02:57:49          292   \n",
       "1       2019-11-03 02:57:49         1315   \n",
       "2       2019-11-03 02:57:49          105   \n",
       "3       2019-11-03 02:57:48          660   \n",
       "4       2019-11-03 02:57:46            0   \n",
       "...                     ...          ...   \n",
       "248646  2019-11-19 17:19:37           72   \n",
       "248647  2019-11-19 17:19:36          796   \n",
       "248648  2019-11-19 17:19:36         1162   \n",
       "248649  2019-11-19 17:19:36            9   \n",
       "248650  2019-11-19 17:19:35           10   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "0                                                                                                                                                           Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                                      #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2                    This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3                         A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                                                #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "...                                                                                                                                                                                                                                                                                                                               ...   \n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                           hashtags  \\\n",
       "0                                                                        [hongkong]   \n",
       "1                                                              [china, policestate]   \n",
       "2                                                               [hkpolicebrutality]   \n",
       "3                                                                  [hongkongpolice]   \n",
       "4       [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]   \n",
       "...                                                                             ...   \n",
       "248646                                                                           []   \n",
       "248647                                                                           []   \n",
       "248648                                                                           []   \n",
       "248649                                 [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                                                           []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "0                               []   \n",
       "1                        [@hkgetv]   \n",
       "2                               []   \n",
       "3                               []   \n",
       "4                               []   \n",
       "...                            ...   \n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \\\n",
       "0                                                                                                                         disgust police mark protestors pen write number hand like nazi germany   \n",
       "1                                                                                                                                    teargas canister stick road citizen try pull failed\\ncredit   \n",
       "2                                          tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence   \n",
       "3                                     fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away   \n",
       "4                                                                                                           threw hand blast citizen’s back make large area burn tear gas read hand grenade kill   \n",
       "...                                                                                                                                                                                          ...   \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy   \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom   \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read   \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n   \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please   \n",
       "\n",
       "          neg    neu    pos  compound  \n",
       "0       0.218  0.391  0.391    0.2732  \n",
       "1       0.256  0.543  0.202   -0.1779  \n",
       "2       0.339  0.590  0.070   -0.8074  \n",
       "3       0.364  0.498  0.137   -0.7430  \n",
       "4       0.203  0.519  0.277    0.1779  \n",
       "...       ...    ...    ...       ...  \n",
       "248646  0.000  0.882  0.118    0.5574  \n",
       "248647  0.337  0.358  0.305   -0.1265  \n",
       "248648  0.107  0.785  0.107    0.0000  \n",
       "248649  0.000  0.794  0.206    0.0772  \n",
       "248650  0.378  0.458  0.163   -0.8176  \n",
       "\n",
       "[248651 rows x 17 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Additional Features: no. of characters and no. of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.0772</td>\n",
       "      <td>40</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "      <td>0.378</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.8176</td>\n",
       "      <td>141</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            username  \\\n",
       "248649  terryyipomni   \n",
       "248650      lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers totaltweets        usercreatedts  \\\n",
       "248649  hong kong     4653.0     2490.0        4627  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0        5042  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts retweetcount  \\\n",
       "248649  2019-11-19 17:19:36            9   \n",
       "248650  2019-11-19 17:19:35           10   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                            hashtags  \\\n",
       "248649  [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                            []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                         cleaned_text  \\\n",
       "248649                                                                                                    huh excuse hong kong government lie \\n.\\n\\n   \n",
       "248650  can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  \n",
       "248649  0.000  0.794  0.206    0.0772        40         7  \n",
       "248650  0.378  0.458  0.163   -0.8176       141        22  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# New column: number of characters in 'review'\n",
    "data['numchars'] = data['cleaned_text'].apply(lambda x: len(x))\n",
    "\n",
    "# New column: number of words in 'review'\n",
    "data['numwords'] = data['cleaned_text'].apply(lambda x: len(x.split(\" \")))\n",
    "\n",
    "# Check the new columns:\n",
    "data.tail(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word Embeddings - Training Word2Vec using Gensim\n",
    "\n",
    "Word embeddings involve the mapping of words in the corpus to numerical vectors, where similar words sharing similar contexts will have similar vectors as well. It will involve a shallow two-layer neural network that will train a matrix/tensor called the embedding matrix. By taking the matrix product of the embedding matrix and one-hot vector representation of each word in the corpus, we will arrive at the embedding vector.\n",
    "\n",
    "We will be using Gensim - an open-source Python library - to generate doc2vec. <br/>\n",
    "\n",
    "To confirm: apparently, doc2vec should be used over word2vec to obtain the vector representation of a 'document', in this case, our 'reviews' from the travellers. Word2vec will only give us the vector representation of a 'word'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Import the Gensim package\n",
    "# from gensim.test.utils import common_texts\n",
    "# from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
    "\n",
    "# documents = [TaggedDocument(doc, [i]) for i, doc in enumerate(data[\"cleaned_text\"].apply(lambda x: x.split(\" \")))]\n",
    "\n",
    "# # Train a Doc2Vec model with our text data\n",
    "# model = Doc2Vec(documents, vector_size = 10, window = 2, min_count = 1, workers = 4)\n",
    "\n",
    "# # Transform each document into a vector data\n",
    "# doc2vec_df = data[\"cleaned_text\"].apply(lambda x: model.infer_vector(x.split(\" \"))).apply(pd.Series)\n",
    "# doc2vec_df.columns = [\"doc2vec_vector_\" + str(x) for x in doc2vec_df.columns]\n",
    "# data = pd.concat([data, doc2vec_df], axis = 1)\n",
    "\n",
    "# # Check out the newly added columns:\n",
    "# data.tail(2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TD-IDF Columns\n",
    "Next, we will compute the TD-IDF of the reviews using the sklearn library. TD-IDF stands for Term Frequency-Inverse Document Frequency, which is used to reflect how important a word is to a document in a collection or corpus. The tf–idf value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word, which helps to adjust for the fact that some words appear more frequently in general.\n",
    "\n",
    "1. Term Frequency - the number of times a term occurs in a document.\n",
    "2. Inverse Document Frequency - an inverse document frequency factor that diminishes the weight of terms that occur very frequently in the document set and increases the weight of terms that occur rarely.\n",
    "\n",
    "Since NLTK does not support TF-IDF, we will use the tfidfvectorizer function from the Python sklearn library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-learn in /Users/shiruizhou/opt/anaconda3/envs/anly645/lib/python3.10/site-packages (1.3.0)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /Users/shiruizhou/opt/anaconda3/envs/anly645/lib/python3.10/site-packages (from scikit-learn) (1.24.3)\n",
      "Requirement already satisfied: scipy>=1.5.0 in /Users/shiruizhou/opt/anaconda3/envs/anly645/lib/python3.10/site-packages (from scikit-learn) (1.10.1)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /Users/shiruizhou/opt/anaconda3/envs/anly645/lib/python3.10/site-packages (from scikit-learn) (1.2.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/shiruizhou/opt/anaconda3/envs/anly645/lib/python3.10/site-packages (from scikit-learn) (3.1.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install --upgrade scikit-learn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "# import pandas as pd\n",
    "\n",
    "# # Assuming 'data' is your DataFrame containing the 'cleaned_text' column.\n",
    "\n",
    "# tfidf = TfidfVectorizer(\n",
    "#     max_features=100,\n",
    "#     min_df=10,\n",
    "#     stop_words='english'\n",
    "# )\n",
    "\n",
    "# tfidf_result = tfidf.fit_transform(data['cleaned_text']).toarray()\n",
    "\n",
    "# # Extract the feature names using the get_feature_names_out() method.\n",
    "# feature_names = tfidf.get_feature_names_out()\n",
    "\n",
    "# # Create a DataFrame from tfidf_result using the feature names\n",
    "# tfidf_df = pd.DataFrame(tfidf_result, columns=feature_names)\n",
    "\n",
    "# # Rename the column names and index\n",
    "# tfidf_df.columns = [\"word_\" + str(x) for x in tfidf_df.columns]\n",
    "# tfidf_df.index = data.index\n",
    "\n",
    "# # Concatenate the two DataFrames - 'data' and 'tfidf_df'\n",
    "# data = pd.concat([data, tfidf_df], axis=1)\n",
    "\n",
    "# # Check out the new 'data' DataFrame\n",
    "# print(data.tail(2))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected Vocabulary \n",
    "\n",
    "Looking at the wordcloud, the only obvious word that has a strong negative connotation would be 'disgust', appearing at the bottom. Most of the remaining words thrown out align with what one has been seeing regularly on news and social media. From our wordcloud, we see that these words are - police, tear gas cannisters, protest, protestors etc. \n",
    "\n",
    "Outdated as more tweets are taken into consideration: Interestingly, 'good' appears in the wordcloud. But as of now, it still remains to be seen as to what or who is mentioned or associated numerously with 'good'. Let's see if we can find out more.\n",
    "\n",
    "### 2 No. of Positive Sentiments vs No. of Negative Sentiments\n",
    "Next, we look at what is the overall distribution of positive and negative tweets. Based on the SentimentIntensityAnalyzer from the NLTK Vader-Lexicon library, this analyzer examines the sentiment of a sentence, on how positive, neutral or negative it is. We can interpret the sentiment in the following manner. If a sentiment is positive, it could mean that it is pro-government and/or police. Whereas, a negative sentiment could mean that it is anti- government and/or police, and supportive towards the protesters.\n",
    "\n",
    "The analyzer returns 4 scores for each sentence namely, 'positive', 'negative', 'neutral' and 'compound'. The score 'compound' returns the overall sentiment of a sentence with range of [-1, 1]. For our current purpose, we want to classify each tweet into 5 classes and assign a range of values for each of them:\n",
    "1. Very positive '5' - [0.55, 1.00]\n",
    "2. Positive '4' - [0.10, 0.55)\n",
    "3. Neutral '3' - (-0.10, 0.10)\n",
    "4. Negative '2' - (-0.55, -0.10]\n",
    "5. Very negative '1' - [-1.00, -0.55]\n",
    "\n",
    "Note: the range of values for a neutral sentiment is more stringent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "248646    5\n",
       "248647    2\n",
       "248648    3\n",
       "248649    3\n",
       "248650    1\n",
       "Name: sentiment_class, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Focus on 'compound' scores\n",
    "# Create a new column called 'sentiment_class'\n",
    "sentimentclass_list = []\n",
    "\n",
    "for i in range(0, len(data)):\n",
    "    \n",
    "    # current 'compound' score:\n",
    "    curr_compound = data.iloc[i,:]['compound']\n",
    "    \n",
    "    if (curr_compound <= 1.0 and curr_compound >= 0.55):\n",
    "        sentimentclass_list.append(5)\n",
    "    elif (curr_compound < 0.55 and curr_compound >= 0.10):\n",
    "        sentimentclass_list.append(4)\n",
    "    elif (curr_compound < 0.10 and curr_compound > -0.10):\n",
    "        sentimentclass_list.append(3)\n",
    "    elif (curr_compound <= -0.10 and curr_compound > -0.55):\n",
    "        sentimentclass_list.append(2)\n",
    "    elif (curr_compound <= -0.55 and curr_compound >= -1.00):\n",
    "        sentimentclass_list.append(1)\n",
    "\n",
    "# Add the new column 'sentiment_class' to the dataframe\n",
    "data['sentiment_class'] = sentimentclass_list\n",
    "\n",
    "# Check out the new column\n",
    "data.tail()['sentiment_class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>compound</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.2732</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.1779</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.8074</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.7430</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.1779</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   compound  sentiment_class\n",
       "0    0.2732                4\n",
       "1   -0.1779                2\n",
       "2   -0.8074                1\n",
       "3   -0.7430                1\n",
       "4    0.1779                4"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify if the classification assignment is correct:\n",
    "data.iloc[0:5, :][['compound', 'sentiment_class']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         4\n",
       "1         2\n",
       "2         1\n",
       "3         1\n",
       "4         4\n",
       "         ..\n",
       "248646    5\n",
       "248647    2\n",
       "248648    3\n",
       "248649    3\n",
       "248650    1\n",
       "Name: sentiment_class, Length: 248651, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['sentiment_class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display full text:\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tabulate in /Users/shiruizhou/opt/anaconda3/envs/anly645/lib/python3.10/site-packages (0.9.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install tabulate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tabulate import tabulate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 random negative original tweets and their sentiment classes:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>102609</th>\n",
       "      <td>Hong Kong lawmaker Eddie Chu arrested over scuffle at legislature in May, with police set to nab 6 other democrats https://t.co/yc2RpblajY #HongKong #China #antiELABhk #antiELAB #HongKongProtests @ChuHoiDick @holmeschan</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14837</th>\n",
       "      <td>Police officers have been filmed shoving a fireman around in Central after he complained they had shot tear gas straight at his truck. The officers also pepper sprayed a journalist who dared to report on the bust up. #HongKong\\n\\nhttps://t.co/rubbu7eQe2 https://t.co/Yt1FKU4BRm</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59944</th>\n",
       "      <td>This photo was taken when the 22 y/o boy was brain-bleeding and waiting for ambulance.\\n\\nFour police cars blocked the way and didn’t allow the ambulance to pick up the casualty.\\n\\nNo CCTV are not released.\\n\\nNo one knows why he fell frok height.\\n\\n#SOSHK #HongKong #HongKongPolice https://t.co/4TpkHjdqfD</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146594</th>\n",
       "      <td>[1631 Sha Tin]\\n\\n#HongKongPolice was trying to obstruct the arrestee from yelling his name &amp;amp; identity no. to the bystanders trying to help him.\\n\\nLook into the boy's eyes only panic &amp;amp; despration could be seen.\\n\\n[h/t SocRec Channel]\\n\\n#HongKongPoliceState #HongKong\\n#HongKongProtests https://t.co/rNa6GGtOEA</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79605</th>\n",
       "      <td>When the #YuenLongAttack took place, cops who patrolled at the station turn their back on the civilians, local police station closed, so when victims calling 999 for help that’s an abuse use of emergency call?! \\n#HongKong\\n#PoliceBrutality https://t.co/5Aay74ugr0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3651</th>\n",
       "      <td>A #Porsche driver was playing #GloryToHongKong the song in car in HK.  Riot police aren't happy about it, brutally pulled driver out of his car &amp;amp; arrested him for unknown cause.  #HKPolice also did car search without consent/presence of owner. \\n\\n#StandWithHK #FightForFreedom https://t.co/LdCr604BCq</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124954</th>\n",
       "      <td>#HongKong people are gathering at Tamar Park today for the memorial prayer meeting, to mourning the death of 22-yr-old college student Chow Tsz-Lok, and remonstrate the #PoliceBrutality. \\n\\n#HumanitarianCrisis \\n#HumanRights \\n\\nhttps://t.co/vWiUmXsWSe</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60817</th>\n",
       "      <td>The men was actually wanted to rescue the students who fell from 3/F. So he tried to stop #HKPolice, in order to let the ambulance in. Unfortunately, he was arrested and got hurt at the end.\\n\\n🥺🙏\\n\\nCredit: appledaily\\n\\n#HongKongProtests #StandWithHongKong https://t.co/JfqEvyPY0g</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128463</th>\n",
       "      <td>Did #HongKong Apple Daily report the name(s) of the arresting officer(s)? https://t.co/CSYVHoaMOe</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2584</th>\n",
       "      <td>Council candidate conducted election rallies today yet #hkpolice claimed they were participating an unlawful assembly, and violently arrested the candidate. \\n#HKprotests #HKPoliceTerrorism #HKRioters #HKPoliceState https://t.co/0BnOM7r51e</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                    text  \\\n",
       "102609                                                                                                       Hong Kong lawmaker Eddie Chu arrested over scuffle at legislature in May, with police set to nab 6 other democrats https://t.co/yc2RpblajY #HongKong #China #antiELABhk #antiELAB #HongKongProtests @ChuHoiDick @holmeschan   \n",
       "14837                                              Police officers have been filmed shoving a fireman around in Central after he complained they had shot tear gas straight at his truck. The officers also pepper sprayed a journalist who dared to report on the bust up. #HongKong\\n\\nhttps://t.co/rubbu7eQe2 https://t.co/Yt1FKU4BRm   \n",
       "59944               This photo was taken when the 22 y/o boy was brain-bleeding and waiting for ambulance.\\n\\nFour police cars blocked the way and didn’t allow the ambulance to pick up the casualty.\\n\\nNo CCTV are not released.\\n\\nNo one knows why he fell frok height.\\n\\n#SOSHK #HongKong #HongKongPolice https://t.co/4TpkHjdqfD   \n",
       "146594  [1631 Sha Tin]\\n\\n#HongKongPolice was trying to obstruct the arrestee from yelling his name &amp; identity no. to the bystanders trying to help him.\\n\\nLook into the boy's eyes only panic &amp; despration could be seen.\\n\\n[h/t SocRec Channel]\\n\\n#HongKongPoliceState #HongKong\\n#HongKongProtests https://t.co/rNa6GGtOEA   \n",
       "79605                                                           When the #YuenLongAttack took place, cops who patrolled at the station turn their back on the civilians, local police station closed, so when victims calling 999 for help that’s an abuse use of emergency call?! \\n#HongKong\\n#PoliceBrutality https://t.co/5Aay74ugr0   \n",
       "3651                   A #Porsche driver was playing #GloryToHongKong the song in car in HK.  Riot police aren't happy about it, brutally pulled driver out of his car &amp; arrested him for unknown cause.  #HKPolice also did car search without consent/presence of owner. \\n\\n#StandWithHK #FightForFreedom https://t.co/LdCr604BCq   \n",
       "124954                                                                     #HongKong people are gathering at Tamar Park today for the memorial prayer meeting, to mourning the death of 22-yr-old college student Chow Tsz-Lok, and remonstrate the #PoliceBrutality. \\n\\n#HumanitarianCrisis \\n#HumanRights \\n\\nhttps://t.co/vWiUmXsWSe   \n",
       "60817                                         The men was actually wanted to rescue the students who fell from 3/F. So he tried to stop #HKPolice, in order to let the ambulance in. Unfortunately, he was arrested and got hurt at the end.\\n\\n🥺🙏\\n\\nCredit: appledaily\\n\\n#HongKongProtests #StandWithHongKong https://t.co/JfqEvyPY0g   \n",
       "128463                                                                                                                                                                                                                                 Did #HongKong Apple Daily report the name(s) of the arresting officer(s)? https://t.co/CSYVHoaMOe   \n",
       "2584                                                                                     Council candidate conducted election rallies today yet #hkpolice claimed they were participating an unlawful assembly, and violently arrested the candidate. \\n#HKprotests #HKPoliceTerrorism #HKRioters #HKPoliceState https://t.co/0BnOM7r51e   \n",
       "\n",
       "        sentiment_class  \n",
       "102609                2  \n",
       "14837                 2  \n",
       "59944                 1  \n",
       "146594                2  \n",
       "79605                 1  \n",
       "3651                  1  \n",
       "124954                1  \n",
       "60817                 2  \n",
       "128463                2  \n",
       "2584                  1  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at some examples of negative, neutral and positive tweets\n",
    "\n",
    "# Filter 10 negative original tweets:\n",
    "print(\"10 random negative original tweets and their sentiment classes:\")\n",
    "data[(data['sentiment_class'] == 1) | (data['sentiment_class'] == 2)].sample(n=10)[['text', 'sentiment_class']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 random neutral original tweets and their sentiment classes:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>37853</th>\n",
       "      <td>This is a photo deleted by facebook for several times. \\n#HKPoliceTerrorism #HongKong #SOSHK \\nsource : internet https://t.co/SiSGTSyV3c</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203814</th>\n",
       "      <td>The tear gas made in China  which has higher burning point (450°c) is even more toxic than that of the USA. The highest burning point is said to be 3000°c! It contain more than80%of CS, when the CS break down, more dioxins and cyanide gas may be released.#StandWithHongKong #SOSHK https://t.co/RxiZGYnvET</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14070</th>\n",
       "      <td>This rat was not moving. People thought it might have been teargassed too. #HKprotests #HKPolice #HongKong #HongKongProtests #HongKongProtesters #HongKongProtest https://t.co/HLvSg7U7Uc</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96958</th>\n",
       "      <td>Lunch break in Central.  These are not the typical young protesters.  You're looking at people in dress shirts and business attire speaking up against #HKPolice during their lunch hour in memory of Chow\\n\\n#StandWithHongKong #HongKongProtests https://t.co/wRqW4THqMS</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181285</th>\n",
       "      <td>Today we’re wearing black to rehearsals to #StandWithHongKong #HKWEARSBLACK \\n#HongKongProtests https://t.co/trZ4NNe828</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77632</th>\n",
       "      <td>A mainland student held a knife towards the graduates of CUHK and sang the Chinese National Anthem.\\n\\nto b honest i think his singing is more scare than the little kinfe…🀄️🔪🀄️\\n\\nfrom @cuhkcampusradio \\n #HongKongProtests #CUHK\\nhttps://t.co/MM1siFPIlf</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151978</th>\n",
       "      <td>This is Central in #HongKong\\n\\nCan you believe #HongKongPolice shot tear gas in random towards most office ladies, man in suit?\\n\\nWill #HongKong still exist tomorrow? https://t.co/KoN0JGnFED</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235557</th>\n",
       "      <td>When dictatorship becomes a fact, revolution becomes a duty.\\n\\n#StandWithHongKong\\n#SOSPolyU \\n#HongKongProtests https://t.co/A1Qh9Vr6hy</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114622</th>\n",
       "      <td>#hongkongpolice basically aren’t professional anymore, just doing everything based on their emotion. no police will be held accountable though.\\n-\\n@SolomonYue @arnohb112 #freedomhk #freehk #hongkongprotest #policebrutality #standwithhongkong # #hongkongprotests #hongkong https://t.co/4GLFEMzJS7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172772</th>\n",
       "      <td>Photo description from Apple Daily\\n#HongKong #CUHK https://t.co/fNakKugNF4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                    text  \\\n",
       "37853                                                                                                                                                                           This is a photo deleted by facebook for several times. \\n#HKPoliceTerrorism #HongKong #SOSHK \\nsource : internet https://t.co/SiSGTSyV3c   \n",
       "203814  The tear gas made in China  which has higher burning point (450°c) is even more toxic than that of the USA. The highest burning point is said to be 3000°c! It contain more than80%of CS, when the CS break down, more dioxins and cyanide gas may be released.#StandWithHongKong #SOSHK https://t.co/RxiZGYnvET   \n",
       "14070                                                                                                                          This rat was not moving. People thought it might have been teargassed too. #HKprotests #HKPolice #HongKong #HongKongProtests #HongKongProtesters #HongKongProtest https://t.co/HLvSg7U7Uc   \n",
       "96958                                         Lunch break in Central.  These are not the typical young protesters.  You're looking at people in dress shirts and business attire speaking up against #HKPolice during their lunch hour in memory of Chow\\n\\n#StandWithHongKong #HongKongProtests https://t.co/wRqW4THqMS   \n",
       "181285                                                                                                                                                                                           Today we’re wearing black to rehearsals to #StandWithHongKong #HKWEARSBLACK \\n#HongKongProtests https://t.co/trZ4NNe828   \n",
       "77632                                                      A mainland student held a knife towards the graduates of CUHK and sang the Chinese National Anthem.\\n\\nto b honest i think his singing is more scare than the little kinfe…🀄️🔪🀄️\\n\\nfrom @cuhkcampusradio \\n #HongKongProtests #CUHK\\nhttps://t.co/MM1siFPIlf   \n",
       "151978                                                                                                                  This is Central in #HongKong\\n\\nCan you believe #HongKongPolice shot tear gas in random towards most office ladies, man in suit?\\n\\nWill #HongKong still exist tomorrow? https://t.co/KoN0JGnFED   \n",
       "235557                                                                                                                                                                         When dictatorship becomes a fact, revolution becomes a duty.\\n\\n#StandWithHongKong\\n#SOSPolyU \\n#HongKongProtests https://t.co/A1Qh9Vr6hy   \n",
       "114622          #hongkongpolice basically aren’t professional anymore, just doing everything based on their emotion. no police will be held accountable though.\\n-\\n@SolomonYue @arnohb112 #freedomhk #freehk #hongkongprotest #policebrutality #standwithhongkong # #hongkongprotests #hongkong https://t.co/4GLFEMzJS7   \n",
       "172772                                                                                                                                                                                                                                       Photo description from Apple Daily\\n#HongKong #CUHK https://t.co/fNakKugNF4   \n",
       "\n",
       "        sentiment_class  \n",
       "37853                 3  \n",
       "203814                3  \n",
       "14070                 3  \n",
       "96958                 3  \n",
       "181285                3  \n",
       "77632                 3  \n",
       "151978                3  \n",
       "235557                3  \n",
       "114622                3  \n",
       "172772                3  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter 10 neutral original tweets:\n",
    "print(\"10 random neutral original tweets and their sentiment classes:\")\n",
    "data[(data['sentiment_class'] == 3)].sample(n=10)[['text', 'sentiment_class']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 random positive original tweets and their sentiment classes:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>205629</th>\n",
       "      <td>US Congress urged to suspend Hong Kong’s special trade status if Chinese troops used in city https://t.co/YrUHNSi0S9 @krislc #HongKong #China</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136970</th>\n",
       "      <td>Thanks again for this #Japanese who concern about the movement in #HongKong, we appreciated every single effort from all over the world to stand in solidarity, #StandWithHongKong \\n\\nhttps://t.co/riJLhLpYn1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49736</th>\n",
       "      <td>Speaking to parliamentarians this morning about an UK #HongKong Human Rights and Democracy Act. Lots of enthusiasm. UK based lawyers who love HK and want to help should DM me. #StandWithHongKong</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238532</th>\n",
       "      <td>This is the beginning of the end for #HongKong. Beijing is usurping the court system of the special administrative region. The rule of law is all Hong Kong had to differentiate itself from the mainland. Now we will have rule by fiat as in China. https://t.co/sK7Ske5wv3 via @FT</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74720</th>\n",
       "      <td>@limlouisa @unimelb The CCP is surely a threat to universal values. Please #StandWithHongKong to defend freedom and human rights https://t.co/ncNlyAUp7d</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211655</th>\n",
       "      <td>@revmahoney @Andychanhotin @FreedomHKG @Fight4HongKong @Stand_with_HK @hk_watch @HKWORLDCITY @mschlapp We do need some spiritual supports...Thank you very much for your kindness. #StandWithHongKong</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21534</th>\n",
       "      <td>Police have fired multiple rounds of tear gas into Victoria Park, where hundreds, if not thousand of people were gathering peacefully to support the ongoing pro-democracy movement. #HongKong https://t.co/YgAxoQnRSy</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192309</th>\n",
       "      <td>@marcorubio We need your help🙏🏻🙏🏻\\n#StandWithHongKong https://t.co/YQWnjcuOKT</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132754</th>\n",
       "      <td>Suzuko Hirano who has been organizing weekly demonstrations in Tokyo since June to show solidarity with the protesters in #HongKong.\\nI’m glad to see people from all over the world stand in solidarity with Hong Kong.\\nThanks #Japan\\n\\n#StandWithHongKong \\n\\nhttps://t.co/lOyjE06sjc</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79011</th>\n",
       "      <td>In the past 7 days, the Hong Kong Human Rights and Democracy Act got 2 more cosponsors. Thank you Senators @SenMcSallyAZ and @CoryBooker! #StandWithHongKong https://t.co/2zA718sNTf</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                             text  \\\n",
       "205629                                                                                                                                              US Congress urged to suspend Hong Kong’s special trade status if Chinese troops used in city https://t.co/YrUHNSi0S9 @krislc #HongKong #China   \n",
       "136970                                                                             Thanks again for this #Japanese who concern about the movement in #HongKong, we appreciated every single effort from all over the world to stand in solidarity, #StandWithHongKong \\n\\nhttps://t.co/riJLhLpYn1   \n",
       "49736                                                                                          Speaking to parliamentarians this morning about an UK #HongKong Human Rights and Democracy Act. Lots of enthusiasm. UK based lawyers who love HK and want to help should DM me. #StandWithHongKong   \n",
       "238532      This is the beginning of the end for #HongKong. Beijing is usurping the court system of the special administrative region. The rule of law is all Hong Kong had to differentiate itself from the mainland. Now we will have rule by fiat as in China. https://t.co/sK7Ske5wv3 via @FT   \n",
       "74720                                                                                                                                    @limlouisa @unimelb The CCP is surely a threat to universal values. Please #StandWithHongKong to defend freedom and human rights https://t.co/ncNlyAUp7d   \n",
       "211655                                                                                      @revmahoney @Andychanhotin @FreedomHKG @Fight4HongKong @Stand_with_HK @hk_watch @HKWORLDCITY @mschlapp We do need some spiritual supports...Thank you very much for your kindness. #StandWithHongKong   \n",
       "21534                                                                      Police have fired multiple rounds of tear gas into Victoria Park, where hundreds, if not thousand of people were gathering peacefully to support the ongoing pro-democracy movement. #HongKong https://t.co/YgAxoQnRSy   \n",
       "192309                                                                                                                                                                                                              @marcorubio We need your help🙏🏻🙏🏻\\n#StandWithHongKong https://t.co/YQWnjcuOKT   \n",
       "132754  Suzuko Hirano who has been organizing weekly demonstrations in Tokyo since June to show solidarity with the protesters in #HongKong.\\nI’m glad to see people from all over the world stand in solidarity with Hong Kong.\\nThanks #Japan\\n\\n#StandWithHongKong \\n\\nhttps://t.co/lOyjE06sjc   \n",
       "79011                                                                                                        In the past 7 days, the Hong Kong Human Rights and Democracy Act got 2 more cosponsors. Thank you Senators @SenMcSallyAZ and @CoryBooker! #StandWithHongKong https://t.co/2zA718sNTf   \n",
       "\n",
       "        sentiment_class  \n",
       "205629                4  \n",
       "136970                5  \n",
       "49736                 5  \n",
       "238532                4  \n",
       "74720                 5  \n",
       "211655                4  \n",
       "21534                 5  \n",
       "192309                4  \n",
       "132754                5  \n",
       "79011                 4  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter 20 positive original tweets:\n",
    "print(\"10 random positive original tweets and their sentiment classes:\")\n",
    "data[(data['sentiment_class'] == 4) | (data['sentiment_class'] == 5)].sample(n=10)[['text', 'sentiment_class']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['username', 'acctdesc', 'location', 'following', 'followers',\n",
      "       'totaltweets', 'usercreatedts', 'tweetcreatedts', 'retweetcount',\n",
      "       'text', 'hashtags', 'mentioned_users', 'cleaned_text', 'neg', 'neu',\n",
      "       'pos', 'compound', 'numchars', 'numwords', 'sentiment_class'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(data.columns)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>n.a</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>70</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[china, policestate]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.543</td>\n",
       "      <td>0.202</td>\n",
       "      <td>-0.1779</td>\n",
       "      <td>58</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#fightforfreedom #standwithhongkong</td>\n",
       "      <td>n.a</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[hkpolicebrutality]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "      <td>0.339</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.070</td>\n",
       "      <td>-0.8074</td>\n",
       "      <td>148</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.498</td>\n",
       "      <td>0.137</td>\n",
       "      <td>-0.7430</td>\n",
       "      <td>152</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.519</td>\n",
       "      <td>0.277</td>\n",
       "      <td>0.1779</td>\n",
       "      <td>84</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>five demands, not one less 🇭🇰#followbackhongkong #科勞手足</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.5574</td>\n",
       "      <td>177</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #standwithhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.305</td>\n",
       "      <td>-0.1265</td>\n",
       "      <td>184</td>\n",
       "      <td>26</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>n.a</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.785</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>184</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.0772</td>\n",
       "      <td>40</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "      <td>0.378</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.8176</td>\n",
       "      <td>141</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>248651 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "0          five5demands   \n",
       "1        Dejavu53328974   \n",
       "2              tsksimon   \n",
       "3       JimmyWo67187904   \n",
       "4              ARCHI418   \n",
       "...                 ...   \n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "0                                                                                                       hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1                                                             #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                                                                                       #fightforfreedom #standwithhongkong   \n",
       "3                                                                                                                                                       n.a   \n",
       "4                                                                                                                                                       n.a   \n",
       "...                                                                                                                                                     ...   \n",
       "248646                                                                                               five demands, not one less 🇭🇰#followbackhongkong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #standwithhongkong   \n",
       "248648                                                                                        hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers totaltweets        usercreatedts  \\\n",
       "0             n.a      437.0      260.0         466  2019-10-05 13:40:37   \n",
       "1       hong kong      581.0      360.0        7835  2019-08-27 15:47:34   \n",
       "2             n.a      825.0      456.0        6728  2014-10-05 15:57:07   \n",
       "3             n.a       22.0        1.0         656  2019-10-08 12:58:51   \n",
       "4             n.a       19.0       11.0         121  2014-06-19 16:19:36   \n",
       "...           ...        ...        ...         ...                  ...   \n",
       "248646  hong kong      505.0      326.0        1301  2019-09-25 17:05:21   \n",
       "248647  hong kong       15.0      155.0         116  2019-09-12 06:32:09   \n",
       "248648        n.a      248.0      170.0         362  2019-09-28 14:29:13   \n",
       "248649  hong kong     4653.0     2490.0        4627  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0        5042  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts retweetcount  \\\n",
       "0       2019-11-03 02:57:49          292   \n",
       "1       2019-11-03 02:57:49         1315   \n",
       "2       2019-11-03 02:57:49          105   \n",
       "3       2019-11-03 02:57:48          660   \n",
       "4       2019-11-03 02:57:46            0   \n",
       "...                     ...          ...   \n",
       "248646  2019-11-19 17:19:37           72   \n",
       "248647  2019-11-19 17:19:36          796   \n",
       "248648  2019-11-19 17:19:36         1162   \n",
       "248649  2019-11-19 17:19:36            9   \n",
       "248650  2019-11-19 17:19:35           10   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "0                                                                                                                                                           Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                                      #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2                    This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3                         A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                                                #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "...                                                                                                                                                                                                                                                                                                                               ...   \n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                           hashtags  \\\n",
       "0                                                                        [hongkong]   \n",
       "1                                                              [china, policestate]   \n",
       "2                                                               [hkpolicebrutality]   \n",
       "3                                                                  [hongkongpolice]   \n",
       "4       [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]   \n",
       "...                                                                             ...   \n",
       "248646                                                                           []   \n",
       "248647                                                                           []   \n",
       "248648                                                                           []   \n",
       "248649                                 [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                                                           []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "0                               []   \n",
       "1                        [@hkgetv]   \n",
       "2                               []   \n",
       "3                               []   \n",
       "4                               []   \n",
       "...                            ...   \n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \\\n",
       "0                                                                                                                         disgust police mark protestors pen write number hand like nazi germany   \n",
       "1                                                                                                                                    teargas canister stick road citizen try pull failed\\ncredit   \n",
       "2                                          tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence   \n",
       "3                                     fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away   \n",
       "4                                                                                                           threw hand blast citizen’s back make large area burn tear gas read hand grenade kill   \n",
       "...                                                                                                                                                                                          ...   \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy   \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom   \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read   \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n   \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "0       0.218  0.391  0.391    0.2732        70        11                4  \n",
       "1       0.256  0.543  0.202   -0.1779        58         8                2  \n",
       "2       0.339  0.590  0.070   -0.8074       148        20                1  \n",
       "3       0.364  0.498  0.137   -0.7430       152        23                1  \n",
       "4       0.203  0.519  0.277    0.1779        84        15                4  \n",
       "...       ...    ...    ...       ...       ...       ...              ...  \n",
       "248646  0.000  0.882  0.118    0.5574       177        27                5  \n",
       "248647  0.337  0.358  0.305   -0.1265       184        26                2  \n",
       "248648  0.107  0.785  0.107    0.0000       184        21                3  \n",
       "248649  0.000  0.794  0.206    0.0772        40         7                3  \n",
       "248650  0.378  0.458  0.163   -0.8176       141        22                1  \n",
       "\n",
       "[248651 rows x 20 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['retweetcount'] = pd.to_numeric(data['retweetcount'], errors='coerce')\n",
    "data['totaltweets'] = pd.to_numeric(data['totaltweets'], errors='coerce')\n",
    "data['followers'] = pd.to_numeric(data['followers'], errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>n.a</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466.0</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292.0</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>70</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835.0</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315.0</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[china, policestate]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.543</td>\n",
       "      <td>0.202</td>\n",
       "      <td>-0.1779</td>\n",
       "      <td>58</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#fightforfreedom #standwithhongkong</td>\n",
       "      <td>n.a</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728.0</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105.0</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[hkpolicebrutality]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "      <td>0.339</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.070</td>\n",
       "      <td>-0.8074</td>\n",
       "      <td>148</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656.0</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660.0</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.498</td>\n",
       "      <td>0.137</td>\n",
       "      <td>-0.7430</td>\n",
       "      <td>152</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.519</td>\n",
       "      <td>0.277</td>\n",
       "      <td>0.1779</td>\n",
       "      <td>84</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>five demands, not one less 🇭🇰#followbackhongkong #科勞手足</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301.0</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72.0</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.5574</td>\n",
       "      <td>177</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #standwithhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116.0</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796.0</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.305</td>\n",
       "      <td>-0.1265</td>\n",
       "      <td>184</td>\n",
       "      <td>26</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>n.a</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362.0</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162.0</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.785</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>184</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627.0</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.0772</td>\n",
       "      <td>40</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042.0</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10.0</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "      <td>0.378</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.8176</td>\n",
       "      <td>141</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>248651 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "0          five5demands   \n",
       "1        Dejavu53328974   \n",
       "2              tsksimon   \n",
       "3       JimmyWo67187904   \n",
       "4              ARCHI418   \n",
       "...                 ...   \n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "0                                                                                                       hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1                                                             #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                                                                                       #fightforfreedom #standwithhongkong   \n",
       "3                                                                                                                                                       n.a   \n",
       "4                                                                                                                                                       n.a   \n",
       "...                                                                                                                                                     ...   \n",
       "248646                                                                                               five demands, not one less 🇭🇰#followbackhongkong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #standwithhongkong   \n",
       "248648                                                                                        hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers  totaltweets        usercreatedts  \\\n",
       "0             n.a      437.0      260.0        466.0  2019-10-05 13:40:37   \n",
       "1       hong kong      581.0      360.0       7835.0  2019-08-27 15:47:34   \n",
       "2             n.a      825.0      456.0       6728.0  2014-10-05 15:57:07   \n",
       "3             n.a       22.0        1.0        656.0  2019-10-08 12:58:51   \n",
       "4             n.a       19.0       11.0        121.0  2014-06-19 16:19:36   \n",
       "...           ...        ...        ...          ...                  ...   \n",
       "248646  hong kong      505.0      326.0       1301.0  2019-09-25 17:05:21   \n",
       "248647  hong kong       15.0      155.0        116.0  2019-09-12 06:32:09   \n",
       "248648        n.a      248.0      170.0        362.0  2019-09-28 14:29:13   \n",
       "248649  hong kong     4653.0     2490.0       4627.0  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0       5042.0  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts  retweetcount  \\\n",
       "0       2019-11-03 02:57:49         292.0   \n",
       "1       2019-11-03 02:57:49        1315.0   \n",
       "2       2019-11-03 02:57:49         105.0   \n",
       "3       2019-11-03 02:57:48         660.0   \n",
       "4       2019-11-03 02:57:46           0.0   \n",
       "...                     ...           ...   \n",
       "248646  2019-11-19 17:19:37          72.0   \n",
       "248647  2019-11-19 17:19:36         796.0   \n",
       "248648  2019-11-19 17:19:36        1162.0   \n",
       "248649  2019-11-19 17:19:36           9.0   \n",
       "248650  2019-11-19 17:19:35          10.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "0                                                                                                                                                           Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                                      #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2                    This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3                         A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                                                #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "...                                                                                                                                                                                                                                                                                                                               ...   \n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                           hashtags  \\\n",
       "0                                                                        [hongkong]   \n",
       "1                                                              [china, policestate]   \n",
       "2                                                               [hkpolicebrutality]   \n",
       "3                                                                  [hongkongpolice]   \n",
       "4       [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]   \n",
       "...                                                                             ...   \n",
       "248646                                                                           []   \n",
       "248647                                                                           []   \n",
       "248648                                                                           []   \n",
       "248649                                 [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                                                           []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "0                               []   \n",
       "1                        [@hkgetv]   \n",
       "2                               []   \n",
       "3                               []   \n",
       "4                               []   \n",
       "...                            ...   \n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \\\n",
       "0                                                                                                                         disgust police mark protestors pen write number hand like nazi germany   \n",
       "1                                                                                                                                    teargas canister stick road citizen try pull failed\\ncredit   \n",
       "2                                          tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence   \n",
       "3                                     fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away   \n",
       "4                                                                                                           threw hand blast citizen’s back make large area burn tear gas read hand grenade kill   \n",
       "...                                                                                                                                                                                          ...   \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy   \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom   \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read   \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n   \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "0       0.218  0.391  0.391    0.2732        70        11                4  \n",
       "1       0.256  0.543  0.202   -0.1779        58         8                2  \n",
       "2       0.339  0.590  0.070   -0.8074       148        20                1  \n",
       "3       0.364  0.498  0.137   -0.7430       152        23                1  \n",
       "4       0.203  0.519  0.277    0.1779        84        15                4  \n",
       "...       ...    ...    ...       ...       ...       ...              ...  \n",
       "248646  0.000  0.882  0.118    0.5574       177        27                5  \n",
       "248647  0.337  0.358  0.305   -0.1265       184        26                2  \n",
       "248648  0.107  0.785  0.107    0.0000       184        21                3  \n",
       "248649  0.000  0.794  0.206    0.0772        40         7                3  \n",
       "248650  0.378  0.458  0.163   -0.8176       141        22                1  \n",
       "\n",
       "[248651 rows x 20 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Writing the DataFrame to a CSV file\n",
    "file_path = 'data_processed.csv'\n",
    "data.to_csv(file_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>n.a</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466.0</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292.0</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>70</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835.0</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315.0</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[china, policestate]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.543</td>\n",
       "      <td>0.202</td>\n",
       "      <td>-0.1779</td>\n",
       "      <td>58</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#fightforfreedom #standwithhongkong</td>\n",
       "      <td>n.a</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728.0</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105.0</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[hkpolicebrutality]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "      <td>0.339</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.070</td>\n",
       "      <td>-0.8074</td>\n",
       "      <td>148</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656.0</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660.0</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.498</td>\n",
       "      <td>0.137</td>\n",
       "      <td>-0.7430</td>\n",
       "      <td>152</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.519</td>\n",
       "      <td>0.277</td>\n",
       "      <td>0.1779</td>\n",
       "      <td>84</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>five demands, not one less 🇭🇰#followbackhongkong #科勞手足</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301.0</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72.0</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.5574</td>\n",
       "      <td>177</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #standwithhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116.0</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796.0</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.305</td>\n",
       "      <td>-0.1265</td>\n",
       "      <td>184</td>\n",
       "      <td>26</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>n.a</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362.0</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162.0</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.785</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>184</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627.0</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.0772</td>\n",
       "      <td>40</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042.0</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10.0</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "      <td>0.378</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.8176</td>\n",
       "      <td>141</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>248651 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "0          five5demands   \n",
       "1        Dejavu53328974   \n",
       "2              tsksimon   \n",
       "3       JimmyWo67187904   \n",
       "4              ARCHI418   \n",
       "...                 ...   \n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "0                                                                                                       hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1                                                             #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                                                                                       #fightforfreedom #standwithhongkong   \n",
       "3                                                                                                                                                       n.a   \n",
       "4                                                                                                                                                       n.a   \n",
       "...                                                                                                                                                     ...   \n",
       "248646                                                                                               five demands, not one less 🇭🇰#followbackhongkong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #standwithhongkong   \n",
       "248648                                                                                        hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers  totaltweets        usercreatedts  \\\n",
       "0             n.a      437.0      260.0        466.0  2019-10-05 13:40:37   \n",
       "1       hong kong      581.0      360.0       7835.0  2019-08-27 15:47:34   \n",
       "2             n.a      825.0      456.0       6728.0  2014-10-05 15:57:07   \n",
       "3             n.a       22.0        1.0        656.0  2019-10-08 12:58:51   \n",
       "4             n.a       19.0       11.0        121.0  2014-06-19 16:19:36   \n",
       "...           ...        ...        ...          ...                  ...   \n",
       "248646  hong kong      505.0      326.0       1301.0  2019-09-25 17:05:21   \n",
       "248647  hong kong       15.0      155.0        116.0  2019-09-12 06:32:09   \n",
       "248648        n.a      248.0      170.0        362.0  2019-09-28 14:29:13   \n",
       "248649  hong kong     4653.0     2490.0       4627.0  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0       5042.0  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts  retweetcount  \\\n",
       "0       2019-11-03 02:57:49         292.0   \n",
       "1       2019-11-03 02:57:49        1315.0   \n",
       "2       2019-11-03 02:57:49         105.0   \n",
       "3       2019-11-03 02:57:48         660.0   \n",
       "4       2019-11-03 02:57:46           0.0   \n",
       "...                     ...           ...   \n",
       "248646  2019-11-19 17:19:37          72.0   \n",
       "248647  2019-11-19 17:19:36         796.0   \n",
       "248648  2019-11-19 17:19:36        1162.0   \n",
       "248649  2019-11-19 17:19:36           9.0   \n",
       "248650  2019-11-19 17:19:35          10.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "0                                                                                                                                                           Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                                      #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2                    This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3                         A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                                                #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "...                                                                                                                                                                                                                                                                                                                               ...   \n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                           hashtags  \\\n",
       "0                                                                        [hongkong]   \n",
       "1                                                              [china, policestate]   \n",
       "2                                                               [hkpolicebrutality]   \n",
       "3                                                                  [hongkongpolice]   \n",
       "4       [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]   \n",
       "...                                                                             ...   \n",
       "248646                                                                           []   \n",
       "248647                                                                           []   \n",
       "248648                                                                           []   \n",
       "248649                                 [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                                                           []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "0                               []   \n",
       "1                        [@hkgetv]   \n",
       "2                               []   \n",
       "3                               []   \n",
       "4                               []   \n",
       "...                            ...   \n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \\\n",
       "0                                                                                                                         disgust police mark protestors pen write number hand like nazi germany   \n",
       "1                                                                                                                                    teargas canister stick road citizen try pull failed\\ncredit   \n",
       "2                                          tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence   \n",
       "3                                     fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away   \n",
       "4                                                                                                           threw hand blast citizen’s back make large area burn tear gas read hand grenade kill   \n",
       "...                                                                                                                                                                                          ...   \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy   \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom   \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read   \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n   \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "0       0.218  0.391  0.391    0.2732        70        11                4  \n",
       "1       0.256  0.543  0.202   -0.1779        58         8                2  \n",
       "2       0.339  0.590  0.070   -0.8074       148        20                1  \n",
       "3       0.364  0.498  0.137   -0.7430       152        23                1  \n",
       "4       0.203  0.519  0.277    0.1779        84        15                4  \n",
       "...       ...    ...    ...       ...       ...       ...              ...  \n",
       "248646  0.000  0.882  0.118    0.5574       177        27                5  \n",
       "248647  0.337  0.358  0.305   -0.1265       184        26                2  \n",
       "248648  0.107  0.785  0.107    0.0000       184        21                3  \n",
       "248649  0.000  0.794  0.206    0.0772        40         7                3  \n",
       "248650  0.378  0.458  0.163   -0.8176       141        22                1  \n",
       "\n",
       "[248651 rows x 20 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10040</th>\n",
       "      <td>SolomanYue</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>15.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2019-10-29 05:41:44</td>\n",
       "      <td>2019-11-03 04:26:16</td>\n",
       "      <td>2817.0</td>\n",
       "      <td>Arrestees ALL GIRLS. \\nHard to believe they're not caught to serve the cops/PLAs in #HK especially when that's happening in #Xinjiang, and there're so many sexual abuse/ra_e allegations against #hkpolice.\\n\\n#HongKongProtests\\n#HKPoliceState\\n#HKPoliceTerrorism \\n@SolomonYue @HawleyMO https://t.co/2wX803P3lH</td>\n",
       "      <td>[hk]</td>\n",
       "      <td>[@SolomonYue, @HawleyMO]</td>\n",
       "      <td>arrestees girl \\nhard believe they're catch serve cops/plas especially that's happen there're many sexual abuse/ra_e allegation \\n\\n\\n\\n</td>\n",
       "      <td>0.085</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.1027</td>\n",
       "      <td>131</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10061</th>\n",
       "      <td>SolomanYue</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>15.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2019-10-29 05:41:44</td>\n",
       "      <td>2019-11-03 04:26:06</td>\n",
       "      <td>2684.0</td>\n",
       "      <td>This is why the Senate should vote on the Hong Kong Human Rights &amp;amp; Democracy Act - and my Be Water Act - now #StandWithHongKong #HongKong https://t.co/DokQaccB2d</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>senate vote hong kong human right amp democracy act water act</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>61</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10137</th>\n",
       "      <td>SolomanYue</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>15.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>2019-10-29 05:41:44</td>\n",
       "      <td>2019-11-03 04:25:39</td>\n",
       "      <td>1498.0</td>\n",
       "      <td>Disturbing reports that Beijing is planning new steps to “safeguard national security” in #HongKong. If so, that will only make things worse.\\n\\n@Senatemajldr McConnell: bring the #HKHumanRightsandDemocracyAct to a vote as soon as possible to support #HK!\\nhttps://t.co/CtAx5vr2Rj</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[@Senatemajldr]</td>\n",
       "      <td>disturb report beijing plan new step “safeguard national security” make thing worse.\\n\\n mcconnell bring vote soon possible support</td>\n",
       "      <td>0.247</td>\n",
       "      <td>0.638</td>\n",
       "      <td>0.115</td>\n",
       "      <td>-0.4767</td>\n",
       "      <td>129</td>\n",
       "      <td>18</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46879</th>\n",
       "      <td>SolomanYue</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>2019-10-29 05:41:44</td>\n",
       "      <td>2019-11-05 12:36:55</td>\n",
       "      <td>4358.0</td>\n",
       "      <td>#LIVE: The ambience at New Town Plaza has intensified again as police have attempted to storm into the mall a 2nd time today. The chaotic scene has led to an arrest of a few. In this footage, a cop covers a man’s face as he tries to shout out his name. #antiRLAB #HongKongProtests https://t.co/vRhNYhtgPR</td>\n",
       "      <td>[live]</td>\n",
       "      <td>[]</td>\n",
       "      <td>ambience new town plaza intensified police attempt storm mall time today chaotic scene lead arrest footage cop cover man’s face try shout name</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0.789</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.6808</td>\n",
       "      <td>142</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50060</th>\n",
       "      <td>SolomanYue</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>2019-10-29 05:41:44</td>\n",
       "      <td>2019-11-05 12:36:55</td>\n",
       "      <td>4380.0</td>\n",
       "      <td>#LIVE: The ambience at New Town Plaza has intensified again as police have attempted to storm into the mall a 2nd time today. The chaotic scene has led to an arrest of a few. In this footage, a cop covers a man’s face as he tries to shout out his name. #antiRLAB #HongKongProtests https://t.co/vRhNYhtgPR</td>\n",
       "      <td>[live]</td>\n",
       "      <td>[]</td>\n",
       "      <td>ambience new town plaza intensified police attempt storm mall time today chaotic scene lead arrest footage cop cover man’s face try shout name</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0.789</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.6808</td>\n",
       "      <td>142</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         username acctdesc location  following  followers  totaltweets  \\\n",
       "10040  SolomanYue      n.a      n.a       15.0        4.0         25.0   \n",
       "10061  SolomanYue      n.a      n.a       15.0        4.0         25.0   \n",
       "10137  SolomanYue      n.a      n.a       15.0        4.0         26.0   \n",
       "46879  SolomanYue      n.a      n.a       21.0        7.0         42.0   \n",
       "50060  SolomanYue      n.a      n.a       21.0        7.0         42.0   \n",
       "\n",
       "             usercreatedts       tweetcreatedts  retweetcount  \\\n",
       "10040  2019-10-29 05:41:44  2019-11-03 04:26:16        2817.0   \n",
       "10061  2019-10-29 05:41:44  2019-11-03 04:26:06        2684.0   \n",
       "10137  2019-10-29 05:41:44  2019-11-03 04:25:39        1498.0   \n",
       "46879  2019-10-29 05:41:44  2019-11-05 12:36:55        4358.0   \n",
       "50060  2019-10-29 05:41:44  2019-11-05 12:36:55        4380.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                        text  \\\n",
       "10040  Arrestees ALL GIRLS. \\nHard to believe they're not caught to serve the cops/PLAs in #HK especially when that's happening in #Xinjiang, and there're so many sexual abuse/ra_e allegations against #hkpolice.\\n\\n#HongKongProtests\\n#HKPoliceState\\n#HKPoliceTerrorism \\n@SolomonYue @HawleyMO https://t.co/2wX803P3lH   \n",
       "10061                                                                                                                                                  This is why the Senate should vote on the Hong Kong Human Rights &amp; Democracy Act - and my Be Water Act - now #StandWithHongKong #HongKong https://t.co/DokQaccB2d   \n",
       "10137                               Disturbing reports that Beijing is planning new steps to “safeguard national security” in #HongKong. If so, that will only make things worse.\\n\\n@Senatemajldr McConnell: bring the #HKHumanRightsandDemocracyAct to a vote as soon as possible to support #HK!\\nhttps://t.co/CtAx5vr2Rj   \n",
       "46879       #LIVE: The ambience at New Town Plaza has intensified again as police have attempted to storm into the mall a 2nd time today. The chaotic scene has led to an arrest of a few. In this footage, a cop covers a man’s face as he tries to shout out his name. #antiRLAB #HongKongProtests https://t.co/vRhNYhtgPR   \n",
       "50060       #LIVE: The ambience at New Town Plaza has intensified again as police have attempted to storm into the mall a 2nd time today. The chaotic scene has led to an arrest of a few. In this footage, a cop covers a man’s face as he tries to shout out his name. #antiRLAB #HongKongProtests https://t.co/vRhNYhtgPR   \n",
       "\n",
       "         hashtags           mentioned_users  \\\n",
       "10040        [hk]  [@SolomonYue, @HawleyMO]   \n",
       "10061          []                        []   \n",
       "10137  [hongkong]           [@Senatemajldr]   \n",
       "46879      [live]                        []   \n",
       "50060      [live]                        []   \n",
       "\n",
       "                                                                                                                                         cleaned_text  \\\n",
       "10040        arrestees girl \\nhard believe they're catch serve cops/plas especially that's happen there're many sexual abuse/ra_e allegation \\n\\n\\n\\n   \n",
       "10061                                                                                   senate vote hong kong human right amp democracy act water act   \n",
       "10137             disturb report beijing plan new step “safeguard national security” make thing worse.\\n\\n mcconnell bring vote soon possible support   \n",
       "46879  ambience new town plaza intensified police attempt storm mall time today chaotic scene lead arrest footage cop cover man’s face try shout name   \n",
       "50060  ambience new town plaza intensified police attempt storm mall time today chaotic scene lead arrest footage cop cover man’s face try shout name   \n",
       "\n",
       "         neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "10040  0.085  0.915  0.000   -0.1027       131        17                2  \n",
       "10061  0.000  1.000  0.000    0.0000        61        11                3  \n",
       "10137  0.247  0.638  0.115   -0.4767       129        18                2  \n",
       "46879  0.211  0.789  0.000   -0.6808       142        23                1  \n",
       "50060  0.211  0.789  0.000   -0.6808       142        23                1  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data['username'] == 'SolomanYue']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6407</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>191.0</td>\n",
       "      <td>66942.0</td>\n",
       "      <td>1404.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-03 03:31:18</td>\n",
       "      <td>87.0</td>\n",
       "      <td>New York City in support of Hong Kong Democracy! \\n\\n#FightForFreedom\\n#StandwithHK https://t.co/5qTBnT2uIj</td>\n",
       "      <td>[fightforfreedom, standwithhk]</td>\n",
       "      <td>[]</td>\n",
       "      <td>new york city support hong kong democracy \\n\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.690</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.4019</td>\n",
       "      <td>45</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12243</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>191.0</td>\n",
       "      <td>66963.0</td>\n",
       "      <td>1406.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-03 04:09:53</td>\n",
       "      <td>90.0</td>\n",
       "      <td>\"HongKongers, we will fight alongside you in the #FightForFreedom\"   -- From Sydney, Australia with love\\n\\n#StandwithHK https://t.co/hu8DUmjhb3</td>\n",
       "      <td>[fightforfreedom, standwithhk]</td>\n",
       "      <td>[]</td>\n",
       "      <td>hongkongers fight alongside sydney australia love\\n\\n</td>\n",
       "      <td>0.241</td>\n",
       "      <td>0.370</td>\n",
       "      <td>0.389</td>\n",
       "      <td>0.3818</td>\n",
       "      <td>51</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30399</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>191.0</td>\n",
       "      <td>67706.0</td>\n",
       "      <td>1422.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-04 14:38:54</td>\n",
       "      <td>21.0</td>\n",
       "      <td>#HongKong supporters in London came together en masse last weekend! Thank you for standing up for democracy, and with #Hongkongers\\n\\n#StandwithHK \\n\\nCredit: @InkyWonders (Perseus) https://t.co/vFz5wVmv4J</td>\n",
       "      <td>[hongkong, hongkongers, standwithhk]</td>\n",
       "      <td>[@InkyWonders]</td>\n",
       "      <td>supporter london come together en masse last weekend thank stand democracy \\n\\n \\n\\ncredit perseus</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.581</td>\n",
       "      <td>0.419</td>\n",
       "      <td>0.7351</td>\n",
       "      <td>94</td>\n",
       "      <td>14</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49700</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>191.0</td>\n",
       "      <td>68046.0</td>\n",
       "      <td>1428.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-05 12:43:52</td>\n",
       "      <td>87.0</td>\n",
       "      <td>House of Commons debates Hong Kong! @DesmondSwayne standing up for the Sino-British Joint Agreement and @joshuawongcf's right to stand as a Council candidate. #StandWithHK #FightForFreedom https://t.co/UXhiqor7CS</td>\n",
       "      <td>[standwithhk, fightforfreedom]</td>\n",
       "      <td>[@DesmondSwayne, @joshuawongcf's]</td>\n",
       "      <td>house common debate hong kong stand sino-british joint agreement right stand council candidate</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.789</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0.4939</td>\n",
       "      <td>94</td>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53332</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>191.0</td>\n",
       "      <td>68050.0</td>\n",
       "      <td>1429.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-05 12:43:52</td>\n",
       "      <td>201.0</td>\n",
       "      <td>House of Commons debates Hong Kong! @DesmondSwayne standing up for the Sino-British Joint Agreement and @joshuawongcf's right to stand as a Council candidate. #StandWithHK #FightForFreedom https://t.co/UXhiqor7CS</td>\n",
       "      <td>[standwithhk, fightforfreedom]</td>\n",
       "      <td>[@DesmondSwayne, @joshuawongcf's]</td>\n",
       "      <td>house common debate hong kong stand sino-british joint agreement right stand council candidate</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.789</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0.4939</td>\n",
       "      <td>94</td>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73262</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>68802.0</td>\n",
       "      <td>1468.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-07 13:49:14</td>\n",
       "      <td>23.0</td>\n",
       "      <td>If this Beijing official really believes the @CommonsForeign report is 'fictitious' then why doesn't his Ministry prove it? \\n\\nThere is no room for foreign interference at UK academic institutions, Mr Shuang. \\n\\n#StandwithHK https://t.co/umeu1vx8H5</td>\n",
       "      <td>[standwithhk]</td>\n",
       "      <td>[@CommonsForeign]</td>\n",
       "      <td>beijing official really believe report fictitious ministry prove \\n\\nthere room foreign interference uk academic institution mr shuang \\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>135</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77283</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>68815.0</td>\n",
       "      <td>1468.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-07 13:49:14</td>\n",
       "      <td>41.0</td>\n",
       "      <td>If this Beijing official really believes the @CommonsForeign report is 'fictitious' then why doesn't his Ministry prove it? \\n\\nThere is no room for foreign interference at UK academic institutions, Mr Shuang. \\n\\n#StandwithHK https://t.co/umeu1vx8H5</td>\n",
       "      <td>[standwithhk]</td>\n",
       "      <td>[@CommonsForeign]</td>\n",
       "      <td>beijing official really believe report fictitious ministry prove \\n\\nthere room foreign interference uk academic institution mr shuang \\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>135</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87358</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>69176.0</td>\n",
       "      <td>1478.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-08 16:45:25</td>\n",
       "      <td>15.0</td>\n",
       "      <td>\"We fear that... the Chinese government’s approach to Hong Kong is moving closer to 'One Country, One System' than it is to maintaining  its treaty commitments under the Joint Declaration”, says the UK's Foreign Affairs Committee. #StandWithHK #HKSOS https://t.co/8wqFnjDZqQ</td>\n",
       "      <td>[standwithhk, hksos]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fear chinese government’s approach hong kong move close one country one system maintain treaty commitment joint declaration” say uk's foreign affair committee</td>\n",
       "      <td>0.124</td>\n",
       "      <td>0.775</td>\n",
       "      <td>0.101</td>\n",
       "      <td>-0.1531</td>\n",
       "      <td>158</td>\n",
       "      <td>22</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96518</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>69203.0</td>\n",
       "      <td>1479.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-08 18:16:27</td>\n",
       "      <td>0.0</td>\n",
       "      <td>//“Rest well on the other side. We will continue walking down this road for you.”\\n\\nChow will not be at Legco, but we will remember him. Hongkongers, keep fighting.//\\n\\nOur Op-Ed at @HongKongFP \\nhttps://t.co/fWRNFiMKan\\n\\n#FightForFreedom \\n#StandwithHK\\n#IfNotNowWhen \\n#DemocracyNow https://t.co/F3lbzHkqPV</td>\n",
       "      <td>[fightforfreedom, standwithhk, ifnotnowwhen, democracynow]</td>\n",
       "      <td>[@HongKongFP]</td>\n",
       "      <td>“rest well side continue walk road you.”\\n\\nchow legco remember hongkongers keep fighting.//\\n\\nour op-ed \\n\\n\\n \\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.870</td>\n",
       "      <td>0.130</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>108</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99763</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>69210.0</td>\n",
       "      <td>1479.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-08 18:20:48</td>\n",
       "      <td>16.0</td>\n",
       "      <td>//“Rest well on the other side. We will continue walking down this road for you.”\\n\\nChow will not be at Legco, but we will remember him. Hongkongers, keep fighting.//\\n\\nOur Op-Ed at @HongKongFP \\nhttps://t.co/fWRNFiMKan\\n\\n#FightForFreedom \\n#StandwithHK\\n#IfNotNowWhen \\n#DemocracyNow</td>\n",
       "      <td>[fightforfreedom, standwithhk, ifnotnowwhen, democracynow]</td>\n",
       "      <td>[@HongKongFP]</td>\n",
       "      <td>“rest well side continue walk road you.”\\n\\nchow legco remember hongkongers keep fighting.//\\n\\nour op-ed \\n\\n\\n \\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.870</td>\n",
       "      <td>0.130</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>108</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117347</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>69442.0</td>\n",
       "      <td>1486.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-09 15:45:40</td>\n",
       "      <td>81.0</td>\n",
       "      <td>Criminals who torture, sexually abuse, verbally abuse #Hongkongers \\n\\nThieves who steal #Hongkongers right to assembly, freedom of expression \\n\\nand Coca Cola \\n\\nThey are called #HongKongPolice \\n\\n#StandwithHK \\n#FightForFreedom https://t.co/XSMoqmbijF</td>\n",
       "      <td>[hongkongers, hongkongers, hongkongpolice, standwithhk, fightforfreedom]</td>\n",
       "      <td>[]</td>\n",
       "      <td>criminal torture sexually abuse verbally abuse \\n\\nthieves steal right assembly freedom expression \\n\\nand coca cola \\n\\nthey call \\n\\n</td>\n",
       "      <td>0.610</td>\n",
       "      <td>0.275</td>\n",
       "      <td>0.115</td>\n",
       "      <td>-0.9584</td>\n",
       "      <td>127</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129118</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>69461.0</td>\n",
       "      <td>1487.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-09 17:29:41</td>\n",
       "      <td>56.0</td>\n",
       "      <td>We demand an explanation from @warwickuni on why it took down the pig image in campus.  \\n\\nThe pig is NOT a symbol of  'racist' and the #HongKong students have the right to appeal.\\n\\nUK Universities should uphold the freedom of speech. \\n\\nhttps://t.co/w61f8VmwwE\\n\\n#StandwithHK</td>\n",
       "      <td>[hongkong, standwithhk]</td>\n",
       "      <td>[@warwickuni]</td>\n",
       "      <td>demand explanation take pig image campus \\n\\nthe pig symbol racist student right appeal.\\n\\nuk university uphold freedom speech \\n\\n\\n\\n</td>\n",
       "      <td>0.223</td>\n",
       "      <td>0.607</td>\n",
       "      <td>0.170</td>\n",
       "      <td>-0.0772</td>\n",
       "      <td>128</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146867</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>194.0</td>\n",
       "      <td>70777.0</td>\n",
       "      <td>1518.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-11 11:15:20</td>\n",
       "      <td>105.0</td>\n",
       "      <td>Good grief. One would think she would have learned by now.  Cambridge University must remove her honorary fellowship, Britain should use Magnitsky laws to freeze her assets &amp;amp; the world must act #HongKong #DemocracyForHK #StandWithHongKong #StandwithHK @hk_watch @lukedepulford https://t.co/KK0aCgPk4s</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@hk_watch, @lukedepulford]</td>\n",
       "      <td>good grief one would think would learn cambridge university must remove honorary fellowship britain use magnitsky law freeze asset amp world must act</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.298</td>\n",
       "      <td>0.5859</td>\n",
       "      <td>149</td>\n",
       "      <td>23</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167263</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>197.0</td>\n",
       "      <td>72065.0</td>\n",
       "      <td>1564.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-12 14:10:24</td>\n",
       "      <td>106.0</td>\n",
       "      <td>Please pass the Hong Kong Human Rights and Democracy Act and the Hong Kong Be Water Act. \\n\\nThe police cannot besiege a University and act with impunity. \\n\\nThe United State must sanction these perpetrators to exhibit its leadership in the world on human rights. \\n\\n#StandwithHK https://t.co/FBEkaikvg8</td>\n",
       "      <td>[standwithhk]</td>\n",
       "      <td>[]</td>\n",
       "      <td>please pas hong kong human right democracy act hong kong water act \\n\\nthe police cannot besiege university act impunity \\n\\nthe united state must sanction perpetrator exhibit leadership world human right \\n\\n</td>\n",
       "      <td>0.091</td>\n",
       "      <td>0.765</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.2263</td>\n",
       "      <td>203</td>\n",
       "      <td>31</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170242</th>\n",
       "      <td>Stand_with_HK</td>\n",
       "      <td>a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>197.0</td>\n",
       "      <td>72171.0</td>\n",
       "      <td>1567.0</td>\n",
       "      <td>2017-11-24 21:37:26</td>\n",
       "      <td>2019-11-12 14:32:05</td>\n",
       "      <td>96.0</td>\n",
       "      <td>\"Resist now or we will not have a chance again!\" \\n\\nPlease #standwithHK during this critical moment in human history. Resist the CCP tyranny! Rise up against Hong Kong government! https://t.co/MTpBrjEG6x</td>\n",
       "      <td>[standwithhk]</td>\n",
       "      <td>[]</td>\n",
       "      <td>resist chance \\n\\nplease critical moment human history resist ccp tyranny rise hong kong government</td>\n",
       "      <td>0.131</td>\n",
       "      <td>0.625</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.2500</td>\n",
       "      <td>97</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             username  \\\n",
       "6407    Stand_with_HK   \n",
       "12243   Stand_with_HK   \n",
       "30399   Stand_with_HK   \n",
       "49700   Stand_with_HK   \n",
       "53332   Stand_with_HK   \n",
       "73262   Stand_with_HK   \n",
       "77283   Stand_with_HK   \n",
       "87358   Stand_with_HK   \n",
       "96518   Stand_with_HK   \n",
       "99763   Stand_with_HK   \n",
       "117347  Stand_with_HK   \n",
       "129118  Stand_with_HK   \n",
       "146867  Stand_with_HK   \n",
       "167263  Stand_with_HK   \n",
       "170242  Stand_with_HK   \n",
       "\n",
       "                                                                                                                               acctdesc  \\\n",
       "6407    a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "12243   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "30399   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "49700   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "53332   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "73262   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "77283   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "87358   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "96518   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "99763   a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "117347  a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "129118  a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "146867  a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "167263  a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "170242  a group of ordinary hongkongers urging the world to safeguard hong kong’s rights and freedoms. #standwithhk  (rt ≠ endorsement)   \n",
       "\n",
       "          location  following  followers  totaltweets        usercreatedts  \\\n",
       "6407    hong kong       191.0    66942.0       1404.0  2017-11-24 21:37:26   \n",
       "12243   hong kong       191.0    66963.0       1406.0  2017-11-24 21:37:26   \n",
       "30399   hong kong       191.0    67706.0       1422.0  2017-11-24 21:37:26   \n",
       "49700   hong kong       191.0    68046.0       1428.0  2017-11-24 21:37:26   \n",
       "53332   hong kong       191.0    68050.0       1429.0  2017-11-24 21:37:26   \n",
       "73262   hong kong       194.0    68802.0       1468.0  2017-11-24 21:37:26   \n",
       "77283   hong kong       194.0    68815.0       1468.0  2017-11-24 21:37:26   \n",
       "87358   hong kong       194.0    69176.0       1478.0  2017-11-24 21:37:26   \n",
       "96518   hong kong       194.0    69203.0       1479.0  2017-11-24 21:37:26   \n",
       "99763   hong kong       194.0    69210.0       1479.0  2017-11-24 21:37:26   \n",
       "117347  hong kong       194.0    69442.0       1486.0  2017-11-24 21:37:26   \n",
       "129118  hong kong       194.0    69461.0       1487.0  2017-11-24 21:37:26   \n",
       "146867  hong kong       194.0    70777.0       1518.0  2017-11-24 21:37:26   \n",
       "167263  hong kong       197.0    72065.0       1564.0  2017-11-24 21:37:26   \n",
       "170242  hong kong       197.0    72171.0       1567.0  2017-11-24 21:37:26   \n",
       "\n",
       "             tweetcreatedts  retweetcount  \\\n",
       "6407    2019-11-03 03:31:18          87.0   \n",
       "12243   2019-11-03 04:09:53          90.0   \n",
       "30399   2019-11-04 14:38:54          21.0   \n",
       "49700   2019-11-05 12:43:52          87.0   \n",
       "53332   2019-11-05 12:43:52         201.0   \n",
       "73262   2019-11-07 13:49:14          23.0   \n",
       "77283   2019-11-07 13:49:14          41.0   \n",
       "87358   2019-11-08 16:45:25          15.0   \n",
       "96518   2019-11-08 18:16:27           0.0   \n",
       "99763   2019-11-08 18:20:48          16.0   \n",
       "117347  2019-11-09 15:45:40          81.0   \n",
       "129118  2019-11-09 17:29:41          56.0   \n",
       "146867  2019-11-11 11:15:20         105.0   \n",
       "167263  2019-11-12 14:10:24         106.0   \n",
       "170242  2019-11-12 14:32:05          96.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                           text  \\\n",
       "6407                                                                                                                                                                                                                New York City in support of Hong Kong Democracy! \\n\\n#FightForFreedom\\n#StandwithHK https://t.co/5qTBnT2uIj   \n",
       "12243                                                                                                                                                                          \"HongKongers, we will fight alongside you in the #FightForFreedom\"   -- From Sydney, Australia with love\\n\\n#StandwithHK https://t.co/hu8DUmjhb3   \n",
       "30399                                                                                                             #HongKong supporters in London came together en masse last weekend! Thank you for standing up for democracy, and with #Hongkongers\\n\\n#StandwithHK \\n\\nCredit: @InkyWonders (Perseus) https://t.co/vFz5wVmv4J   \n",
       "49700                                                                                                      House of Commons debates Hong Kong! @DesmondSwayne standing up for the Sino-British Joint Agreement and @joshuawongcf's right to stand as a Council candidate. #StandWithHK #FightForFreedom https://t.co/UXhiqor7CS   \n",
       "53332                                                                                                      House of Commons debates Hong Kong! @DesmondSwayne standing up for the Sino-British Joint Agreement and @joshuawongcf's right to stand as a Council candidate. #StandWithHK #FightForFreedom https://t.co/UXhiqor7CS   \n",
       "73262                                                                If this Beijing official really believes the @CommonsForeign report is 'fictitious' then why doesn't his Ministry prove it? \\n\\nThere is no room for foreign interference at UK academic institutions, Mr Shuang. \\n\\n#StandwithHK https://t.co/umeu1vx8H5   \n",
       "77283                                                                If this Beijing official really believes the @CommonsForeign report is 'fictitious' then why doesn't his Ministry prove it? \\n\\nThere is no room for foreign interference at UK academic institutions, Mr Shuang. \\n\\n#StandwithHK https://t.co/umeu1vx8H5   \n",
       "87358                                        \"We fear that... the Chinese government’s approach to Hong Kong is moving closer to 'One Country, One System' than it is to maintaining  its treaty commitments under the Joint Declaration”, says the UK's Foreign Affairs Committee. #StandWithHK #HKSOS https://t.co/8wqFnjDZqQ   \n",
       "96518   //“Rest well on the other side. We will continue walking down this road for you.”\\n\\nChow will not be at Legco, but we will remember him. Hongkongers, keep fighting.//\\n\\nOur Op-Ed at @HongKongFP \\nhttps://t.co/fWRNFiMKan\\n\\n#FightForFreedom \\n#StandwithHK\\n#IfNotNowWhen \\n#DemocracyNow https://t.co/F3lbzHkqPV   \n",
       "99763                           //“Rest well on the other side. We will continue walking down this road for you.”\\n\\nChow will not be at Legco, but we will remember him. Hongkongers, keep fighting.//\\n\\nOur Op-Ed at @HongKongFP \\nhttps://t.co/fWRNFiMKan\\n\\n#FightForFreedom \\n#StandwithHK\\n#IfNotNowWhen \\n#DemocracyNow   \n",
       "117347                                                         Criminals who torture, sexually abuse, verbally abuse #Hongkongers \\n\\nThieves who steal #Hongkongers right to assembly, freedom of expression \\n\\nand Coca Cola \\n\\nThey are called #HongKongPolice \\n\\n#StandwithHK \\n#FightForFreedom https://t.co/XSMoqmbijF   \n",
       "129118                                We demand an explanation from @warwickuni on why it took down the pig image in campus.  \\n\\nThe pig is NOT a symbol of  'racist' and the #HongKong students have the right to appeal.\\n\\nUK Universities should uphold the freedom of speech. \\n\\nhttps://t.co/w61f8VmwwE\\n\\n#StandwithHK   \n",
       "146867         Good grief. One would think she would have learned by now.  Cambridge University must remove her honorary fellowship, Britain should use Magnitsky laws to freeze her assets &amp; the world must act #HongKong #DemocracyForHK #StandWithHongKong #StandwithHK @hk_watch @lukedepulford https://t.co/KK0aCgPk4s   \n",
       "167263        Please pass the Hong Kong Human Rights and Democracy Act and the Hong Kong Be Water Act. \\n\\nThe police cannot besiege a University and act with impunity. \\n\\nThe United State must sanction these perpetrators to exhibit its leadership in the world on human rights. \\n\\n#StandwithHK https://t.co/FBEkaikvg8   \n",
       "170242                                                                                                             \"Resist now or we will not have a chance again!\" \\n\\nPlease #standwithHK during this critical moment in human history. Resist the CCP tyranny! Rise up against Hong Kong government! https://t.co/MTpBrjEG6x   \n",
       "\n",
       "                                                                        hashtags  \\\n",
       "6407                                              [fightforfreedom, standwithhk]   \n",
       "12243                                             [fightforfreedom, standwithhk]   \n",
       "30399                                       [hongkong, hongkongers, standwithhk]   \n",
       "49700                                             [standwithhk, fightforfreedom]   \n",
       "53332                                             [standwithhk, fightforfreedom]   \n",
       "73262                                                              [standwithhk]   \n",
       "77283                                                              [standwithhk]   \n",
       "87358                                                       [standwithhk, hksos]   \n",
       "96518                 [fightforfreedom, standwithhk, ifnotnowwhen, democracynow]   \n",
       "99763                 [fightforfreedom, standwithhk, ifnotnowwhen, democracynow]   \n",
       "117347  [hongkongers, hongkongers, hongkongpolice, standwithhk, fightforfreedom]   \n",
       "129118                                                   [hongkong, standwithhk]   \n",
       "146867                                                                        []   \n",
       "167263                                                             [standwithhk]   \n",
       "170242                                                             [standwithhk]   \n",
       "\n",
       "                          mentioned_users  \\\n",
       "6407                                   []   \n",
       "12243                                  []   \n",
       "30399                      [@InkyWonders]   \n",
       "49700   [@DesmondSwayne, @joshuawongcf's]   \n",
       "53332   [@DesmondSwayne, @joshuawongcf's]   \n",
       "73262                   [@CommonsForeign]   \n",
       "77283                   [@CommonsForeign]   \n",
       "87358                                  []   \n",
       "96518                       [@HongKongFP]   \n",
       "99763                       [@HongKongFP]   \n",
       "117347                                 []   \n",
       "129118                      [@warwickuni]   \n",
       "146867        [@hk_watch, @lukedepulford]   \n",
       "167263                                 []   \n",
       "170242                                 []   \n",
       "\n",
       "                                                                                                                                                                                                             cleaned_text  \\\n",
       "6407                                                                                                                                                                     new york city support hong kong democracy \\n\\n\\n   \n",
       "12243                                                                                                                                                               hongkongers fight alongside sydney australia love\\n\\n   \n",
       "30399                                                                                                                  supporter london come together en masse last weekend thank stand democracy \\n\\n \\n\\ncredit perseus   \n",
       "49700                                                                                                                      house common debate hong kong stand sino-british joint agreement right stand council candidate   \n",
       "53332                                                                                                                      house common debate hong kong stand sino-british joint agreement right stand council candidate   \n",
       "73262                                                                         beijing official really believe report fictitious ministry prove \\n\\nthere room foreign interference uk academic institution mr shuang \\n\\n   \n",
       "77283                                                                         beijing official really believe report fictitious ministry prove \\n\\nthere room foreign interference uk academic institution mr shuang \\n\\n   \n",
       "87358                                                      fear chinese government’s approach hong kong move close one country one system maintain treaty commitment joint declaration” say uk's foreign affair committee   \n",
       "96518                                                                                               “rest well side continue walk road you.”\\n\\nchow legco remember hongkongers keep fighting.//\\n\\nour op-ed \\n\\n\\n \\n\\n   \n",
       "99763                                                                                               “rest well side continue walk road you.”\\n\\nchow legco remember hongkongers keep fighting.//\\n\\nour op-ed \\n\\n\\n \\n\\n   \n",
       "117347                                                                            criminal torture sexually abuse verbally abuse \\n\\nthieves steal right assembly freedom expression \\n\\nand coca cola \\n\\nthey call \\n\\n   \n",
       "129118                                                                           demand explanation take pig image campus \\n\\nthe pig symbol racist student right appeal.\\n\\nuk university uphold freedom speech \\n\\n\\n\\n   \n",
       "146867                                                              good grief one would think would learn cambridge university must remove honorary fellowship britain use magnitsky law freeze asset amp world must act   \n",
       "167263  please pas hong kong human right democracy act hong kong water act \\n\\nthe police cannot besiege university act impunity \\n\\nthe united state must sanction perpetrator exhibit leadership world human right \\n\\n   \n",
       "170242                                                                                                                resist chance \\n\\nplease critical moment human history resist ccp tyranny rise hong kong government   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "6407    0.000  0.690  0.310    0.4019        45         8                4  \n",
       "12243   0.241  0.370  0.389    0.3818        51         6                4  \n",
       "30399   0.000  0.581  0.419    0.7351        94        14                5  \n",
       "49700   0.000  0.789  0.211    0.4939        94        13                4  \n",
       "53332   0.000  0.789  0.211    0.4939        94        13                4  \n",
       "73262   0.000  1.000  0.000    0.0000       135        18                3  \n",
       "77283   0.000  1.000  0.000    0.0000       135        18                3  \n",
       "87358   0.124  0.775  0.101   -0.1531       158        22                2  \n",
       "96518   0.000  0.870  0.130    0.2732       108        15                4  \n",
       "99763   0.000  0.870  0.130    0.2732       108        15                4  \n",
       "117347  0.610  0.275  0.115   -0.9584       127        18                1  \n",
       "129118  0.223  0.607  0.170   -0.0772       128        18                3  \n",
       "146867  0.106  0.596  0.298    0.5859       149        23                5  \n",
       "167263  0.091  0.765  0.144    0.2263       203        31                4  \n",
       "170242  0.131  0.625  0.244    0.2500        97        14                4  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data['username'] == 'Stand_with_HK']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "username_frequency = data['username'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "username\n",
       "xzxzanalazy      643\n",
       "belleng324       540\n",
       "wdamidoinhere    395\n",
       "natalie_hoyin    272\n",
       "hky2147          252\n",
       "                ... \n",
       "Dawnwillcome1      8\n",
       "yeung08572660      8\n",
       "J_oktokki_k        8\n",
       "irisluv3           8\n",
       "HandelnH           8\n",
       "Name: count, Length: 7498, dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "username_frequency[username_frequency > 7]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5483</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>708.0</td>\n",
       "      <td>680.0</td>\n",
       "      <td>27446.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-03 03:37:49</td>\n",
       "      <td>33.0</td>\n",
       "      <td>Here are what happened yesterday in HK. \\nIf you think it is ridiculous, it happens in HK almost everyday.  \\n@SolomonYue @SouthPark \\n#HKprotests #HKPoliceState #StandwithHonKong #antichinazi #hongkongpolicebrutality #HKPoliceTerrorism https://t.co/ysaVesiAkg</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SolomonYue, @SouthPark]</td>\n",
       "      <td>happen yesterday hk \\nif think ridiculous happens hk almost everyday</td>\n",
       "      <td>0.217</td>\n",
       "      <td>0.783</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.3612</td>\n",
       "      <td>67</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5716</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>708.0</td>\n",
       "      <td>680.0</td>\n",
       "      <td>27446.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-03 03:36:05</td>\n",
       "      <td>59.0</td>\n",
       "      <td>#HKPolice attacked a firefighter, after they fired tear gas at the fire engine he was operating\\n\\nAbsolutely disgusting by the #HKPoliceTerrorists\\n\\nYou should support the Emergency service personnel (Fire Service and Hospital workers and Medics)\\n\\nSource: FB\\n#SOSHK\\n#HKPoliceState https://t.co/HiQJzLdbdf</td>\n",
       "      <td>[hkpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>attack firefighter fire tear gas fire engine operating\\n\\nabsolutely disgust \\n\\nyou support emergency service personnel fire service hospital worker medics)\\n\\nsource fb\\n\\n</td>\n",
       "      <td>0.488</td>\n",
       "      <td>0.428</td>\n",
       "      <td>0.085</td>\n",
       "      <td>-0.9206</td>\n",
       "      <td>166</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5730</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>708.0</td>\n",
       "      <td>680.0</td>\n",
       "      <td>27446.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-03 03:36:01</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2 Nov 2019 Mongkok\\n#HKPolice bombed people on the street with tear gas right over their heads\\n#Murderer #HKRioters #HKPoliceBrutality #HKPoliceTerrorism #HKHumanRightsandDemocracyAct \\n\\n@benedictrogers @HawleyMO @WhiteHouse @senatemajldr @SolomonYue https://t.co/cIIzZ4PbHP</td>\n",
       "      <td>[hkpolice, murderer, hkrioters]</td>\n",
       "      <td>[@benedictrogers, @HawleyMO, @WhiteHouse, @senatemajldr, @SolomonYue]</td>\n",
       "      <td>nov mongkok\\n bomb people street tear gas right heads\\n \\n\\n</td>\n",
       "      <td>0.286</td>\n",
       "      <td>0.714</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.4939</td>\n",
       "      <td>56</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5734</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>708.0</td>\n",
       "      <td>680.0</td>\n",
       "      <td>27446.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-03 03:35:59</td>\n",
       "      <td>636.0</td>\n",
       "      <td>#SOSHK It is worrying that #HongKong police allegedly target young female protestors for unjustified mass arrests. \\nShot in HK in the city centre near Southern Hotel https://t.co/DFKewXWFYs</td>\n",
       "      <td>[soshk, hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>worry police allegedly target young female protestors unjustified mass arrest \\nshot hk city centre near southern hotel</td>\n",
       "      <td>0.261</td>\n",
       "      <td>0.739</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.6486</td>\n",
       "      <td>118</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5754</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>708.0</td>\n",
       "      <td>680.0</td>\n",
       "      <td>27446.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-03 03:35:51</td>\n",
       "      <td>678.0</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.498</td>\n",
       "      <td>0.137</td>\n",
       "      <td>-0.7430</td>\n",
       "      <td>152</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240453</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>847.0</td>\n",
       "      <td>849.0</td>\n",
       "      <td>40113.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-19 16:15:14</td>\n",
       "      <td>7.0</td>\n",
       "      <td>This is a video of last midnight,\\nHong Kong Police arrest citizens on a roof top.hand tied behind their backs.\\n\\nWhat did they do?You need to tied them like this??unbelievable!!\\n\\n#HKPoliceTerrorism \\n#soshongkong https://t.co/Ft6DLec2BL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>video last midnight,\\nhong kong police arrest citizen roof top.hand tie behind backs.\\n\\nwhat do?you need tie like this??unbelievable!!\\n\\n</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.689</td>\n",
       "      <td>0.108</td>\n",
       "      <td>-0.3013</td>\n",
       "      <td>134</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240553</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>847.0</td>\n",
       "      <td>849.0</td>\n",
       "      <td>40113.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-19 16:14:49</td>\n",
       "      <td>7.0</td>\n",
       "      <td>@hklemontea Tear gas was shot by #HKPoliceTerrorism https://t.co/NcvNlcK2cP</td>\n",
       "      <td>[hkpoliceterrorism]</td>\n",
       "      <td>[@hklemontea]</td>\n",
       "      <td>tear gas shot</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240580</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>847.0</td>\n",
       "      <td>849.0</td>\n",
       "      <td>40113.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-19 16:14:44</td>\n",
       "      <td>6293.0</td>\n",
       "      <td>Under the Sino-Brit Joint Declaration China agreed to respect Hong Kong’s sovereignty. This weekend, mainland Chinese soldiers were deployed to Hong Kong's streets\\n\\nTrump admin officials should loudly oppose China's aggression &amp;amp; support #HongKongProtests\\nhttps://t.co/jxNBI1pRDt</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>sino-brit joint declaration china agree respect hong kong’s sovereignty weekend mainland chinese soldier deploy hong kong's streets\\n\\ntrump admin official loudly oppose china's aggression amp support</td>\n",
       "      <td>0.068</td>\n",
       "      <td>0.677</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.7269</td>\n",
       "      <td>198</td>\n",
       "      <td>25</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240756</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>847.0</td>\n",
       "      <td>849.0</td>\n",
       "      <td>40113.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-19 16:14:05</td>\n",
       "      <td>20.0</td>\n",
       "      <td>The first of the group that made a last minute dash for freedom from Polytechnic University are taken away in handcuffs. The standoff is nearing an end. #HongKong https://t.co/F4yRI0k5hg</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>first group make last minute dash freedom polytechnic university take away handcuffs standoff near end</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.769</td>\n",
       "      <td>0.231</td>\n",
       "      <td>0.6369</td>\n",
       "      <td>102</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240794</th>\n",
       "      <td>xzxzanalazy</td>\n",
       "      <td>hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>847.0</td>\n",
       "      <td>849.0</td>\n",
       "      <td>40113.0</td>\n",
       "      <td>2018-03-04 14:47:07</td>\n",
       "      <td>2019-11-19 16:14:00</td>\n",
       "      <td>11.0</td>\n",
       "      <td>A simple cartoon explaining why #Hongkongers need to fight for what they want for 5 months. If we lose this time, we will never can get our things back and will be added more regulations by the central government. \\n#StandWithHongKong #SOSHK #SOSPolyU https://t.co/2S23i2Qfvl</td>\n",
       "      <td>[hongkongers]</td>\n",
       "      <td>[]</td>\n",
       "      <td>simple cartoon explain need fight want month lose time never get thing back add regulation central government</td>\n",
       "      <td>0.257</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.063</td>\n",
       "      <td>-0.6124</td>\n",
       "      <td>109</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>643 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           username  \\\n",
       "5483    xzxzanalazy   \n",
       "5716    xzxzanalazy   \n",
       "5730    xzxzanalazy   \n",
       "5734    xzxzanalazy   \n",
       "5754    xzxzanalazy   \n",
       "...             ...   \n",
       "240453  xzxzanalazy   \n",
       "240553  xzxzanalazy   \n",
       "240580  xzxzanalazy   \n",
       "240756  xzxzanalazy   \n",
       "240794  xzxzanalazy   \n",
       "\n",
       "                                                                                                                acctdesc  \\\n",
       "5483        hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong   \n",
       "5716        hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong   \n",
       "5730        hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong   \n",
       "5734        hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong   \n",
       "5754        hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. #followbackhongkong   \n",
       "...                                                                                                                  ...   \n",
       "240453  hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong   \n",
       "240553  hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong   \n",
       "240580  hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong   \n",
       "240756  hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong   \n",
       "240794  hongkonger, my love is for liverpool!!!! fight for freedom, please stand with hong kong. 🖐🏻😷🎗#followbackhongkong   \n",
       "\n",
       "         location  following  followers  totaltweets        usercreatedts  \\\n",
       "5483    hong kong      708.0      680.0      27446.0  2018-03-04 14:47:07   \n",
       "5716    hong kong      708.0      680.0      27446.0  2018-03-04 14:47:07   \n",
       "5730    hong kong      708.0      680.0      27446.0  2018-03-04 14:47:07   \n",
       "5734    hong kong      708.0      680.0      27446.0  2018-03-04 14:47:07   \n",
       "5754    hong kong      708.0      680.0      27446.0  2018-03-04 14:47:07   \n",
       "...           ...        ...        ...          ...                  ...   \n",
       "240453  hong kong      847.0      849.0      40113.0  2018-03-04 14:47:07   \n",
       "240553  hong kong      847.0      849.0      40113.0  2018-03-04 14:47:07   \n",
       "240580  hong kong      847.0      849.0      40113.0  2018-03-04 14:47:07   \n",
       "240756  hong kong      847.0      849.0      40113.0  2018-03-04 14:47:07   \n",
       "240794  hong kong      847.0      849.0      40113.0  2018-03-04 14:47:07   \n",
       "\n",
       "             tweetcreatedts  retweetcount  \\\n",
       "5483    2019-11-03 03:37:49          33.0   \n",
       "5716    2019-11-03 03:36:05          59.0   \n",
       "5730    2019-11-03 03:36:01          22.0   \n",
       "5734    2019-11-03 03:35:59         636.0   \n",
       "5754    2019-11-03 03:35:51         678.0   \n",
       "...                     ...           ...   \n",
       "240453  2019-11-19 16:15:14           7.0   \n",
       "240553  2019-11-19 16:14:49           7.0   \n",
       "240580  2019-11-19 16:14:44        6293.0   \n",
       "240756  2019-11-19 16:14:05          20.0   \n",
       "240794  2019-11-19 16:14:00          11.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                          text  \\\n",
       "5483                                                      Here are what happened yesterday in HK. \\nIf you think it is ridiculous, it happens in HK almost everyday.  \\n@SolomonYue @SouthPark \\n#HKprotests #HKPoliceState #StandwithHonKong #antichinazi #hongkongpolicebrutality #HKPoliceTerrorism https://t.co/ysaVesiAkg   \n",
       "5716    #HKPolice attacked a firefighter, after they fired tear gas at the fire engine he was operating\\n\\nAbsolutely disgusting by the #HKPoliceTerrorists\\n\\nYou should support the Emergency service personnel (Fire Service and Hospital workers and Medics)\\n\\nSource: FB\\n#SOSHK\\n#HKPoliceState https://t.co/HiQJzLdbdf   \n",
       "5730                                      2 Nov 2019 Mongkok\\n#HKPolice bombed people on the street with tear gas right over their heads\\n#Murderer #HKRioters #HKPoliceBrutality #HKPoliceTerrorism #HKHumanRightsandDemocracyAct \\n\\n@benedictrogers @HawleyMO @WhiteHouse @senatemajldr @SolomonYue https://t.co/cIIzZ4PbHP   \n",
       "5734                                                                                                                            #SOSHK It is worrying that #HongKong police allegedly target young female protestors for unjustified mass arrests. \\nShot in HK in the city centre near Southern Hotel https://t.co/DFKewXWFYs   \n",
       "5754               A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "...                                                                                                                                                                                                                                                                                                                        ...   \n",
       "240453                                                                        This is a video of last midnight,\\nHong Kong Police arrest citizens on a roof top.hand tied behind their backs.\\n\\nWhat did they do?You need to tied them like this??unbelievable!!\\n\\n#HKPoliceTerrorism \\n#soshongkong https://t.co/Ft6DLec2BL   \n",
       "240553                                                                                                                                                                                                                                             @hklemontea Tear gas was shot by #HKPoliceTerrorism https://t.co/NcvNlcK2cP   \n",
       "240580                           Under the Sino-Brit Joint Declaration China agreed to respect Hong Kong’s sovereignty. This weekend, mainland Chinese soldiers were deployed to Hong Kong's streets\\n\\nTrump admin officials should loudly oppose China's aggression &amp; support #HongKongProtests\\nhttps://t.co/jxNBI1pRDt   \n",
       "240756                                                                                                                              The first of the group that made a last minute dash for freedom from Polytechnic University are taken away in handcuffs. The standoff is nearing an end. #HongKong https://t.co/F4yRI0k5hg   \n",
       "240794                                     A simple cartoon explaining why #Hongkongers need to fight for what they want for 5 months. If we lose this time, we will never can get our things back and will be added more regulations by the central government. \\n#StandWithHongKong #SOSHK #SOSPolyU https://t.co/2S23i2Qfvl   \n",
       "\n",
       "                               hashtags  \\\n",
       "5483                                 []   \n",
       "5716                         [hkpolice]   \n",
       "5730    [hkpolice, murderer, hkrioters]   \n",
       "5734                  [soshk, hongkong]   \n",
       "5754                   [hongkongpolice]   \n",
       "...                                 ...   \n",
       "240453                               []   \n",
       "240553              [hkpoliceterrorism]   \n",
       "240580                               []   \n",
       "240756                               []   \n",
       "240794                    [hongkongers]   \n",
       "\n",
       "                                                              mentioned_users  \\\n",
       "5483                                                [@SolomonYue, @SouthPark]   \n",
       "5716                                                                       []   \n",
       "5730    [@benedictrogers, @HawleyMO, @WhiteHouse, @senatemajldr, @SolomonYue]   \n",
       "5734                                                                       []   \n",
       "5754                                                                       []   \n",
       "...                                                                       ...   \n",
       "240453                                                                     []   \n",
       "240553                                                          [@hklemontea]   \n",
       "240580                                                                     []   \n",
       "240756                                                                     []   \n",
       "240794                                                                     []   \n",
       "\n",
       "                                                                                                                                                                                                    cleaned_text  \\\n",
       "5483                                                                                                                                        happen yesterday hk \\nif think ridiculous happens hk almost everyday   \n",
       "5716                              attack firefighter fire tear gas fire engine operating\\n\\nabsolutely disgust \\n\\nyou support emergency service personnel fire service hospital worker medics)\\n\\nsource fb\\n\\n   \n",
       "5730                                                                                                                                                nov mongkok\\n bomb people street tear gas right heads\\n \\n\\n   \n",
       "5734                                                                                     worry police allegedly target young female protestors unjustified mass arrest \\nshot hk city centre near southern hotel   \n",
       "5754                                                  fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away   \n",
       "...                                                                                                                                                                                                          ...   \n",
       "240453                                                               video last midnight,\\nhong kong police arrest citizen roof top.hand tie behind backs.\\n\\nwhat do?you need tie like this??unbelievable!!\\n\\n   \n",
       "240553                                                                                                                                                                                             tear gas shot   \n",
       "240580  sino-brit joint declaration china agree respect hong kong’s sovereignty weekend mainland chinese soldier deploy hong kong's streets\\n\\ntrump admin official loudly oppose china's aggression amp support   \n",
       "240756                                                                                                    first group make last minute dash freedom polytechnic university take away handcuffs standoff near end   \n",
       "240794                                                                                             simple cartoon explain need fight want month lose time never get thing back add regulation central government   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "5483    0.217  0.783  0.000   -0.3612        67        10                2  \n",
       "5716    0.488  0.428  0.085   -0.9206       166        20                1  \n",
       "5730    0.286  0.714  0.000   -0.4939        56        10                2  \n",
       "5734    0.261  0.739  0.000   -0.6486       118        17                1  \n",
       "5754    0.364  0.498  0.137   -0.7430       152        23                1  \n",
       "...       ...    ...    ...       ...       ...       ...              ...  \n",
       "240453  0.203  0.689  0.108   -0.3013       134        17                2  \n",
       "240553  0.000  1.000  0.000    0.0000        13         3                3  \n",
       "240580  0.068  0.677  0.255    0.7269       198        25                5  \n",
       "240756  0.000  0.769  0.231    0.6369       102        15                5  \n",
       "240794  0.257  0.680  0.063   -0.6124       109        17                1  \n",
       "\n",
       "[643 rows x 20 columns]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data['username'] == 'xzxzanalazy']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>five5demands</td>\n",
       "      <td>hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk</td>\n",
       "      <td>n.a</td>\n",
       "      <td>437.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>466.0</td>\n",
       "      <td>2019-10-05 13:40:37</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>292.0</td>\n",
       "      <td>Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>disgust police mark protestors pen write number hand like nazi germany</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.391</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>70</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dejavu53328974</td>\n",
       "      <td>#followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>581.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>7835.0</td>\n",
       "      <td>2019-08-27 15:47:34</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>1315.0</td>\n",
       "      <td>#China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH</td>\n",
       "      <td>[china, policestate]</td>\n",
       "      <td>[@hkgetv]</td>\n",
       "      <td>teargas canister stick road citizen try pull failed\\ncredit</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.543</td>\n",
       "      <td>0.202</td>\n",
       "      <td>-0.1779</td>\n",
       "      <td>58</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tsksimon</td>\n",
       "      <td>#fightforfreedom #standwithhongkong</td>\n",
       "      <td>n.a</td>\n",
       "      <td>825.0</td>\n",
       "      <td>456.0</td>\n",
       "      <td>6728.0</td>\n",
       "      <td>2014-10-05 15:57:07</td>\n",
       "      <td>2019-11-03 02:57:49</td>\n",
       "      <td>105.0</td>\n",
       "      <td>This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V</td>\n",
       "      <td>[hkpolicebrutality]</td>\n",
       "      <td>[]</td>\n",
       "      <td>tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence</td>\n",
       "      <td>0.339</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.070</td>\n",
       "      <td>-0.8074</td>\n",
       "      <td>148</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JimmyWo67187904</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>656.0</td>\n",
       "      <td>2019-10-08 12:58:51</td>\n",
       "      <td>2019-11-03 02:57:48</td>\n",
       "      <td>660.0</td>\n",
       "      <td>A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL</td>\n",
       "      <td>[hongkongpolice]</td>\n",
       "      <td>[]</td>\n",
       "      <td>fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away</td>\n",
       "      <td>0.364</td>\n",
       "      <td>0.498</td>\n",
       "      <td>0.137</td>\n",
       "      <td>-0.7430</td>\n",
       "      <td>152</td>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ARCHI418</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>19.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>2014-06-19 16:19:36</td>\n",
       "      <td>2019-11-03 02:57:46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>#police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z</td>\n",
       "      <td>[police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>threw hand blast citizen’s back make large area burn tear gas read hand grenade kill</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.519</td>\n",
       "      <td>0.277</td>\n",
       "      <td>0.1779</td>\n",
       "      <td>84</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248646</th>\n",
       "      <td>zizihui1</td>\n",
       "      <td>five demands, not one less 🇭🇰#followbackhongkong #科勞手足</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>505.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>1301.0</td>\n",
       "      <td>2019-09-25 17:05:21</td>\n",
       "      <td>2019-11-19 17:19:37</td>\n",
       "      <td>72.0</td>\n",
       "      <td>@marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@marcorubio]</td>\n",
       "      <td>ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.118</td>\n",
       "      <td>0.5574</td>\n",
       "      <td>177</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248647</th>\n",
       "      <td>Candy96023268</td>\n",
       "      <td>#手足互科 #standwithhongkong</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>15.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>116.0</td>\n",
       "      <td>2019-09-12 06:32:09</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>796.0</td>\n",
       "      <td>As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp;amp; stand w the students &amp;amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.305</td>\n",
       "      <td>-0.1265</td>\n",
       "      <td>184</td>\n",
       "      <td>26</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248648</th>\n",
       "      <td>Vincent03231883</td>\n",
       "      <td>hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足</td>\n",
       "      <td>n.a</td>\n",
       "      <td>248.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>362.0</td>\n",
       "      <td>2019-09-28 14:29:13</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>1162.0</td>\n",
       "      <td>“We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SecPompeo]</td>\n",
       "      <td>“we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.785</td>\n",
       "      <td>0.107</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>184</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248649</th>\n",
       "      <td>terryyipomni</td>\n",
       "      <td>hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom</td>\n",
       "      <td>hong kong</td>\n",
       "      <td>4653.0</td>\n",
       "      <td>2490.0</td>\n",
       "      <td>4627.0</td>\n",
       "      <td>2015-03-09 06:57:20</td>\n",
       "      <td>2019-11-19 17:19:36</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR</td>\n",
       "      <td>[soshk, standwithhongkong, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>huh excuse hong kong government lie \\n.\\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.794</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.0772</td>\n",
       "      <td>40</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248650</th>\n",
       "      <td>lajiaer1</td>\n",
       "      <td>言論自由是一切自由之母。川普总统支持者；支持爆料革命；</td>\n",
       "      <td>n.a</td>\n",
       "      <td>348.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>5042.0</td>\n",
       "      <td>2019-06-01 23:58:30</td>\n",
       "      <td>2019-11-19 17:19:35</td>\n",
       "      <td>10.0</td>\n",
       "      <td>@y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@y345a678, @YuriStoyanov]</td>\n",
       "      <td>can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please</td>\n",
       "      <td>0.378</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.8176</td>\n",
       "      <td>141</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>248651 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               username  \\\n",
       "0          five5demands   \n",
       "1        Dejavu53328974   \n",
       "2              tsksimon   \n",
       "3       JimmyWo67187904   \n",
       "4              ARCHI418   \n",
       "...                 ...   \n",
       "248646         zizihui1   \n",
       "248647    Candy96023268   \n",
       "248648  Vincent03231883   \n",
       "248649     terryyipomni   \n",
       "248650         lajiaer1   \n",
       "\n",
       "                                                                                                                                                   acctdesc  \\\n",
       "0                                                                                                       hker fight for freedom 當獨裁成為事實 革命就是義務 #followbackhk   \n",
       "1                                                             #followbackhongkong\\n#standwithhk\\n#hkhumanrightsanddemocracyact\\n #科勞手足\\n#hongkongprotesters   \n",
       "2                                                                                                                       #fightforfreedom #standwithhongkong   \n",
       "3                                                                                                                                                       n.a   \n",
       "4                                                                                                                                                       n.a   \n",
       "...                                                                                                                                                     ...   \n",
       "248646                                                                                               five demands, not one less 🇭🇰#followbackhongkong #科勞手足   \n",
       "248647                                                                                                                             #手足互科 #standwithhongkong   \n",
       "248648                                                                                        hongkonger never surrender\\n願榮光歸香港\\n#followbackhongkong #科勞手足   \n",
       "248649  hongkonger 🇭🇰 stand with hong kong! fight for freedom! 香港人加油!! #followbackhongkong #fivedemandsnotoneless #standwithhongkong #科勞手足 #fightforfreedom   \n",
       "248650                                                                                                                          言論自由是一切自由之母。川普总统支持者；支持爆料革命；   \n",
       "\n",
       "         location  following  followers  totaltweets        usercreatedts  \\\n",
       "0             n.a      437.0      260.0        466.0  2019-10-05 13:40:37   \n",
       "1       hong kong      581.0      360.0       7835.0  2019-08-27 15:47:34   \n",
       "2             n.a      825.0      456.0       6728.0  2014-10-05 15:57:07   \n",
       "3             n.a       22.0        1.0        656.0  2019-10-08 12:58:51   \n",
       "4             n.a       19.0       11.0        121.0  2014-06-19 16:19:36   \n",
       "...           ...        ...        ...          ...                  ...   \n",
       "248646  hong kong      505.0      326.0       1301.0  2019-09-25 17:05:21   \n",
       "248647  hong kong       15.0      155.0        116.0  2019-09-12 06:32:09   \n",
       "248648        n.a      248.0      170.0        362.0  2019-09-28 14:29:13   \n",
       "248649  hong kong     4653.0     2490.0       4627.0  2015-03-09 06:57:20   \n",
       "248650        n.a      348.0       45.0       5042.0  2019-06-01 23:58:30   \n",
       "\n",
       "             tweetcreatedts  retweetcount  \\\n",
       "0       2019-11-03 02:57:49         292.0   \n",
       "1       2019-11-03 02:57:49        1315.0   \n",
       "2       2019-11-03 02:57:49         105.0   \n",
       "3       2019-11-03 02:57:48         660.0   \n",
       "4       2019-11-03 02:57:46           0.0   \n",
       "...                     ...           ...   \n",
       "248646  2019-11-19 17:19:37          72.0   \n",
       "248647  2019-11-19 17:19:36         796.0   \n",
       "248648  2019-11-19 17:19:36        1162.0   \n",
       "248649  2019-11-19 17:19:36           9.0   \n",
       "248650  2019-11-19 17:19:35          10.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                 text  \\\n",
       "0                                                                                                                                                           Disgusting: Police in #HongKong are marking protestors with a pen, and writing a number on their hands just like the Nazis did in Germany. #China https://t.co/aAF6Jw7yXN   \n",
       "1                                                                                                                      #China-made teargas canister stuck in the road. Citizen tried to pull it out, but failed\\nCredit - @hkgetv \\n#PoliceState #StandWithHongKong #FreeHongKong #HongKong #HongKongProtests https://t.co/IkyHfFtrPH   \n",
       "2                    This is only tiny bits of #HKPoliceBrutality.\\nWe've been protesting for 5 months, yet the govn't still haven't established an Independent Commission of Inquiry into police conduct -- #HKPolice are allowed to torture and insult civilians w/o consequences. #HongKongProtests #SOSHK https://t.co/Ew6yVG5p8V   \n",
       "3                         A fireman complained to the #HongKongPolice for the unnecessary firing of Tear Gas that landed on the fire engine. Fireman was affected by the TG as well.\\n\\nA few police had lost their cool; confronted, ”we are dispersing crowd” and pushed the fireman away. \\n#hkpolicestate https://t.co/AAGFkuzmxL   \n",
       "4                                                                                                #police threw a #Teargas hand #grenade. It blasted on a citizen’s back and made a large area burn. It is not tear gas, it is read hand grenade and killing #hkger. #PoliceState #PoliceTerrorism #HKprotests https://t.co/hcDG6FNd7z   \n",
       "...                                                                                                                                                                                                                                                                                                                               ...   \n",
       "248646                             @marcorubio CCP is sending PLA to Hong Kong already, using exactly the same tactics as it invaded East Turkestan\\nCCP has no intention to keep \"one country, two systems\", otherwise HKers would have genuine autonomy 22 years ago, CCP is not trustworthy \\n#antichinazi https://t.co/l5FQVFBFC1   \n",
       "248647                       As violence continues to escalate in Hong Kong, the cause for freedom has never been greater. The world is watching as police shamefully attack students. I will continue to support &amp; stand w the students &amp; brave activists fighting to preserve freedom in #HongKong. https://t.co/mMncxgAehL   \n",
       "248648                  “We call on Chief Exec. Carrie Lam to promote accountability by supplementing the Independent Police Complains Council review with an independent investigation into the protest-related incidents.” - @SecPompeo on the political unrest in #HongKong. READ: https://t.co/RZMCdcgLQN https://t.co/i00X8iM8ky   \n",
       "248649                                                                                                                                                      Huh excuse me😃, Hong Kong government lie again and again \\n.\\n#SOSHK\\n#StandWithHongKong \\n#HongKongProtests \\n#HongKongPoliceTerrorists \\n#PolyU https://t.co/W5vRNMePOR   \n",
       "248650  @y345a678 @YuriStoyanov I can’t stop crying watching clips showing how our young protesters are being assaulted and heavily injured by #HKpolice. This is utterly unlawful for police to think they can use this kind of violence to anyone! #SOSHK #SOSPolyU may the world please #StandWithHongKong https://t.co/0OsOrGI55l   \n",
       "\n",
       "                                                                           hashtags  \\\n",
       "0                                                                        [hongkong]   \n",
       "1                                                              [china, policestate]   \n",
       "2                                                               [hkpolicebrutality]   \n",
       "3                                                                  [hongkongpolice]   \n",
       "4       [police, teargas, grenade, hkger, policestate, policeterrorism, hkprotests]   \n",
       "...                                                                             ...   \n",
       "248646                                                                           []   \n",
       "248647                                                                           []   \n",
       "248648                                                                           []   \n",
       "248649                                 [soshk, standwithhongkong, hongkongprotests]   \n",
       "248650                                                                           []   \n",
       "\n",
       "                   mentioned_users  \\\n",
       "0                               []   \n",
       "1                        [@hkgetv]   \n",
       "2                               []   \n",
       "3                               []   \n",
       "4                               []   \n",
       "...                            ...   \n",
       "248646               [@marcorubio]   \n",
       "248647                          []   \n",
       "248648                [@SecPompeo]   \n",
       "248649                          []   \n",
       "248650  [@y345a678, @YuriStoyanov]   \n",
       "\n",
       "                                                                                                                                                                                    cleaned_text  \\\n",
       "0                                                                                                                         disgust police mark protestors pen write number hand like nazi germany   \n",
       "1                                                                                                                                    teargas canister stick road citizen try pull failed\\ncredit   \n",
       "2                                          tiny bit \\nwe've protest month yet govn't still establish independent commission inquiry police conduct allow torture insult civilian w/o consequence   \n",
       "3                                     fireman complain unnecessary firing tear gas land fire engine fireman affect tg well.\\n\\na police lose cool confront ”we disperse crowd” push fireman away   \n",
       "4                                                                                                           threw hand blast citizen’s back make large area burn tear gas read hand grenade kill   \n",
       "...                                                                                                                                                                                          ...   \n",
       "248646        ccp send pla hong kong already use exactly tactic invade east turkestan\\nccp intention keep one country two system otherwise hkers would genuine autonomy year ago ccp trustworthy   \n",
       "248647  violence continue escalate hong kong cause freedom never great world watch police shamefully attack student continue support amp stand student amp brave activist fight preserve freedom   \n",
       "248648  “we call chief exec carrie lam promote accountability supplement independent police complains council review independent investigation protest-related incidents.” political unrest read   \n",
       "248649                                                                                                                                               huh excuse hong kong government lie \\n.\\n\\n   \n",
       "248650                                             can’t stop cry watch clip show young protester assault heavily injure utterly unlawful police think use kind violence anyone may world please   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "0       0.218  0.391  0.391    0.2732        70        11                4  \n",
       "1       0.256  0.543  0.202   -0.1779        58         8                2  \n",
       "2       0.339  0.590  0.070   -0.8074       148        20                1  \n",
       "3       0.364  0.498  0.137   -0.7430       152        23                1  \n",
       "4       0.203  0.519  0.277    0.1779        84        15                4  \n",
       "...       ...    ...    ...       ...       ...       ...              ...  \n",
       "248646  0.000  0.882  0.118    0.5574       177        27                5  \n",
       "248647  0.337  0.358  0.305   -0.1265       184        26                2  \n",
       "248648  0.107  0.785  0.107    0.0000       184        21                3  \n",
       "248649  0.000  0.794  0.206    0.0772        40         7                3  \n",
       "248650  0.378  0.458  0.163   -0.8176       141        22                1  \n",
       "\n",
       "[248651 rows x 20 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_df = data[(data['retweetcount'] > 20) & (data['totaltweets'] > 20) &  (data['followers'] > 20)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>location</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>tweetcreatedts</th>\n",
       "      <th>retweetcount</th>\n",
       "      <th>text</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>mentioned_users</th>\n",
       "      <th>cleaned_text</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "      <th>numchars</th>\n",
       "      <th>numwords</th>\n",
       "      <th>sentiment_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>107827</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>240.0</td>\n",
       "      <td>87014.0</td>\n",
       "      <td>10967.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-09 02:26:03</td>\n",
       "      <td>581.0</td>\n",
       "      <td>Picking up my passport to  travel to Hong Kong to pray and be a witness for freedom! I also invite you to join me in Chater Garden Nov. 15-18 at 9 AM and 7 PM! #StandWithHongKong #HongKong ⁦@Andychanhotin⁩ ⁦@FreedomHKG⁩ ⁦@Fight4HongKong⁩ ⁦@mschlapp⁩ https://t.co/F91UmnT1s2</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@Andychanhotin⁩, @FreedomHKG⁩, @Fight4HongKong⁩, @mschlapp⁩]</td>\n",
       "      <td>pick passport travel hong kong pray witness freedom also invite join chater garden nov pm</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.516</td>\n",
       "      <td>0.484</td>\n",
       "      <td>0.8519</td>\n",
       "      <td>89</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110216</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>240.0</td>\n",
       "      <td>87028.0</td>\n",
       "      <td>10969.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-09 02:47:15</td>\n",
       "      <td>45.0</td>\n",
       "      <td>Queen Elizabeth Hospital staff set up a temporary memorial last night with flowers, paper cranes and a Lennon wall in memory of HKUST student Alex Chow. Source: USP Photo: Kevin Cheng #StandWithHongKong #RIPChow https://t.co/Mlen9hh8om</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>queen elizabeth hospital staff set temporary memorial last night flower paper crane lennon wall memory hkust student alex chow source usp photo kevin cheng</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>155</td>\n",
       "      <td>24</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110665</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>240.0</td>\n",
       "      <td>87030.0</td>\n",
       "      <td>10969.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-09 02:43:18</td>\n",
       "      <td>688.0</td>\n",
       "      <td>Watch #HongKong protesters completely dismantle a road barricade in 22 seconds\\n😭😭😭😭😭 \\nhttps://t.co/2IMjHNBQ5T https://t.co/USVf4nQ8Dz</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>watch protester completely dismantle road barricade seconds\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>60</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122329</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>240.0</td>\n",
       "      <td>87304.0</td>\n",
       "      <td>11000.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-09 16:24:40</td>\n",
       "      <td>38.0</td>\n",
       "      <td>IPCC hires intl experts.\\nIntl experts complaints IPCC lack of capacity. #antiELAB #AntiMaskLaw #HongKongProtests https://t.co/WabhmtLaWw</td>\n",
       "      <td>[antielab, antimasklaw, hongkongprotests]</td>\n",
       "      <td>[]</td>\n",
       "      <td>ipcc hire intl experts.\\nintl expert complaint ipcc lack capacity</td>\n",
       "      <td>0.360</td>\n",
       "      <td>0.640</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.5423</td>\n",
       "      <td>64</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154610</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>241.0</td>\n",
       "      <td>89385.0</td>\n",
       "      <td>11136.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-11 13:50:21</td>\n",
       "      <td>35.0</td>\n",
       "      <td>@SolomonYue 9/6/1987 Yonsei University and 11/11/2019 Hong Kong Chinese University 😢🇨🇳🔪🇭🇰☠️ WHERE IS JUSTICE? #StandwithHK https://t.co/hdmqXIPiS2</td>\n",
       "      <td>[standwithhk]</td>\n",
       "      <td>[@SolomonYue]</td>\n",
       "      <td>yonsei university hong kong chinese university justice</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.638</td>\n",
       "      <td>0.362</td>\n",
       "      <td>0.5267</td>\n",
       "      <td>54</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173026</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>241.0</td>\n",
       "      <td>90521.0</td>\n",
       "      <td>11263.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-12 14:53:37</td>\n",
       "      <td>447.0</td>\n",
       "      <td>.@standnewshk reporter: \"Why can't the police inspector directly order frontline police to stop firing?\"\\n\\nDennis Ng, Pro-VC of #CUHK, \"They did already, a few times,\" shaking his head. Au Nok-hin adds that there's a disconnect between inspector &amp;amp; frontline police. \\n#HongKong #HK https://t.co/qQC9uM0nUc</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@standnewshk]</td>\n",
       "      <td>reporter can't police inspector directly order frontline police stop firing?\"\\n\\ndennis ng pro-vc already time shake head au nok-hin add there's disconnect inspector amp frontline police</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.860</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.4404</td>\n",
       "      <td>184</td>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>219520</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>97986.0</td>\n",
       "      <td>11965.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 13:49:16</td>\n",
       "      <td>218.0</td>\n",
       "      <td>According to the #BasicLaw and the underlying #1C2S principles, #PLA can *ONLY* be deployed under #HongKong Government's REQUEST for 2 reasons:\\n1. Maintain public order\\n2. Disaster relief\\n \\nIs PLA picking up bricks requested by HK Government? Is it about public order / disaster? https://t.co/Y8mFo5SR0I https://t.co/mFdQE9gZ2U</td>\n",
       "      <td>[basiclaw, 1c2s, pla, hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>accord underlying principle deploy government's request maintain public disaster relief\\n \\nis pla pick brick request hk government public order disaster</td>\n",
       "      <td>0.290</td>\n",
       "      <td>0.601</td>\n",
       "      <td>0.110</td>\n",
       "      <td>-0.7269</td>\n",
       "      <td>151</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220867</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>97991.0</td>\n",
       "      <td>11965.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 13:42:51</td>\n",
       "      <td>406.0</td>\n",
       "      <td>Pan-dems say Saturday's PLA deployment in #HongKong breached the law one way or another, while a group of protesters warn \"voluntary service\" today can easily be \"violent suppression\" tomorrow https://t.co/Cv9RowPkHS</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>pan-dems say saturday's pla deployment breach law one way another group protesters warn voluntary service today easily violent suppression tomorrow</td>\n",
       "      <td>0.281</td>\n",
       "      <td>0.625</td>\n",
       "      <td>0.094</td>\n",
       "      <td>-0.5859</td>\n",
       "      <td>147</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221360</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>98028.0</td>\n",
       "      <td>11969.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 14:15:09</td>\n",
       "      <td>835.0</td>\n",
       "      <td>Highly contentious move as the PLA directly intervenes to clear protester barricades. Under the Basic Law, the PLA “shall not interfere in the local affairs” of the SAR. Carrie Lam silent on this so far #HK #HongKongProtests #StandWithHongKong https://t.co/Ba0YNtxevt</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>highly contentious move pla directly intervene clear protester barricade basic law pla “shall interfere local affairs” sar carrie lam silent far</td>\n",
       "      <td>0.103</td>\n",
       "      <td>0.789</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.0276</td>\n",
       "      <td>144</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229015</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>98231.0</td>\n",
       "      <td>11980.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 15:22:13</td>\n",
       "      <td>81.0</td>\n",
       "      <td>Hong Kong journalism groups condemn alleged police firing of projectile at reporter as officer put on leave https://t.co/zXwqbgzpWu @krislc #HongKong #China #antielab #antiELABhk #HongKongProtests</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@krislc]</td>\n",
       "      <td>hong kong journalism group condemn allege police fire projectile reporter officer put leave</td>\n",
       "      <td>0.383</td>\n",
       "      <td>0.617</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.6369</td>\n",
       "      <td>91</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230560</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>98238.0</td>\n",
       "      <td>11980.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 15:16:41</td>\n",
       "      <td>333.0</td>\n",
       "      <td>Preparing for a public witness and demonstration on the streets of Hong Kong tonight for freedom and democracy. On with the revolution! #hongkong #StandwithHonKong @FreedomHKG @Stand_with_HK @HKWORLDCITY @hk_watch @Andychanhotin @joshuawongcf @SolomonYue https://t.co/daUx1Whvng</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@FreedomHKG, @Stand_with_HK, @HKWORLDCITY, @hk_watch, @Andychanhotin, @joshuawongcf, @SolomonYue]</td>\n",
       "      <td>prepare public witness demonstration street hong kong tonight freedom democracy revolution</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.616</td>\n",
       "      <td>0.384</td>\n",
       "      <td>0.6808</td>\n",
       "      <td>90</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231542</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>98280.0</td>\n",
       "      <td>11992.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 15:45:09</td>\n",
       "      <td>122.0</td>\n",
       "      <td>Thank you to a lovely HongKonger for joining our flashmob in Brisbane today with her instrument. \\n\\n#GloryToHongKong #DoYouHearThePeopleSing\\n#StandWithHongKong\\n#FightForFreedom #FightForFreedomStandWithHongKong https://t.co/sUTLtg1UV8</td>\n",
       "      <td>[glorytohongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>thank lovely hongkonger join flashmob brisbane today instrument \\n\\n \\n\\n</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.370</td>\n",
       "      <td>0.630</td>\n",
       "      <td>0.8176</td>\n",
       "      <td>69</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231868</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>98281.0</td>\n",
       "      <td>11994.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-16 15:44:04</td>\n",
       "      <td>1202.0</td>\n",
       "      <td>This is the first poll we have since \"Siege of #CUHK\". The result is similar to other polls in the last couple of months. Over 80% saying its the gov fault; and around 40% saying it's the protesters' fault. #StandwithHK https://t.co/UIo7is09KZ</td>\n",
       "      <td>[cuhk]</td>\n",
       "      <td>[]</td>\n",
       "      <td>first poll since siege result similar poll last couple month say gov fault around say protester fault</td>\n",
       "      <td>0.265</td>\n",
       "      <td>0.735</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.6597</td>\n",
       "      <td>101</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239796</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>105537.0</td>\n",
       "      <td>12350.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-19 16:17:46</td>\n",
       "      <td>478.0</td>\n",
       "      <td>I wonder if this photo by HK01 via Reuters will have a place in history books on #HongKong, or our world’s social movements.\\n\\nProtesters trapped inside the besieged #PolyU abseiling onto a highway, where motorbikes came to rescue, before a forced surrender #HongKongProtests # https://t.co/W8WzDnfCQN</td>\n",
       "      <td>[hongkong]</td>\n",
       "      <td>[]</td>\n",
       "      <td>wonder photo via reuters place history book world’s social movements.\\n\\nprotesters trap inside besiege abseil onto highway motorbike come rescue force surrender</td>\n",
       "      <td>0.158</td>\n",
       "      <td>0.717</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.0258</td>\n",
       "      <td>159</td>\n",
       "      <td>21</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240055</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>105537.0</td>\n",
       "      <td>12351.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-19 16:16:50</td>\n",
       "      <td>118.0</td>\n",
       "      <td>Leading a Facebook Live prayer vigil for over 1000 seiged students surrounded by the police at HK Polytechnic University on Saturday. That night,  there was NO violence. #AnsweredPrayer #HongKongPolice #HongKong @FreedomHKG @hk_watch @Andychanhotin @joshuawongcf #PrayForHongkong https://t.co/xWWb9SBMAY</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@FreedomHKG, @hk_watch, @Andychanhotin, @joshuawongcf]</td>\n",
       "      <td>lead facebook live prayer vigil seiged student surrounded police hk polytechnic university saturday night violence</td>\n",
       "      <td>0.227</td>\n",
       "      <td>0.773</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.6249</td>\n",
       "      <td>114</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241599</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>105585.0</td>\n",
       "      <td>12353.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-19 16:43:26</td>\n",
       "      <td>23.0</td>\n",
       "      <td>https://t.co/wdN4Y4hN5a \\nThis is what HK Police did to an armless &amp;amp;unconscious protester!!! Pls help spread scary truth!                                          \\n@SolomonYue\\n@WBYeats1865\\n@marcorubio                                 #PolyU \\n#HKPoliceTerrorism \\n#HKPoliceTerrorists</td>\n",
       "      <td>[]</td>\n",
       "      <td>[@SolomonYue, @WBYeats1865, @marcorubio]</td>\n",
       "      <td>\\nthis hk police armless amp;unconscious protester pls help spread scary truth \\n\\n\\n</td>\n",
       "      <td>0.194</td>\n",
       "      <td>0.424</td>\n",
       "      <td>0.382</td>\n",
       "      <td>0.2732</td>\n",
       "      <td>81</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246383</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>105626.0</td>\n",
       "      <td>12362.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-19 17:31:38</td>\n",
       "      <td>34.0</td>\n",
       "      <td>\"Never Surrender,\" \"Live Free or Die,\" \"Don't Tread On Me.\" #HongKong #PolyU https://t.co/0Ie9dPeTh7</td>\n",
       "      <td>[hongkong, polyu]</td>\n",
       "      <td>[]</td>\n",
       "      <td>never surrender live free die tread</td>\n",
       "      <td>0.623</td>\n",
       "      <td>0.377</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.7651</td>\n",
       "      <td>35</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246439</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>244.0</td>\n",
       "      <td>105626.0</td>\n",
       "      <td>12362.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>2019-11-19 17:31:22</td>\n",
       "      <td>31.0</td>\n",
       "      <td>HK Police (in addition to arresting doctors, first-aiders, journalists, and social workers) are now even arresting human rights observers. It appears that the power of arrest is now being abused as a tool of intimidation - to deter anyone from getting near the protests. #hongkong https://t.co/QSxiYgSETj</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>hk police addition arrest doctor first-aiders journalist social worker even arrest human right observer appear power arrest abuse tool intimidation deter anyone get near protest</td>\n",
       "      <td>0.460</td>\n",
       "      <td>0.540</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.9349</td>\n",
       "      <td>177</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          username  \\\n",
       "107827  SolomonYue   \n",
       "110216  SolomonYue   \n",
       "110665  SolomonYue   \n",
       "122329  SolomonYue   \n",
       "154610  SolomonYue   \n",
       "173026  SolomonYue   \n",
       "219520  SolomonYue   \n",
       "220867  SolomonYue   \n",
       "221360  SolomonYue   \n",
       "229015  SolomonYue   \n",
       "230560  SolomonYue   \n",
       "231542  SolomonYue   \n",
       "231868  SolomonYue   \n",
       "239796  SolomonYue   \n",
       "240055  SolomonYue   \n",
       "241599  SolomonYue   \n",
       "246383  SolomonYue   \n",
       "246439  SolomonYue   \n",
       "\n",
       "                                                                                                                                                               acctdesc  \\\n",
       "107827  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "110216  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "110665  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "122329  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "154610  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "173026  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "219520  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "220867  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "221360  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "229015  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "230560  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "231542  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "231868  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "239796  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "240055  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "241599  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "246383  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "246439  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "\n",
       "                  location  following  followers  totaltweets  \\\n",
       "107827  salem, oregon, usa      240.0    87014.0      10967.0   \n",
       "110216  salem, oregon, usa      240.0    87028.0      10969.0   \n",
       "110665  salem, oregon, usa      240.0    87030.0      10969.0   \n",
       "122329  salem, oregon, usa      240.0    87304.0      11000.0   \n",
       "154610  salem, oregon, usa      241.0    89385.0      11136.0   \n",
       "173026  salem, oregon, usa      241.0    90521.0      11263.0   \n",
       "219520  salem, oregon, usa      244.0    97986.0      11965.0   \n",
       "220867  salem, oregon, usa      244.0    97991.0      11965.0   \n",
       "221360  salem, oregon, usa      244.0    98028.0      11969.0   \n",
       "229015  salem, oregon, usa      244.0    98231.0      11980.0   \n",
       "230560  salem, oregon, usa      244.0    98238.0      11980.0   \n",
       "231542  salem, oregon, usa      244.0    98280.0      11992.0   \n",
       "231868  salem, oregon, usa      244.0    98281.0      11994.0   \n",
       "239796  salem, oregon, usa      244.0   105537.0      12350.0   \n",
       "240055  salem, oregon, usa      244.0   105537.0      12351.0   \n",
       "241599  salem, oregon, usa      244.0   105585.0      12353.0   \n",
       "246383  salem, oregon, usa      244.0   105626.0      12362.0   \n",
       "246439  salem, oregon, usa      244.0   105626.0      12362.0   \n",
       "\n",
       "              usercreatedts       tweetcreatedts  retweetcount  \\\n",
       "107827  2013-04-09 15:38:48  2019-11-09 02:26:03         581.0   \n",
       "110216  2013-04-09 15:38:48  2019-11-09 02:47:15          45.0   \n",
       "110665  2013-04-09 15:38:48  2019-11-09 02:43:18         688.0   \n",
       "122329  2013-04-09 15:38:48  2019-11-09 16:24:40          38.0   \n",
       "154610  2013-04-09 15:38:48  2019-11-11 13:50:21          35.0   \n",
       "173026  2013-04-09 15:38:48  2019-11-12 14:53:37         447.0   \n",
       "219520  2013-04-09 15:38:48  2019-11-16 13:49:16         218.0   \n",
       "220867  2013-04-09 15:38:48  2019-11-16 13:42:51         406.0   \n",
       "221360  2013-04-09 15:38:48  2019-11-16 14:15:09         835.0   \n",
       "229015  2013-04-09 15:38:48  2019-11-16 15:22:13          81.0   \n",
       "230560  2013-04-09 15:38:48  2019-11-16 15:16:41         333.0   \n",
       "231542  2013-04-09 15:38:48  2019-11-16 15:45:09         122.0   \n",
       "231868  2013-04-09 15:38:48  2019-11-16 15:44:04        1202.0   \n",
       "239796  2013-04-09 15:38:48  2019-11-19 16:17:46         478.0   \n",
       "240055  2013-04-09 15:38:48  2019-11-19 16:16:50         118.0   \n",
       "241599  2013-04-09 15:38:48  2019-11-19 16:43:26          23.0   \n",
       "246383  2013-04-09 15:38:48  2019-11-19 17:31:38          34.0   \n",
       "246439  2013-04-09 15:38:48  2019-11-19 17:31:22          31.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                               text  \\\n",
       "107827                                                            Picking up my passport to  travel to Hong Kong to pray and be a witness for freedom! I also invite you to join me in Chater Garden Nov. 15-18 at 9 AM and 7 PM! #StandWithHongKong #HongKong ⁦@Andychanhotin⁩ ⁦@FreedomHKG⁩ ⁦@Fight4HongKong⁩ ⁦@mschlapp⁩ https://t.co/F91UmnT1s2   \n",
       "110216                                                                                                  Queen Elizabeth Hospital staff set up a temporary memorial last night with flowers, paper cranes and a Lennon wall in memory of HKUST student Alex Chow. Source: USP Photo: Kevin Cheng #StandWithHongKong #RIPChow https://t.co/Mlen9hh8om   \n",
       "110665                                                                                                                                                                                                      Watch #HongKong protesters completely dismantle a road barricade in 22 seconds\\n😭😭😭😭😭 \\nhttps://t.co/2IMjHNBQ5T https://t.co/USVf4nQ8Dz   \n",
       "122329                                                                                                                                                                                                    IPCC hires intl experts.\\nIntl experts complaints IPCC lack of capacity. #antiELAB #AntiMaskLaw #HongKongProtests https://t.co/WabhmtLaWw   \n",
       "154610                                                                                                                                                                                           @SolomonYue 9/6/1987 Yonsei University and 11/11/2019 Hong Kong Chinese University 😢🇨🇳🔪🇭🇰☠️ WHERE IS JUSTICE? #StandwithHK https://t.co/hdmqXIPiS2   \n",
       "173026                       .@standnewshk reporter: \"Why can't the police inspector directly order frontline police to stop firing?\"\\n\\nDennis Ng, Pro-VC of #CUHK, \"They did already, a few times,\" shaking his head. Au Nok-hin adds that there's a disconnect between inspector &amp; frontline police. \\n#HongKong #HK https://t.co/qQC9uM0nUc   \n",
       "219520  According to the #BasicLaw and the underlying #1C2S principles, #PLA can *ONLY* be deployed under #HongKong Government's REQUEST for 2 reasons:\\n1. Maintain public order\\n2. Disaster relief\\n \\nIs PLA picking up bricks requested by HK Government? Is it about public order / disaster? https://t.co/Y8mFo5SR0I https://t.co/mFdQE9gZ2U   \n",
       "220867                                                                                                                     Pan-dems say Saturday's PLA deployment in #HongKong breached the law one way or another, while a group of protesters warn \"voluntary service\" today can easily be \"violent suppression\" tomorrow https://t.co/Cv9RowPkHS   \n",
       "221360                                                                  Highly contentious move as the PLA directly intervenes to clear protester barricades. Under the Basic Law, the PLA “shall not interfere in the local affairs” of the SAR. Carrie Lam silent on this so far #HK #HongKongProtests #StandWithHongKong https://t.co/Ba0YNtxevt   \n",
       "229015                                                                                                                                         Hong Kong journalism groups condemn alleged police firing of projectile at reporter as officer put on leave https://t.co/zXwqbgzpWu @krislc #HongKong #China #antielab #antiELABhk #HongKongProtests   \n",
       "230560                                                       Preparing for a public witness and demonstration on the streets of Hong Kong tonight for freedom and democracy. On with the revolution! #hongkong #StandwithHonKong @FreedomHKG @Stand_with_HK @HKWORLDCITY @hk_watch @Andychanhotin @joshuawongcf @SolomonYue https://t.co/daUx1Whvng   \n",
       "231542                                                                                                Thank you to a lovely HongKonger for joining our flashmob in Brisbane today with her instrument. \\n\\n#GloryToHongKong #DoYouHearThePeopleSing\\n#StandWithHongKong\\n#FightForFreedom #FightForFreedomStandWithHongKong https://t.co/sUTLtg1UV8   \n",
       "231868                                                                                          This is the first poll we have since \"Siege of #CUHK\". The result is similar to other polls in the last couple of months. Over 80% saying its the gov fault; and around 40% saying it's the protesters' fault. #StandwithHK https://t.co/UIo7is09KZ   \n",
       "239796                               I wonder if this photo by HK01 via Reuters will have a place in history books on #HongKong, or our world’s social movements.\\n\\nProtesters trapped inside the besieged #PolyU abseiling onto a highway, where motorbikes came to rescue, before a forced surrender #HongKongProtests # https://t.co/W8WzDnfCQN   \n",
       "240055                              Leading a Facebook Live prayer vigil for over 1000 seiged students surrounded by the police at HK Polytechnic University on Saturday. That night,  there was NO violence. #AnsweredPrayer #HongKongPolice #HongKong @FreedomHKG @hk_watch @Andychanhotin @joshuawongcf #PrayForHongkong https://t.co/xWWb9SBMAY   \n",
       "241599                                            https://t.co/wdN4Y4hN5a \\nThis is what HK Police did to an armless &amp;unconscious protester!!! Pls help spread scary truth!                                          \\n@SolomonYue\\n@WBYeats1865\\n@marcorubio                                 #PolyU \\n#HKPoliceTerrorism \\n#HKPoliceTerrorists   \n",
       "246383                                                                                                                                                                                                                                         \"Never Surrender,\" \"Live Free or Die,\" \"Don't Tread On Me.\" #HongKong #PolyU https://t.co/0Ie9dPeTh7   \n",
       "246439                             HK Police (in addition to arresting doctors, first-aiders, journalists, and social workers) are now even arresting human rights observers. It appears that the power of arrest is now being abused as a tool of intimidation - to deter anyone from getting near the protests. #hongkong https://t.co/QSxiYgSETj   \n",
       "\n",
       "                                         hashtags  \\\n",
       "107827                                         []   \n",
       "110216                                         []   \n",
       "110665                                 [hongkong]   \n",
       "122329  [antielab, antimasklaw, hongkongprotests]   \n",
       "154610                              [standwithhk]   \n",
       "173026                                         []   \n",
       "219520            [basiclaw, 1c2s, pla, hongkong]   \n",
       "220867                                 [hongkong]   \n",
       "221360                                         []   \n",
       "229015                                         []   \n",
       "230560                                         []   \n",
       "231542                          [glorytohongkong]   \n",
       "231868                                     [cuhk]   \n",
       "239796                                 [hongkong]   \n",
       "240055                                         []   \n",
       "241599                                         []   \n",
       "246383                          [hongkong, polyu]   \n",
       "246439                                         []   \n",
       "\n",
       "                                                                                           mentioned_users  \\\n",
       "107827                                       [@Andychanhotin⁩, @FreedomHKG⁩, @Fight4HongKong⁩, @mschlapp⁩]   \n",
       "110216                                                                                                  []   \n",
       "110665                                                                                                  []   \n",
       "122329                                                                                                  []   \n",
       "154610                                                                                       [@SolomonYue]   \n",
       "173026                                                                                      [@standnewshk]   \n",
       "219520                                                                                                  []   \n",
       "220867                                                                                                  []   \n",
       "221360                                                                                                  []   \n",
       "229015                                                                                           [@krislc]   \n",
       "230560  [@FreedomHKG, @Stand_with_HK, @HKWORLDCITY, @hk_watch, @Andychanhotin, @joshuawongcf, @SolomonYue]   \n",
       "231542                                                                                                  []   \n",
       "231868                                                                                                  []   \n",
       "239796                                                                                                  []   \n",
       "240055                                             [@FreedomHKG, @hk_watch, @Andychanhotin, @joshuawongcf]   \n",
       "241599                                                            [@SolomonYue, @WBYeats1865, @marcorubio]   \n",
       "246383                                                                                                  []   \n",
       "246439                                                                                                  []   \n",
       "\n",
       "                                                                                                                                                                                      cleaned_text  \\\n",
       "107827                                                                                                   pick passport travel hong kong pray witness freedom also invite join chater garden nov pm   \n",
       "110216                                 queen elizabeth hospital staff set temporary memorial last night flower paper crane lennon wall memory hkust student alex chow source usp photo kevin cheng   \n",
       "110665                                                                                                                               watch protester completely dismantle road barricade seconds\\n   \n",
       "122329                                                                                                                           ipcc hire intl experts.\\nintl expert complaint ipcc lack capacity   \n",
       "154610                                                                                                                                      yonsei university hong kong chinese university justice   \n",
       "173026  reporter can't police inspector directly order frontline police stop firing?\"\\n\\ndennis ng pro-vc already time shake head au nok-hin add there's disconnect inspector amp frontline police   \n",
       "219520                                   accord underlying principle deploy government's request maintain public disaster relief\\n \\nis pla pick brick request hk government public order disaster   \n",
       "220867                                         pan-dems say saturday's pla deployment breach law one way another group protesters warn voluntary service today easily violent suppression tomorrow   \n",
       "221360                                            highly contentious move pla directly intervene clear protester barricade basic law pla “shall interfere local affairs” sar carrie lam silent far   \n",
       "229015                                                                                                 hong kong journalism group condemn allege police fire projectile reporter officer put leave   \n",
       "230560                                                                                                  prepare public witness demonstration street hong kong tonight freedom democracy revolution   \n",
       "231542                                                                                                                   thank lovely hongkonger join flashmob brisbane today instrument \\n\\n \\n\\n   \n",
       "231868                                                                                       first poll since siege result similar poll last couple month say gov fault around say protester fault   \n",
       "239796                           wonder photo via reuters place history book world’s social movements.\\n\\nprotesters trap inside besiege abseil onto highway motorbike come rescue force surrender   \n",
       "240055                                                                          lead facebook live prayer vigil seiged student surrounded police hk polytechnic university saturday night violence   \n",
       "241599                                                                                                       \\nthis hk police armless amp;unconscious protester pls help spread scary truth \\n\\n\\n   \n",
       "246383                                                                                                                                                         never surrender live free die tread   \n",
       "246439           hk police addition arrest doctor first-aiders journalist social worker even arrest human right observer appear power arrest abuse tool intimidation deter anyone get near protest   \n",
       "\n",
       "          neg    neu    pos  compound  numchars  numwords  sentiment_class  \n",
       "107827  0.000  0.516  0.484    0.8519        89        15                5  \n",
       "110216  0.000  1.000  0.000    0.0000       155        24                3  \n",
       "110665  0.000  1.000  0.000    0.0000        60         7                3  \n",
       "122329  0.360  0.640  0.000   -0.5423        64         9                2  \n",
       "154610  0.000  0.638  0.362    0.5267        54         7                4  \n",
       "173026  0.140  0.860  0.000   -0.4404       184        25                2  \n",
       "219520  0.290  0.601  0.110   -0.7269       151        20                1  \n",
       "220867  0.281  0.625  0.094   -0.5859       147        20                1  \n",
       "221360  0.103  0.789  0.108    0.0276       144        21                3  \n",
       "229015  0.383  0.617  0.000   -0.6369        91        13                1  \n",
       "230560  0.000  0.616  0.384    0.6808        90        11                5  \n",
       "231542  0.000  0.370  0.630    0.8176        69        10                5  \n",
       "231868  0.265  0.735  0.000   -0.6597       101        17                1  \n",
       "239796  0.158  0.717  0.125    0.0258       159        21                3  \n",
       "240055  0.227  0.773  0.000   -0.6249       114        15                1  \n",
       "241599  0.194  0.424  0.382    0.2732        81        12                4  \n",
       "246383  0.623  0.377  0.000   -0.7651        35         6                1  \n",
       "246439  0.460  0.540  0.000   -0.9349       177        25                1  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now, let's find the rows where user_name is 'SolomonYue'\n",
    "original_df[original_df['username'] == 'SolomonYue']\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(123104, 20)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_users(df):\n",
    "    # df: dataframe to look at\n",
    "    # returns a list of usernames\n",
    "    \n",
    "    # Create empty list\n",
    "    list_users = []\n",
    "    \n",
    "    for i in range(0, len(df)):\n",
    "        users_ith_text = re.findall('@[^\\s]+', df.iloc[i,:]['text'])\n",
    "        # returns a list\n",
    "        # append to list_users by going through a for-loop:\n",
    "        for j in range(0, len(users_ith_text)):\n",
    "            list_users.append(users_ith_text[j])\n",
    "    \n",
    "    return list_users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mentioned_users</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@hkgetv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>@SenateMajLdr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>@SolomonYue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>@SenRickScott</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@seariousforhkg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mentioned_users\n",
       "0          @hkgetv\n",
       "1    @SenateMajLdr\n",
       "2      @SolomonYue\n",
       "3    @SenRickScott\n",
       "4  @seariousforhkg"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply on dataframe data['text']\n",
    "list_users = find_users(data)\n",
    "\n",
    "mentioned_users_df = pd.DataFrame({\n",
    "    'mentioned_users': list_users\n",
    "})\n",
    "\n",
    "mentioned_users_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the DataFrame to a CSV file\n",
    "original_df.to_csv('original_df.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_df = pd.read_csv('/Users/shiruizhou/Desktop/network analysis/hk_2019_social network/original_df.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(123104, 20)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'original_df' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[1], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m unique_username_count \u001b[39m=\u001b[39m original_df[\u001b[39m'\u001b[39m\u001b[39musername\u001b[39m\u001b[39m'\u001b[39m]\u001b[39m.\u001b[39mnunique()\n\u001b[1;32m      2\u001b[0m \u001b[39mprint\u001b[39m(unique_username_count)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'original_df' is not defined"
     ]
    }
   ],
   "source": [
    "unique_username_count = original_df['username'].nunique()\n",
    "print(unique_username_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Assuming 'original_df' is your original DataFrame containing tweet and user information\n",
    "\n",
    "# Create a dictionary to store user information and their associated tweets\n",
    "user_data = {}\n",
    "\n",
    "# Iterate through the original DataFrame\n",
    "for index, row in original_df.iterrows():\n",
    "    username = row['username']\n",
    "    user_info = {\n",
    "        'following': row['following'],\n",
    "        'followers': row['followers'],\n",
    "        'totaltweets': row['totaltweets'],\n",
    "        'usercreatedts': row['usercreatedts'],\n",
    "        'location': row['location'],\n",
    "        'acctdesc': row['acctdesc']\n",
    "    }\n",
    "    tweet_info = {\n",
    "        'tweetcreatedts': row['tweetcreatedts'],\n",
    "        'retweetcount': row['retweetcount'],\n",
    "        'hashtags': row['hashtags'],\n",
    "        'mentioned_users': row['mentioned_users'],\n",
    "        'cleaned_text': row['cleaned_text'],\n",
    "        'compound': row['compound'],    # Added 'compound' field\n",
    "        'sentiment_class': row['sentiment_class']    # Added 'sentiment_class' field\n",
    "    }\n",
    "    # If the user already exists in the dictionary, add the tweet information to the user's list of tweets\n",
    "    if username in user_data:\n",
    "        user_data[username]['tweets'].append(tweet_info)\n",
    "    # If the user does not exist, create a new entry with the user and tweet information\n",
    "    else:\n",
    "        user_data[username] = {'user_info': user_info, 'tweets': [tweet_info]}\n",
    "\n",
    "# Create a list to store rows for the transformed DataFrame\n",
    "transformed_rows = []\n",
    "\n",
    "# Iterate through the user_data dictionary to create rows for the transformed DataFrame\n",
    "for username, user_info in user_data.items():\n",
    "    user_row = {'username': username}\n",
    "    # Add user information to the user_row\n",
    "    user_row.update(user_info['user_info'])\n",
    "    # Add tweet information as separate columns for each tweet associated with the user\n",
    "    for i, tweet in enumerate(user_info['tweets'], start=1):\n",
    "        user_row[f'tweetcreatedts_{i}'] = tweet['tweetcreatedts']\n",
    "        user_row[f'retweetcount_{i}'] = tweet['retweetcount']\n",
    "        user_row[f'hashtags_{i}'] = tweet['hashtags']\n",
    "        user_row[f'mentioned_users_{i}'] = tweet['mentioned_users']\n",
    "        user_row[f'cleaned_text_{i}'] = tweet['cleaned_text']\n",
    "        user_row[f'compound_{i}'] = tweet['compound']    # Added 'compound' field\n",
    "        user_row[f'sentiment_class_{i}'] = tweet['sentiment_class']    # Added 'sentiment_class' field\n",
    "    transformed_rows.append(user_row)\n",
    "\n",
    "# Create the transformed DataFrame\n",
    "transformed_df = pd.DataFrame(transformed_rows)\n",
    "\n",
    "# Display the transformed DataFrame\n",
    "print(transformed_df)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>location</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>tweetcreatedts_1</th>\n",
       "      <th>retweetcount_1</th>\n",
       "      <th>hashtags_1</th>\n",
       "      <th>...</th>\n",
       "      <th>cleaned_text_331</th>\n",
       "      <th>compound_331</th>\n",
       "      <th>sentiment_class_331</th>\n",
       "      <th>tweetcreatedts_332</th>\n",
       "      <th>retweetcount_332</th>\n",
       "      <th>hashtags_332</th>\n",
       "      <th>mentioned_users_332</th>\n",
       "      <th>cleaned_text_332</th>\n",
       "      <th>compound_332</th>\n",
       "      <th>sentiment_class_332</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9813</th>\n",
       "      <td>belleng324</td>\n",
       "      <td>2562.0</td>\n",
       "      <td>730.0</td>\n",
       "      <td>37698.0</td>\n",
       "      <td>2019-08-22 09:11:37</td>\n",
       "      <td>n.a</td>\n",
       "      <td>n.a</td>\n",
       "      <td>2019-11-08 16:37:35</td>\n",
       "      <td>100.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>break car windows baton attempt arrest driver ...</td>\n",
       "      <td>-0.6996</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2019-11-12 14:31:16</td>\n",
       "      <td>187.0</td>\n",
       "      <td>['hongkongprotests']</td>\n",
       "      <td>['@annielab_jmsc', '@business:']</td>\n",
       "      <td>spot fake news \\n\\nuniversity hong kong studen...</td>\n",
       "      <td>-0.4767</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 2331 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        username  following  followers  totaltweets        usercreatedts  \\\n",
       "9813  belleng324     2562.0      730.0      37698.0  2019-08-22 09:11:37   \n",
       "\n",
       "     location acctdesc     tweetcreatedts_1  retweetcount_1 hashtags_1  ...  \\\n",
       "9813      n.a      n.a  2019-11-08 16:37:35           100.0         []  ...   \n",
       "\n",
       "                                       cleaned_text_331 compound_331  \\\n",
       "9813  break car windows baton attempt arrest driver ...      -0.6996   \n",
       "\n",
       "      sentiment_class_331   tweetcreatedts_332 retweetcount_332  \\\n",
       "9813                  1.0  2019-11-12 14:31:16            187.0   \n",
       "\n",
       "              hashtags_332               mentioned_users_332  \\\n",
       "9813  ['hongkongprotests']  ['@annielab_jmsc', '@business:']   \n",
       "\n",
       "                                       cleaned_text_332 compound_332  \\\n",
       "9813  spot fake news \\n\\nuniversity hong kong studen...      -0.4767   \n",
       "\n",
       "      sentiment_class_332  \n",
       "9813                  2.0  \n",
       "\n",
       "[1 rows x 2331 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed_df[transformed_df['cleaned_text_332'].notnull()]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>following</th>\n",
       "      <th>followers</th>\n",
       "      <th>totaltweets</th>\n",
       "      <th>usercreatedts</th>\n",
       "      <th>location</th>\n",
       "      <th>acctdesc</th>\n",
       "      <th>tweetcreatedts_1</th>\n",
       "      <th>retweetcount_1</th>\n",
       "      <th>hashtags_1</th>\n",
       "      <th>...</th>\n",
       "      <th>cleaned_text_331</th>\n",
       "      <th>compound_331</th>\n",
       "      <th>sentiment_class_331</th>\n",
       "      <th>tweetcreatedts_332</th>\n",
       "      <th>retweetcount_332</th>\n",
       "      <th>hashtags_332</th>\n",
       "      <th>mentioned_users_332</th>\n",
       "      <th>cleaned_text_332</th>\n",
       "      <th>compound_332</th>\n",
       "      <th>sentiment_class_332</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11304</th>\n",
       "      <td>SolomonYue</td>\n",
       "      <td>240.0</td>\n",
       "      <td>87014.0</td>\n",
       "      <td>10967.0</td>\n",
       "      <td>2013-04-09 15:38:48</td>\n",
       "      <td>salem, oregon, usa</td>\n",
       "      <td>vice chairman &amp; ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus &amp; conservative steering committee</td>\n",
       "      <td>2019-11-09 02:26:03</td>\n",
       "      <td>581.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 2331 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         username  following  followers  totaltweets        usercreatedts  \\\n",
       "11304  SolomonYue      240.0    87014.0      10967.0  2013-04-09 15:38:48   \n",
       "\n",
       "                 location  \\\n",
       "11304  salem, oregon, usa   \n",
       "\n",
       "                                                                                                                                                              acctdesc  \\\n",
       "11304  vice chairman & ceo at republicans overseas, rnc member since 2000, co-founder of rnc republican national conservative caucus & conservative steering committee   \n",
       "\n",
       "          tweetcreatedts_1  retweetcount_1 hashtags_1  ... cleaned_text_331  \\\n",
       "11304  2019-11-09 02:26:03           581.0         []  ...              NaN   \n",
       "\n",
       "      compound_331  sentiment_class_331  tweetcreatedts_332 retweetcount_332  \\\n",
       "11304          NaN                  NaN                 NaN              NaN   \n",
       "\n",
       "       hashtags_332 mentioned_users_332 cleaned_text_332 compound_332  \\\n",
       "11304           NaN                 NaN              NaN          NaN   \n",
       "\n",
       "       sentiment_class_332  \n",
       "11304                  NaN  \n",
       "\n",
       "[1 rows x 2331 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed_df[transformed_df['username'] == 'SolomonYue']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed_df.to_csv('transformed_df.csv', index=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hashtag and time windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'data' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[15], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m data\n",
      "\u001b[0;31mNameError\u001b[0m: name 'data' is not defined"
     ]
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'data' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[14], line 7\u001b[0m\n\u001b[1;32m      4\u001b[0m date \u001b[39m=\u001b[39m []\n\u001b[1;32m      6\u001b[0m \u001b[39m# First find out which 'tweetcreatedts' is not a string or in other weird formats\u001b[39;00m\n\u001b[0;32m----> 7\u001b[0m \u001b[39mfor\u001b[39;00m i \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(\u001b[39m0\u001b[39m, \u001b[39mlen\u001b[39m(data)):\n\u001b[1;32m      8\u001b[0m     ith_date_str \u001b[39m=\u001b[39m data\u001b[39m.\u001b[39miloc[i,:][\u001b[39m'\u001b[39m\u001b[39mtweetcreatedts\u001b[39m\u001b[39m'\u001b[39m]\n\u001b[1;32m      9\u001b[0m     ith_match \u001b[39m=\u001b[39m re\u001b[39m.\u001b[39msearch(\u001b[39mr\u001b[39m\u001b[39m'\u001b[39m\u001b[39m\\\u001b[39m\u001b[39md\u001b[39m\u001b[39m{4}\u001b[39;00m\u001b[39m-\u001b[39m\u001b[39m\\\u001b[39m\u001b[39md\u001b[39m\u001b[39m{2}\u001b[39;00m\u001b[39m-\u001b[39m\u001b[39m\\\u001b[39m\u001b[39md\u001b[39m\u001b[39m{2}\u001b[39;00m\u001b[39m'\u001b[39m, ith_date_str)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'data' is not defined"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "ind_to_drop = []\n",
    "date = []\n",
    "\n",
    "# First find out which 'tweetcreatedts' is not a string or in other weird formats\n",
    "for i in range(0, len(data)):\n",
    "    ith_date_str = data.iloc[i,:]['tweetcreatedts']\n",
    "    ith_match = re.search(r'\\d{4}-\\d{2}-\\d{2}', ith_date_str)\n",
    "    if ith_match == None:\n",
    "        ind_to_drop.append(i)\n",
    "    else:\n",
    "        continue"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "anly645",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}