notebooks/1.4_create_test_sets_cch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UNCOVERING STATE-BACKED TROLLS ON TWITTER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NOTEOOK 1.4:\n",
    "In this project, I'll be testing the model(s) against several unseen test sets to see whether the model(s) can be generalised, ie, detect state-backed tweets from different countries. I'll create the following test sets for this purpose:\n",
    "\n",
    "1. Unseen test set 1: Russian/IRA state-backed tweets + American real users \n",
    "2. Unseen test set 2: Iranian state-backed tweets + American real users\n",
    "3. Unseen test set 3: Venezuelian state-backed tweets + American real users\n",
    "\n",
    "You can download the state-backed datasets [here](https://about.twitter.com/en_us/values/elections-integrity.html#data). I'm not including the raw datasets in this repo as they run into several Gbs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langdetect import detect\n",
    "from textblob import TextBlob\n",
    "\n",
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. CREATING RUSSIAN STATE-BACKED/AMERICAN REAL USERS UNSEEN TEST SET\n",
    "This is part of the Russian/Internet Research Agency corpus of tweets released by Twitter in Oct 2018. It comprises about 9 million tweets from 3613 accounts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calling up the full unseen test sets created in the earlier notebooks\n",
    "fake_test = pd.read_csv('../data/fake_test.csv')\n",
    "real_test = pd.read_csv('../data/real_test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1000, 9), (1000, 9))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fake_test.shape, real_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is same function I used in notebook 1.3 to clean up the tweet text\n",
    "def clean_tweet(text):\n",
    "    text = text.lower()\n",
    "    text = re.sub(r\"http\\S+\", \"\", text)\n",
    "    text = re.sub(\"\\W\", \" \", text)\n",
    "    text = re.sub(\"\\s+\", \" \", text)\n",
    "    text = text.strip(\" \")\n",
    "    text = text.strip(\"\\n\")\n",
    "    text = re.sub(\"[^\\w\\s]\", \"\", text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_test['clean_tweet_text'] = real_test['tweet_text'].map(lambda tweet: clean_tweet(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "fake_test['clean_tweet_text'] = fake_test['tweet_text'].map(lambda tweet: clean_tweet(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating a new col to classify the real Vs state-backed tweets\n",
    "fake_test['bot_or_not'] = 1\n",
    "real_test['bot_or_not'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "russian_unseen = pd.concat((fake_test, real_test), axis=0, sort=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: This CSV file is included in the repo\n",
    "#russian_set = russian_unseen.to_csv('../data/russian_unseen.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NOTE:\n",
    "I won't further feature engineer this particular set as I'll only need the \"bot or not\" and \"clean tweet text\" cols for testing the model. I'll also outputted the unseen Russian test set out as a separate CSV file that I'll use during the testing of the model in the 3.0 series of the notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. CREATING IRANIAN STATE-BACKED/AMERICAN REAL USERS UNSEEN TEST SET\n",
    "This is part of the Iranian corpus of tweets released by Twitter in October 2018, comprising 1.1 million tweets from 770 accounts. I won't do extensive work on this dataset, beyond getting them into the same shape as the other unseen tests and most importantly, filtering out the non-English tweets and retweets.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 PRE-PROCESSING THE IRANIAN STATE-BACKED TWEETS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: This CSV file is not included in the repo due to its file size. Download it with the link above\n",
    "iran = pd.read_csv('../data/iranian_tweets.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran = iran.drop(\n",
    "    columns=[\n",
    "        \"userid\",\n",
    "        \"user_display_name\",\n",
    "        \"user_profile_url\",\n",
    "        \"tweet_client_name\",\n",
    "        \"in_reply_to_tweetid\",\n",
    "        \"in_reply_to_userid\",\n",
    "        \"quoted_tweet_tweetid\",\n",
    "        \"is_retweet\",\n",
    "        \"retweet_userid\",\n",
    "        \"retweet_tweetid\",\n",
    "        \"latitude\",\n",
    "        \"longitude\",\n",
    "        \"quote_count\",\n",
    "        \"reply_count\",\n",
    "        \"like_count\",\n",
    "        \"retweet_count\",\n",
    "        \"urls\",\n",
    "        \"user_mentions\",\n",
    "        \"poll_choices\",\n",
    "        \"hashtags\",\n",
    "        \"account_language\",\n",
    "        \"tweet_language\"\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The goal here is to get 1,000 English tweets from this set, but we'll have to sample a larger\n",
    "#number due to the likelihood of large number of non_English tweets\n",
    "iran_tweets = iran.sample(n=50000, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_language_langdetect(text):\n",
    "    try:\n",
    "        return detect(text)\n",
    "    except:\n",
    "        return 'unk'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_language_textblob(text):\n",
    "    try:\n",
    "        return TextBlob(text).detect_language\n",
    "    except:\n",
    "        return 'unk'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets['lang_textblob'] = iran_tweets['tweet_text'].apply(detect_language_textblob)\n",
    "iran_tweets['lang_textblob_loc'] = iran_tweets['user_reported_location'].apply(detect_language_textblob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets['langdetect'] = iran_tweets['tweet_text'].apply(detect_language_langdetect)\n",
    "iran_tweets['langdetect_loc'] = iran_tweets['user_reported_location'].apply(detect_language_langdetect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets = iran_tweets[(iran_tweets['langdetect'] == 'en') & (iran_tweets['langdetect_loc'] == 'en')].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "en    3134\n",
       "Name: langdetect, dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iran_tweets['langdetect'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets = iran_tweets[~iran_tweets['tweet_text'].str.startswith(\"RT @\")].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets = iran_tweets.drop(columns=['langdetect', 'langdetect_loc', 'lang_textblob', 'lang_textblob_loc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets = iran_tweets[\n",
    "    [\n",
    "        \"tweetid\",\n",
    "        \"user_screen_name\",\n",
    "        \"user_reported_location\",\n",
    "        \"user_profile_description\",\n",
    "        \"follower_count\",\n",
    "        \"following_count\",\n",
    "        \"account_creation_date\",\n",
    "        \"tweet_time\",\n",
    "        \"tweet_text\",\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_tweets = iran_tweets.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_test = iran_tweets[:1000].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 COMBINING THE IRANIAN STATE-BACKED/AMERICAN REAL TWEETS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is the second set of real tweets set aside for part of the unseen test sets\n",
    "real_test2 = pd.read_csv('../data/real_test2.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1000, 9), (1000, 9))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iran_test.shape, real_test2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is same function I used in notebook 1.3 to clean up the tweet text\n",
    "def clean_tweet(text):\n",
    "    text = text.lower()\n",
    "    text = re.sub(r\"http\\S+\", \"\", text)\n",
    "    text = re.sub(\"\\W\", \" \", text)\n",
    "    text = re.sub(\"\\s+\", \" \", text)\n",
    "    text = text.strip(\" \")\n",
    "    text = text.strip(\"\\n\")\n",
    "    text = re.sub(\"[^\\w\\s]\", \"\", text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_test2['clean_tweet_text'] = real_test2['tweet_text'].map(lambda tweet: clean_tweet(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "iran_test['clean_tweet_text'] = iran_test['tweet_text'].map(lambda tweet: clean_tweet(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating a new col to classify the real Vs state-backed tweets\n",
    "iran_test['bot_or_not'] = 1\n",
    "real_test2['bot_or_not'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "iranian_unseen = pd.concat((iran_test, real_test2), axis=0, sort=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Outputting the csv file for use in notebook 3.1\n",
    "# NOTE: This CSV is included in the repo\n",
    "\n",
    "#iranian_set = iranian_unseen.to_csv('../data/iranian_unseen.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. CREATING VENEZUELIAN STATE-BACKED/AMERICAN REAL USERS UNSEEN TEST SET\n",
    "This is part of the Venezuela (Set 2) corpus of tweets released by Twitter in January 2019, of about 1 million tweets from 764 accounts. I won't do extensive work on this dataset, beyond getting them into the same shape as the other unseen tests and most importantly, filtering out the non-English tweets and retweets.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 PRE-PROCESSING THE VENEZUELIAN STATE-BACKED TWEETS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: This CSV file is not included in the repo due to its file size. Download it with the link above\n",
    "vz = pd.read_csv('../data/venezuelian_tweets.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz = vz.drop(\n",
    "    columns=[\n",
    "        \"userid\",\n",
    "        \"user_display_name\",\n",
    "        \"user_profile_url\",\n",
    "        \"tweet_client_name\",\n",
    "        \"in_reply_to_tweetid\",\n",
    "        \"in_reply_to_userid\",\n",
    "        \"quoted_tweet_tweetid\",\n",
    "        \"is_retweet\",\n",
    "        \"retweet_userid\",\n",
    "        \"retweet_tweetid\",\n",
    "        \"latitude\",\n",
    "        \"longitude\",\n",
    "        \"quote_count\",\n",
    "        \"reply_count\",\n",
    "        \"like_count\",\n",
    "        \"retweet_count\",\n",
    "        \"urls\",\n",
    "        \"user_mentions\",\n",
    "        \"poll_choices\",\n",
    "        \"hashtags\",\n",
    "        \"account_language\",\n",
    "        #\"tweet_language\"\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The goal here is to get 1,000 English tweets from this set, but we'll have to sample a larger\n",
    "#number due to the likelihood of large number of non_English tweets\n",
    "vz_tweets = vz.sample(n=250000, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_language_langdetect(text):\n",
    "    try:\n",
    "        return detect(text)\n",
    "    except:\n",
    "        return 'unk'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_language_textblob(text):\n",
    "    try:\n",
    "        return TextBlob(text).detect_language\n",
    "    except:\n",
    "        return 'unk'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets['lang_textblob'] = vz_tweets['tweet_text'].apply(detect_language_textblob)\n",
    "vz_tweets['lang_textblob_loc'] = vz_tweets['user_reported_location'].apply(detect_language_textblob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets['langdetect'] = vz_tweets['tweet_text'].apply(detect_language_langdetect)\n",
    "vz_tweets['langdetect_loc'] = vz_tweets['user_reported_location'].apply(detect_language_langdetect) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets = vz_tweets[(vz_tweets['langdetect'] == 'en') & (vz_tweets['langdetect_loc'] == 'en')].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "en    1521\n",
       "Name: langdetect, dtype: int64"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vz_tweets['langdetect'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets = vz_tweets[~vz_tweets['tweet_text'].str.startswith(\"RT @\")].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets = vz_tweets.drop(columns=['langdetect', 'langdetect_loc', 'lang_textblob', 'lang_textblob_loc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets = vz_tweets[\n",
    "    [\n",
    "        \"tweetid\",\n",
    "        \"user_screen_name\",\n",
    "        \"user_reported_location\",\n",
    "        \"user_profile_description\",\n",
    "        \"follower_count\",\n",
    "        \"following_count\",\n",
    "        \"account_creation_date\",\n",
    "        \"tweet_time\",\n",
    "        \"tweet_text\",\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_tweets = vz_tweets.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_test = vz_tweets[:1000].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 COMBINING THE VENEZUELIAN STATE-BACKED/AMERICAN REAL TWEETS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recalling the third test-set of real tweets created earlier for this purpose \n",
    "real_test3 = pd.read_csv('../data/real_test3.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1000, 9), (1000, 9))"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vz_test.shape, real_test3.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_test3['clean_tweet_text'] = real_test3['tweet_text'].map(lambda tweet: clean_tweet(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_test['clean_tweet_text'] = vz_test['tweet_text'].map(lambda tweet: clean_tweet(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating a new col to classify the real Vs state-backed tweets\n",
    "vz_test['bot_or_not'] = 1\n",
    "real_test3['bot_or_not'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "vz_unseen = pd.concat((vz_test, real_test3), axis=0, sort=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Outputting the csv file for use in notebook 3.1\n",
    "# NOTE: This CSV is included in the repo\n",
    "#vz_set = vz_unseen.to_csv('../data/vz_unseen.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}