twitter_streaming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains the actual runnable Twitter stream! **Jump to section 3** to avoid reading all the setup.\n",
    "\n",
    "> *Section 1 contains library imports, neural network and tokenizer imports, and a handful of functions that are necessary for processing tweets and  making predictions. It is nothing that wasn't already touched on in other notebooks.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from twython import Twython, TwythonStreamer\n",
    "import time\n",
    "\n",
    "# Below is all just for neural network\n",
    "\n",
    "import string\n",
    "import re\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = stopwords.words('english')\n",
    "from nltk.stem import PorterStemmer \n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.layers import Input, Dense, LSTM, Embedding\n",
    "from keras.layers import Dropout, Activation, GlobalMaxPool1D\n",
    "from keras.models import Sequential\n",
    "from keras import initializers, regularizers, constraints, optimizers, layers\n",
    "from keras.preprocessing import text, sequence\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "import pickle\n",
    "\n",
    "import smtplib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Twython Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We instantiate a Twython object that stores the authentication data required to run the following `Conversation` class in conjunction with the Twitter API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set API version to 1.1. Version 2 doesn't have DM support yet.\n",
    "\n",
    "t = Twython(app_key=TWITTER_APP_KEY, \n",
    "            app_secret=TWITTER_APP_KEY_SECRET, \n",
    "            oauth_token=TWITTER_ACCESS_TOKEN, \n",
    "            oauth_token_secret=TWITTER_ACCESS_TOKEN_SECRET,\n",
    "            api_version='1.1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Conversation` Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A full explanation of this class is available in the notebook `twitter_chatbot.ipynb`. In brief, it allows for DM interactions with one Twitter user to be contained within a single instance Python class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Conversation:\n",
    "    \n",
    "    '''\n",
    "    Class designed to be a representation of a unique conversation with one\n",
    "    individual. Takes in a Twython instance, the unique user ID, and a brand\n",
    "    that the bot is currently representing.\n",
    "    \n",
    "    Contains functions for storing, cleaning, and replying to messages.\n",
    "    ---\n",
    "    t\n",
    "        Twython instance.\n",
    "    user_id\n",
    "        Twitter user ID. String or int.\n",
    "    brand\n",
    "        Defaults to Amazon, but can be any string.\n",
    "    help_count\n",
    "        Number of times Twitter user has sent \"HELP\" to chatbot. If count\n",
    "        is >= 3, bot sets no_contact to True.\n",
    "    no_contact\n",
    "        Boolean. When True, bot will not reply to user.\n",
    "    passed_to_human\n",
    "        Whether or not the interaction has been passed on to a human employee.\n",
    "        Defaults to False. When a user says \"YES\" they want to work with brand,\n",
    "        this switches to True.\n",
    "    '''\n",
    "    \n",
    "    def __init__(self, t, user_id, brand = 'Amazon'):\n",
    "        self.t = t\n",
    "        self.user_id = str(user_id)\n",
    "        self.brand = brand\n",
    "        self.help_count = 0\n",
    "        self.messages = None\n",
    "        self.no_contact = False\n",
    "        self.passed_to_human = False\n",
    "    \n",
    "    \n",
    "    \n",
    "    def store_messages(self):\n",
    "\n",
    "        '''\n",
    "        Gets and saves all messages sent from Twitter user self.user_id to\n",
    "        the bot. Output returns a list of dictionaries, where all keys and\n",
    "        values are strings.\n",
    "        '''\n",
    "        \n",
    "        messages = self.t.get_direct_messages()\n",
    "        \n",
    "        message_list = []\n",
    "\n",
    "        for i in range(len(messages['events'])):\n",
    "\n",
    "            message = messages['events'][i]\n",
    "            sender_id = message['message_create']['sender_id']\n",
    "\n",
    "            if sender_id == self.user_id:\n",
    "                message_dict = {}\n",
    "                message_dict['time'] = message['created_timestamp']\n",
    "                message_dict['user_id'] = message['message_create']['sender_id']\n",
    "                message_dict['text'] = message['message_create']['message_data']['text']\n",
    "                message_list.append(message_dict)\n",
    "\n",
    "        self.messages = message_list\n",
    "        \n",
    "        \n",
    "\n",
    "    def clean_message(self, message):\n",
    "\n",
    "        '''\n",
    "        Cleans out punctuation and capitals from a user message. Returns the\n",
    "        string as a list of strings so that replies can be parsed easily.\n",
    "        ---\n",
    "        message\n",
    "            Must be a string, ideally as seen in store_messages()[i]['text']\n",
    "        '''\n",
    "\n",
    "        allowed_replies = ['yes', 'no', 'help', 'stop']\n",
    "\n",
    "        for i in string.punctuation:\n",
    "            message = message.replace(i, '').lower()\n",
    "\n",
    "        new_message = []\n",
    "        for word in message.split():\n",
    "            if word in allowed_replies:\n",
    "                new_message.append(word)\n",
    "\n",
    "        return new_message\n",
    "\n",
    "    \n",
    "\n",
    "    def send_message(self, text):\n",
    "\n",
    "        '''\n",
    "        Sends a message to a specified user.\n",
    "        ---\n",
    "        text\n",
    "            String of what to send to user.\n",
    "        '''\n",
    "\n",
    "        if self.no_contact == True:\n",
    "            return \n",
    "        \n",
    "        self.t.send_direct_message(\n",
    "            event = {\"type\" : \"message_create\",\n",
    "                     \"message_create\" : {\"target\": {\"recipient_id\" : self.user_id},\n",
    "                                         \"message_data\" : {\"text\" : text}}}\n",
    "        )\n",
    "        \n",
    "\n",
    "\n",
    "    def reply(self, message_in):\n",
    "        \n",
    "        '''\n",
    "        This function allows the bot to talk back to users. It replies to four\n",
    "        possible, pre-specified inputs, and provides an alternative for when the\n",
    "        user input is not in the pre-specified list.\n",
    "        ---\n",
    "        message_in\n",
    "            Must be a string. This is what the bot replies to.\n",
    "        '''\n",
    "\n",
    "        message_in = self.clean_message(message_in)\n",
    "        allowed_replies = ['yes', 'no', 'help', 'stop']\n",
    "        \n",
    "        if self.no_contact == True:\n",
    "            return\n",
    "        \n",
    "        if (len(message_in) > 1) or (len(message_in) == 0):\n",
    "            if self.help_count >= 3:\n",
    "                self.send_message(\"Looks like you're havin a hard' time bud. I'll leave you alone :/\")\n",
    "                self.no_contact = True\n",
    "            else:\n",
    "                self.send_message(\"I'm sorry but I don't know what you want. Please reply only with YES, NO, HELP, or STOP.\")\n",
    "            self.help_count += 1\n",
    "            return\n",
    "\n",
    "        if message_in[0] == 'yes':\n",
    "            self.send_message(\"We're happy to hear it! A spokesperson will be in touch shortly :)\")\n",
    "            print(f'User at ID {self.user_id} is interested in collaborating! An employee has been emailed!')\n",
    "            self.passed_to_human = True\n",
    "            return\n",
    "\n",
    "        if message_in[0] == 'no':\n",
    "            self.send_message(\"We're sad you won't be joining us. Have a nice day!\")\n",
    "            return\n",
    "\n",
    "        if message_in[0] == 'help':\n",
    "            self.send_message(\"Reply YES to show interest in a brand deal, NO to decline, HELP to see this message, and STOP to be put on our no-contact list.\")\n",
    "            self.help_count += 1\n",
    "            return\n",
    "\n",
    "        if message_in[0] == 'stop':\n",
    "            self.no_contact = True\n",
    "            return\n",
    "    \n",
    "\n",
    "    \n",
    "    def greet(self):\n",
    "        \n",
    "        '''\n",
    "        Greets a user with the chatbot, introduces what brand the bot works for,\n",
    "        and offers four options for replying to the bot.\n",
    "        ---\n",
    "        All parameters pre-determined in __init__.\n",
    "        '''\n",
    "        \n",
    "        text = f'Hi there! Wanna collaborate with {self.brand}? Please respond with YES, NO, HELP, or STOP.'\n",
    "        self.send_message(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network And Sentiment Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we're loading in the neural network trained in `models.ipynb` as well as defining a few functions for cleaning tweets so that the model can predict on them. They're almost the same functions as used in `models.ipynb` for preprocessing data, just slightly changed to handle single data points instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model('models/stem_model_5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `stem_clean()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stem_clean(text):\n",
    "    '''\n",
    "    Takes in a piece of text and cleans and stems it. Removes all punctuation,\n",
    "    URL's, usernames, and hashtags. Sets everything to lowercase, tokenizes,\n",
    "    removes stopwords, stems and returns it as a list of tokens.\n",
    "    ---\n",
    "    text\n",
    "        String input to be cleaned.\n",
    "    '''\n",
    "    stop_words = stopwords.words(\"english\")\n",
    "    \n",
    "    text = re.sub('@\\S+', '', text)\n",
    "    text = re.sub('http\\S+', '', text)\n",
    "    text = re.sub('#\\S+', '', text)\n",
    "    for i in string.punctuation:\n",
    "        text = text.replace(i, '').lower()\n",
    "    \n",
    "    tokens = nltk.word_tokenize(text)\n",
    "    new_tokens = []\n",
    "    for token in tokens:\n",
    "        if token.lower() not in stop_words:\n",
    "            new_tokens.append(ps.stem(token))\n",
    "            \n",
    "    return new_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/tokenizer.pickle', 'rb') as handle:\n",
    "    tokenizer = pickle.load(handle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `tweet_to_sequence()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tweet_to_sequence(tweet):\n",
    "    '''\n",
    "    Takes in a tweet and returns a padded tokenized array.\n",
    "    ---\n",
    "    tweet\n",
    "        Tweet that has been cleaned by stem_clean(), ideally. Otherwise a list\n",
    "        of token strings works, but less optimized.\n",
    "    '''\n",
    "    tokenized_tweet = tokenizer.texts_to_sequences([tweet])\n",
    "    tweet_seq = sequence.pad_sequences(tokenized_tweet, maxlen = 45)\n",
    "# Needs o index because returns an array of arrays and we only want the one\n",
    "    return tweet_seq[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `tweet_sentiment()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tweet_sentiment(tweet, model):\n",
    "    '''\n",
    "    Takes in a string and predicts sentiment.\n",
    "    ---\n",
    "    tweet\n",
    "        Tweet that has been cleaned by stem_clean(), ideally. Otherwise a list\n",
    "        of token strings works, but less optimized.\n",
    "    model\n",
    "        Trained neural network.\n",
    "    '''\n",
    "    tweet_seq = tweet_to_sequence(tweet)\n",
    "\n",
    "    preds = model.predict(np.array([tweet_seq]))\n",
    "\n",
    "# Needs 0 index because built for many but only want the first\n",
    "    pred = list(preds[0])\n",
    "    max_pred = max(pred)\n",
    "\n",
    "    return pred.index(max_pred) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of prediction. Value `3` means positive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet_sentiment(\"Best belive imma take advantage of my amazon prime account\", model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Emailer` Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This case is explained in full in the notebook `emailer_class.ipynb`, but in short: It exists to take in a user ID and employee email, then send said ID to said employee so that they may reach out to finish the interaction with this user. This is imperative, as we want a human handling money-bargaining for collaboration and sponsored content, not a chatbot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Emailer:\n",
    "\n",
    "    '''\n",
    "    Class designed to email users. Mainly an SMTP wrapper.\n",
    "    ---\n",
    "    sender_email\n",
    "        User email that emails are sent from.\n",
    "    password\n",
    "        Password associated with email. May need to be set to app password on\n",
    "        your email service.\n",
    "    '''\n",
    "\n",
    "    def __init__(self, sender_email, password):\n",
    "        self.sender_email = str(sender_email)\n",
    "        self.password = str(password)\n",
    "        self.server = smtplib.SMTP('smtp.gmail.com', 587)\n",
    "    \n",
    "    def assign_chat(self, rec_email, user_id):\n",
    "        \n",
    "        '''\n",
    "        Sends an email containing a user ID to an employee.\n",
    "        ---\n",
    "        rec_email\n",
    "            The recipient email, i.e. the employee being emailed.\n",
    "        user_id\n",
    "            The unique ID of a Twitter user. Should be string.\n",
    "        '''\n",
    "        \n",
    "        self.server.starttls()\n",
    "        self.server.login(sender_email, password)\n",
    "        message = f\"Subject: New Contact!\\n\\nUser at ID {user_id} wants to collaborate!\"\n",
    "        self.server.sendmail(self.sender_email, rec_email, message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twython Streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The moment we've all been waiting for! **This `Stream` can be set to run on its own,** and will wait until a user with at least a thousand followers tweets positively about Amazon. Then, **it initializes a `Conversation`** with that user and greets them. It **waits** for two minutes and **checks for a response.** (Twitter's API doesn't update DM information quickly enough to be faster than this, at least not in the free version). If the user has sent a message, the `Conversation` bot will **reply,** and the **process repeats.**\n",
    "\n",
    "The bot's operation interrupts the `Stream` and no more searching occurs until the `Conversation` is over. In the future, I intend to learn more about **asynchronous execution** so that I can have a **master `Stream`** always running, with numerous `Conversation` bots assigned to different users as they appear."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Stream` Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This class is based on the default `MyStreamer` class provided in the [Twython documentation](https://twython.readthedocs.io/en/latest/usage/streaming_api.html). I've immensely changed the method `on_success()` such that it intitializes and has a `Conversation` with the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Stream(TwythonStreamer):\n",
    "\n",
    "    def on_success(self, data):\n",
    "# Check for enough clout\n",
    "        if data['user']['followers_count'] > 1000:\n",
    "# Check for positive sentiment\n",
    "            if tweet_sentiment(data['text']) == 3:\n",
    "# Instantiate Emailer\n",
    "                emailer = Emailer(sender_email = \"user@company.com\",\n",
    "                                  password = \"password\")\n",
    "# Instantiate Conversation\n",
    "                c = Conversation(user_id = data['id'], t = t)\n",
    "# Greet user\n",
    "                c.greet()\n",
    "# Converse with user until they opt out or collaborate\n",
    "                while (c.no_contact == False) and (c.passed_to_human == False):\n",
    "                    time.sleep(120)\n",
    "                    c.store_messages()\n",
    "                    if c.messages != None:\n",
    "                        newest_message = c.messages[0]['text']\n",
    "                        c.reply(newest_message)\n",
    "# Send an email to the employee tasked with chat\n",
    "                if c.passed_to_human == True:\n",
    "                    emailer.assign_chat(rec_email = \"employee@company.com\",\n",
    "                                        user_id = c.user_id)\n",
    "                    \n",
    "\n",
    "    def on_error(self, status_code, data):\n",
    "        print(status_code)\n",
    "        self.disconnect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stream Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "stream = Stream(TWITTER_APP_KEY, TWITTER_APP_KEY_SECRET,\n",
    "                TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "stream.statuses.filter(language = 'en',\n",
    "                       track = '@amazon',\n",
    "                       tweet_mode = 'extended')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, the cell above outputs a user ID whenever someone agrees to work with the brand."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}