3-2-simple-transf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{admonition} Information\n",
    "__Section__: Simple tweet preprocessing  \n",
    "__Goal__: Learn and understand the simple methods of text preprocessing and normalization using regular expressions.  \n",
    "__Time needed__: 30 min  \n",
    "__Prerequisites__: None\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple tweet preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On this page, we examine some basic ways of changing the text to increase its ability to be processed by a natural language processing algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "# Get example tweets\n",
    "tweet_1 = 'Hospitalizations from COVID-19 have increased nearly 90% and California officials say they could triple by Christmas. https://t.co/hrBnP04HnB'\n",
    "tweet_2 = 'Something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool Christmas tunes intercut with Christmas film samples and scratching @BBCSounds https://t.co/rHovIA3u5e'\n",
    "tweet_3 = 'This happened in Adelaide the other day. #koala #Adelaide https://t.co/vAQFkd5r7q'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove URLs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "URLs do not give any information when we try to analyze text from words, especially on Twitter as they are reduced to a code to take less space. One of the first reasonable thing to do is then to just remove them from the text.\n",
    "\n",
    "For that, we can simply remove all chains of characters starting with ``http``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{toggle} Advanced level\n",
    "To remove all chains matching a certain pattern in a string, we use regular expressions.  \n",
    "In Python, we will use the function [sub()](https://docs.python.org/3/library/re.html#re.sub) from the library ``re``, which allows us to use regular expressions. This function replaces each occurence of a specified chain by another specified chain. In our case, as we want to remove the URL, the replacement chain will be an empty string, i.e. ``''``.\n",
    "\n",
    "The regular expression we use here would be: ``https?:\\/\\/.+``. The part ``http`` is here because we expect a URL to start with those 4 characters, then we add a ``s?``, because some URL are 'https' but not all of them. Then, we match ``://`` exactly by adding an escape character for ``/``. We continue by adding ``[^ ]+``, meaning any character but a space, an unlimited amount of times.\n",
    "\n",
    "Change the string in ``tweet_1`` to see how the transformation changes.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hospitalizations from COVID-19 have increased nearly 90% and California officials say they could triple by Christmas. https://t.co/hrBnP04HnB\n",
      "Hospitalizations from COVID-19 have increased nearly 90% and California officials say they could triple by Christmas. \n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "print(tweet_1)\n",
    "transf = re.sub(r'https?://[^ ]+', '', tweet_1)\n",
    "print(transf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove usernames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same as for the URL, a username in a tweet won't give any valuable information because it won't be recognized as a word carrying meaning. We will then remove it.\n",
    "\n",
    "Specifically for Twitter, all usernames start with the character ``@``. To remove them, we only have to remove all chains of characters starting with ``@``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{toggle} Advanced level\n",
    "The regular expression here will be: ``@[^ ]+``, to match any string starting with ``@`` and ending with a space.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool Christmas tunes intercut with Christmas film samples and scratching @BBCSounds https://t.co/rHovIA3u5e\n",
      "Something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool Christmas tunes intercut with Christmas film samples and scratching  https://t.co/rHovIA3u5e\n"
     ]
    }
   ],
   "source": [
    "print(tweet_2)\n",
    "transf = re.sub(r'@[^ ]+', '', tweet_2)\n",
    "print(transf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hashtags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hashtags are hard to apprehend, but usually contain useful information about the context of a tweet and its content. The problem with hashtags is that the words are all after the other, without a space. This kind of word is hard to understand with a basic algorithm for word extraction. However, most of the time, hashtags consist on only one word, preceeded by the symbol ``#``. It can then be useful to keep the part following the ``#``. If the word is made of two or more words, it will stay as noise in the data.\n",
    "\n",
    "To deal with hashtags, we only remove the character ``#``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{toggle} Advanced level\n",
    "The regular expression is very simple in that case: ``#``.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This happened in Adelaide the other day. #koala #Adelaide https://t.co/vAQFkd5r7q\n",
      "This happened in Adelaide the other day. koala Adelaide https://t.co/vAQFkd5r7q\n"
     ]
    }
   ],
   "source": [
    "print(tweet_3)\n",
    "transf = re.sub(r'#', '', tweet_3)\n",
    "print(transf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Character normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As Twitter is used mostly informally, it is very common to find unregularly written words. One of them is a repetition of characters to accentuate a statement, for example: \"It starts todaaaaaaaaay\". The word \"todaaaaaaaaay\" won't be recognized by our algorithm, while the word \"today\" would and could convey important information.\n",
    "\n",
    "We can replace each character that is repeated more than 2 times in a row by its single value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{toggle} Advanced level\n",
    "Here, we use a regular expression to match a letter repeated more than 2 times: ``([A-Za-z])\\1{2,}``. This one is a bit more complicated than the previous ones. First, we use ``[A-Za-z]`` to match only letters. This group is in between parentheses so we can add ``\\1``, to match the __same__ character that was first matched, and not any letter. Finally, we add ``{2,}`` to specify that we need a repeatition of more than 2 characters.\n",
    "\n",
    "In the function ``re.sub()``, as the second parameter, we use ``r'\\1'`` to replace the identified group with the matched character.\n",
    "\n",
    "Change the value of ``string`` to see what happens with other examples.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "todaaaaaaaaay\n",
      "today\n"
     ]
    }
   ],
   "source": [
    "string = 'todaaaaaaaaay'\n",
    "print(string)\n",
    "print(re.sub(r'([A-Za-z])\\1{2,}', r'\\1', string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Punctuation, special characters and numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same way, punctuation and single characters do not add any information with the method we use to process the text, as the algorithm for sentiment analysis only detects words.  \n",
    "Same goes for numbers: they are not processed, understandably as they do not represent a sentiment. An exception could be for the number 0, as it can convey a negative sense. To make sure of that, we can keep the number 0 and translate into its textual form, \"zero\".\n",
    "\n",
    "We decide to detect all the single ``0``, transforming them into ``zero``, and keep only letters otherwise. This has for effect to get rid of all the special characters and digits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{toggle} Advanced level\n",
    "First, we decide to change all the zeros. For that, we select all zeros preceeding and following a space, in order to only keep real zeros. This is simply done with the regular expression `` 0 ``, replacing with ``zero``.\n",
    "\n",
    "Then, it is easier to remove all characters that are not letters or blank spaces:``[^A-Za-z ]``, ``^`` having the effect of removing all characters that are not specified.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool Christmas tunes intercut with Christmas film samples and scratching @BBCSounds https://t.co/rHovIA3u5e\n",
      "Something for the afternoon slump  journey home  after school  cooking dinner  a special  minute mix of cool Christmas tunes intercut with Christmas film samples and scratching BBCSounds httpstcorHovIAue\n"
     ]
    }
   ],
   "source": [
    "print(tweet_2)\n",
    "transf = re.sub(r' 0 ', 'zero', tweet_2)\n",
    "transf = re.sub(r'[^A-Za-z ]', '', tweet_2)\n",
    "print(transf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lower casing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be sure that all words are equally analyzed, we cannot keep the distinction between upper and lower case. To ensure of that, we transform all capital letters to their lower case equivalent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{toggle} Advanced level\n",
    "This is a very simple transformation in Python, as there is a built-in method to do that easily. No need for regular expressions, we just use ``string.lower()``.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool Christmas tunes intercut with Christmas film samples and scratching @BBCSounds https://t.co/rHovIA3u5e\n",
      "something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool christmas tunes intercut with christmas film samples and scratching @bbcsounds https://t.co/rhovia3u5e\n"
     ]
    }
   ],
   "source": [
    "print(tweet_2)\n",
    "transf = tweet_2.lower()\n",
    "print(transf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we went through the simple techniques for tweet preprocessing, we can go onto the next page to see some more advanced algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"959\"\n",
       "            height=\"309\"\n",
       "            src=\"https://blog.hoou.de/wp-admin/admin-ajax.php?action=h5p_embed&id=66\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1d81d7093c8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame(\"https://blog.hoou.de/wp-admin/admin-ajax.php?action=h5p_embed&id=66\", \"959\", \"309\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- Kumar, Harish, 2018, Classification of Short Text Using Various Preprocessing Techniques: An Empirical Evaluation, Recent Findings in Intelligent Computing Techniques, pp. 19-30.\n",
    "- Arpita et al., 2020, Data Cleaning of Raw Tweets for Sentiment Analysis, 2020 Indo – Taiwan 2nd International Conference on Computing, Analytics and Networks (Indo-Taiwan ICAN)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}