session_9_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "15817af3",
   "metadata": {},
   "source": [
    "# Session 9: Analyze data using statistical libraries, text analysis, and web scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5941540a",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This three-part session starts with data analysis using two stats libraries:\n",
    "\n",
    "- [statsmodels](https://www.statsmodels.org/dev/index.html) and \n",
    "- [linearmodels](https://bashtage.github.io/linearmodels/panel/introduction.html)\n",
    "\n",
    "The linearmodels library works well with panel data and data frames with multi indexes.\n",
    "\n",
    "The libraries allow you to easily formulate you regression models, and conveniently select the results from the regression output. You can select coefficients and *t*-stats, or *p*-values, add them to a data frame (i.e. a table), and exported that data frame to LaTeX, markdown, or cvs / Excel format.\n",
    "\n",
    "**The second part** is a brief intro into text analysis using Natural Language Toolkit: [NLTK](https://www.nltk.org/).\n",
    "\n",
    "**The last part** is about web-scraping, which features a script that I wrote to collect information from `ssrn.com` for reference management. It uses the [beautifulsoup](https://beautiful-soup-4.readthedocs.io/en/latest/#) library.\n",
    "\n",
    "Learning  objectives:\n",
    "\n",
    "- How to create tables with descriptives of your data.\n",
    "- Perform regression analyses with [statsmodels](https://www.statsmodels.org/dev/index.html) and [linearmodels](https://bashtage.github.io/linearmodels/panel/introduction.html) and extract parameters for presentations in tabular form.\n",
    "- Analyze text from financial reports using [nltk](https://www.nltk.org/) and pandas.\n",
    "- Scrape a website using [beautifulsoup](https://beautiful-soup-4.readthedocs.io/en/latest/#).\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2cc2c53",
   "metadata": {},
   "source": [
    "---\n",
    "### Part 2: Text analysis ###\n",
    "\n",
    "For this part we need to install two libraries: \n",
    "\n",
    "- For extracting text from pdf files we need [**PyMuPDF**](https://pymupdf.readthedocs.io/en/latest/intro.html),  lightweight PDF, XPS, and E-book viewer, renderer, and toolkit. **See this** [**link**](https://pymupdf.readthedocs.io/en/latest/installation.html) for installation.\n",
    "\n",
    "- For text analysis we need the Natural Language Toolkit: [NLTK](https://www.nltk.org/). Install this using `pip install nltk`.\n",
    "\n",
    "- You may want to learn a bit about regular expressions, or regex. **Regex** allows you to select text using specific patterns. Rexeg is a powerful tool, but it has a steep learning curve. Nevertheless, most of your regex queries can be solved by either using Google or by way of [this excellent website](https://regex101.com/) for testing regex.\n",
    "\n",
    "\n",
    "We will analyze a page from an annual report of [BNP-Paribas](https://group.bnpparibas/uploads/file/bnp2019_urd_en_20_03_13.pdf), following the approach taken in [here](https://towardsdatascience.com/getting-started-with-text-analysis-in-python-ca13590eb4f7https://towardsdatascience.com/getting-started-with-text-analysis-in-python-ca13590eb4f7).\n",
    "\n",
    "**Note** NLTK may ask you to install additional modules if you run this script for the first time. Please follow the instructions shown in the warnings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68d22274",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the familiar preamble\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# For this session\n",
    "import re\n",
    "import fitz  # to connect to the PyMuPDF library\n",
    "import nltk\n",
    "import requests\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "792d71ff",
   "metadata": {},
   "source": [
    "**Get that very big annual report**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38edca2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get('https://group.bnpparibas/uploads/file/bnp2019_urd_en_20_03_13.pdf')\n",
    "# Write content in pdf file\n",
    "pdf = open(\"bnp2019_urd_en_20_03_13_web.pdf\", 'wb')\n",
    "pdf.write(response.content)\n",
    "pdf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9957cd7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = fitz.open('bnp2019_urd_en_20_03_13_web.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "588f1e4f",
   "metadata": {},
   "source": [
    "**Find a page with some text**, e.g. the page on with `ethics of the highest standard`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55494148",
   "metadata": {},
   "outputs": [],
   "source": [
    "for page in doc:\n",
    "    s = page.get_text(\"text\")\n",
    "    found = s.find(\"ethics of the highest standard\")\n",
    "    if found != -1:\n",
    "        print(page.number)\n",
    "        #print(page.get_text('text'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37ca0709",
   "metadata": {},
   "source": [
    "**Alas, let's try to ignore the case**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb9056b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Using regex instead, finding the page with the most text\n",
    "def find_long_page(search_term):\n",
    "    words_page_dict = {}\n",
    "    for page in doc:\n",
    "        #print(page)\n",
    "        s = page.get_text(\"text\")\n",
    "        matches = re.findall(search_term,s, re.IGNORECASE)\n",
    "        if matches:\n",
    "            p = page.get_text('words')\n",
    "            df = pd.DataFrame(p)\n",
    "            # print(page.number, len(df))\n",
    "            words_page_dict[page.number] = len(df)\n",
    "    return(max(words_page_dict, key=words_page_dict.get))\n",
    "\n",
    "find_long_page(r'ethics of the highest standard')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d97c571",
   "metadata": {},
   "source": [
    "**Show the text after removing line breaks `\\n`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bf4d1b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = doc.load_page(find_long_page(r'ethics of the highest standard')).get_text()\n",
    "text =  re.sub(\"\\n\", \" \",text)  # get rid of line breaks\n",
    "text[0:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6067d33d",
   "metadata": {},
   "source": [
    "---\n",
    "Let's **tokenize** the text and add all separate items to a data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65b69c6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "tokenized_text = sent_tokenize(text)\n",
    "\n",
    "df = pd.DataFrame(tokenized_text, columns=['full_text'])\n",
    "df['len_text'] = df['full_text'].apply(len)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f12fec0",
   "metadata": {},
   "source": [
    "**Okay, now we are going to prepare the text for sentiment analysis, which requires cleaning, removing stopwords, stemming or lemmatizing words.**\n",
    "\n",
    "However, it makes sense to apply this approach to the separate items of the data frame. \n",
    "\n",
    "So, I will first show these steps applied to one cell of the data frame. Then I show a function for that does all these steps in one go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff07bb33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the longest string in df:\n",
    "text_max = df['len_text'].idxmax()\n",
    "text_max = df.iloc[text_max]['full_text']\n",
    "print(text_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae64aa1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleaning the text\n",
    "def clean_text(text):\n",
    "    text =  re.sub(\"[^a-zA-Z]+\", \" \", text ) # get rid everything except words\n",
    "    return text\n",
    "\n",
    "letters_only_text = clean_text(text_max)\n",
    "print(letters_only_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf391e3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split text in a list of words\n",
    "words = letters_only_text.lower().split()\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5880ed0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get rid of stop words\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "stop_words\n",
    "\n",
    "def words_cleanup(words, stop_words):\n",
    "    cleaned_words = []\n",
    "    # remove stopwords\n",
    "    for word in words:\n",
    "        if word not in stop_words:\n",
    "            cleaned_words.append(word)\n",
    "    return cleaned_words\n",
    "\n",
    "cleaned_words = words_cleanup(words, stop_words)\n",
    "print(cleaned_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7751772a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer, PorterStemmer, SnowballStemmer\n",
    "lemmatizer = PorterStemmer() #plug in here any other stemmer or lemmatiser you want to try out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf743b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stemm or lemmatise words\n",
    "def words_lemmatizer(words):\n",
    "    stemmed_words = []\n",
    "    for word in words:\n",
    "        word = lemmatizer.stem(word)   #dont forget to change stem to lemmatize if you are using a lemmatizer\n",
    "        stemmed_words.append(word)   \n",
    "    return stemmed_words\n",
    "    \n",
    "stemmed_words = words_lemmatizer(cleaned_words)\n",
    "print(stemmed_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84f261cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting list back to string\n",
    "pre_senti_text = \" \".join(stemmed_words)\n",
    "pre_senti_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c39d9124",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "#### Analyze sentiment ####"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11fe0d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.sentiment import SentimentIntensityAnalyzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8005eb9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "sia = SentimentIntensityAnalyzer()\n",
    "polarity_score =  sia.polarity_scores(pre_senti_text)\n",
    "polarity_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e64afd8",
   "metadata": {},
   "source": [
    "Following this [page](https://realpython.com/python-nltk-sentiment-analysis/), I use only the positivity of the compound score to assess sentiment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1b35276",
   "metadata": {},
   "outputs": [],
   "source": [
    "polarity_score['compound']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f57ed382",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "#### Bringing it all together ####\n",
    "\n",
    "The function below determines the polarity score of a text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f4883a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.sentiment import SentimentIntensityAnalyzer\n",
    "\n",
    "def preprocess(raw_text):\n",
    "    letters_only_text =  re.sub(\"[^a-zA-Z]+\", \" \", raw_text ) # get rid everything except words, numbers, spaces\n",
    "    #text =  re.sub(\"\\s\\.\\s+\", \". \",text ) # get rid of space dot space\n",
    "    #text =  re.sub(\"\\s+\", \" \",text )  # get rid of multiple spaces\n",
    "    # convert to lower case and split into words -> convert string into list ( 'hello world' -> ['hello', 'world'])\n",
    "    words = letters_only_text.lower().split()\n",
    "    \n",
    "    cleaned_words = []\n",
    "    # remove stopwords\n",
    "    for word in words:\n",
    "        if word not in stop_words:\n",
    "            cleaned_words.append(word)\n",
    "\n",
    "    lemmatizer = PorterStemmer() # plug in here any other stemmer or lemmatiser you want to try out\n",
    "\n",
    "    # stemm or lemmatise words\n",
    "    stemmed_words = []\n",
    "    for word in cleaned_words:\n",
    "        word = lemmatizer.stem(word)   #don't forget to change stem to lemmatize if you are using a lemmatizer\n",
    "        stemmed_words.append(word)\n",
    "\n",
    "    # converting list back to string\n",
    "    pre_senti_text = \" \".join(stemmed_words)\n",
    "    sia = SentimentIntensityAnalyzer()\n",
    "    polarity_score =  sia.polarity_scores(pre_senti_text)\n",
    "    return polarity_score['compound']\n",
    "\n",
    "preprocess(text_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c55cc571",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the original text of the first entry  \n",
    "df.loc[0]['full_text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e1ecb6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the function to that text \n",
    "preprocess(df.loc[0]['full_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ddd92a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the function to all entries in the data frame\n",
    "df['compound_senti_score'] = df['full_text'].apply(preprocess)\n",
    "df.sort_values('compound_senti_score')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}