10.Text_Mining_1z3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warsztaty Python w Data Science\n",
    "\n",
    "---\n",
    "## Text Mining - część 1 z 3  \n",
    "\n",
    "- ### Tokenizacja\n",
    "- ### Statystyki\n",
    "- ### Prosta Lemmatyzacja\n",
    "- ### TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://github.com/MichalKorzycki/WarsztatPythonDataScience"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://drive.google.com/drive/folders/1HR8VCledCwD7BRMO1AucUM3x7cYC-AVT?usp=sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Zadanie 1.\n",
    "Wyciągnąć z _*kilku*_ ogłoszeń ich tytuły i treści"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrapy\n",
    "import datetime\n",
    "import pandas as pd\n",
    "import scrapy.crawler as crawler\n",
    "from bs4 import BeautifulSoup\n",
    "from scrapy.exporters import CsvItemExporter\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "url_results = []\n",
    "desc_results = []\n",
    "title_results = []\n",
    "\n",
    "\n",
    "class GumtreeApartmentsSpider(scrapy.Spider):\n",
    "    name = 'gumtreeapartmentsspider'\n",
    "    start_urls = [ \n",
    "    \n",
    "        'https://www.gumtree.pl/s-mieszkania-i-domy-sprzedam-i-kupie/mazowieckie/page-'+str(i)+'/v1c9073l3200001p'+str(i)  for i in range(2, 11)\n",
    "        ]\n",
    "    start_urls.append(\n",
    "        'https://www.gumtree.pl/s-mieszkania-i-domy-sprzedam-i-kupie/mazowieckie/v1c9073l3200001p1'\n",
    "    )\n",
    "    found_apartments = []\n",
    "   \n",
    "    custom_settings = {\n",
    "        'DOWNLOAD_DELAY': '2.0',\n",
    "        'ROBOTSTXT_OBEY': True,\n",
    "        'AUTOTHROTTLE_ENABLED': True,\n",
    "        'USER_AGENT': 'My Bot (email@myemail.com)'\n",
    "    }\n",
    "\n",
    "    top_url = 'https://www.gumtree.pl'\n",
    "    def parse(self, response):\n",
    "        self.logger.info('Got successful response from {}'.format(response.url))\n",
    "        soup = BeautifulSoup(response.body, 'lxml')\n",
    "        titles = [flat.next_element for flat in soup.find_all('a', class_ = \"href-link tile-title-text\")] \n",
    "        links = ['https://www.gumtree.pl' + link.get('href')\n",
    "                for link in soup.find_all('a', class_ =\"href-link tile-title-text\")]\n",
    "            \n",
    "        for item_url in links:\n",
    "            yield scrapy.Request(item_url, self.parse_item)\n",
    "        \n",
    "    def parse_item(self, response): \n",
    "        self.logger.info('Got successful response from {}'.format(response.url))\n",
    "        soup = BeautifulSoup(response.body, 'lxml')\n",
    "        title = soup.find('span', class_ =\"myAdTitle\")\n",
    "        description = soup.find('div', class_ =\"description\")\n",
    "        item = {\n",
    "            \"url\": response.url,\n",
    "            \"title\": title,\n",
    "            \"description\": description,\n",
    "        }\n",
    "\n",
    "        url_results.append(response.url)\n",
    "        desc_results.append(description)\n",
    "        title_results.append(title)\n",
    "        \n",
    "    def spider_closed(self, spider):\n",
    "        spider.logger.info('Spider closed: %s', spider.name)\n",
    "        \n",
    "        df = pd.Dataframe({\n",
    "            \"title\": title_results,\n",
    "            \"description\": desc_results,\n",
    "            \"url\": url_results\n",
    "        })\n",
    "        fname = f\"gumtree-{now}.csv\"\n",
    "        print(fname)\n",
    "        df.to_csv(fname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = CrawlerProcess()\n",
    "process.crawl(GumtreeApartmentsSpider)\n",
    "process.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\n",
    "            \"title\": title_results,\n",
    "            \"description\": desc_results,\n",
    "            \"url\": url_results\n",
    "        })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "now = datetime.datetime.now().strftime(\"%Y-%m-%d\")\n",
    "fname = f\"data\\gumtree-{now}.csv\"\n",
    "print(fname)\n",
    "df.to_csv(fname, sep=\"|\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Tokenizacja"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data\\gumtree-2021-03-09.csv', sep='|')\n",
    "data.set_index('Unnamed: 0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opis = data['description'][0]\n",
    "opis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def no_tags(s):\n",
    "    return re.sub(r'<[^<]+?>','',s)\n",
    "\n",
    "opis = no_tags(opis)\n",
    "opis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "tokenizer = re.compile(r'[^ąąćęńłóóśśżżź\\w]+')\n",
    "tokenized = tokenizer.split(opis)\n",
    "str(tokenized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized = [ x.lower() for x in tokenized ]\n",
    "str(tokenized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing(opis):\n",
    "    opis = no_tags(opis)\n",
    "    tokenized = tokenizer.split(opis)\n",
    "    l = list(tokenized)\n",
    "    l = [ x.lower() for x in l ]\n",
    "    return l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus=[]\n",
    "n=4\n",
    "for row in data.iterrows():\n",
    "    opis = row[1][2]\n",
    "    l = preprocessing(opis)\n",
    "    corpus.append(l)\n",
    "    n-=1\n",
    "    if n==0: break\n",
    "\n",
    "for opis in corpus:\n",
    "    print(opis)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "---\n",
    "# Statystyki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = []\n",
    "for row in data.iterrows():\n",
    "    opis = row[1][2]\n",
    "    if type(opis) == str:\n",
    "        l = preprocessing(opis)\n",
    "        corpus.append(l)\n",
    "\n",
    "    \n",
    "print(f\"Mamy tekstów: {len(corpus)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_words = []\n",
    "for t in corpus:\n",
    "    all_words += t\n",
    " \n",
    "print(f\"Mamy {len(all_words)} wyrazów\")\n",
    "all_words[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = {}\n",
    "\n",
    "for w in all_words:\n",
    "    counter[w] = counter.get(w,0)+1\n",
    "\n",
    "print(f\"Mamy {len(counter.keys())} RÓŻNYCH wyrazów\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counted_words= [ (word,cnt) for word,cnt in counter.items() ]\n",
    "counted_words[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from operator import itemgetter\n",
    "\n",
    "counted_words.sort(key=itemgetter(1), reverse=True)\n",
    "counted_words[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counted_words[-20:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counts = [ x[1] for x in counted_words ]\n",
    "len(counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(counts[:120])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counted_words[110:130]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_df = pd.DataFrame(counts[:120])\n",
    "count_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.dates as mdates\n",
    "from pandas.plotting import register_matplotlib_converters\n",
    "register_matplotlib_converters()\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "plt.figure(figsize=(24,12))\n",
    "plt.style.use(\"dark_background\")\n",
    "\n",
    "chart = sns.scatterplot(\n",
    "                     color='purple', \n",
    "                     data=count_df\n",
    "                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Prosta Lematyzacja"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Lematyzacja_ - sprowadzenie wyrazu do formy podstawowej tak aby różne formy tego wyrazu (*kot*, *kota*, *kotu*) były rozpatrywane jako ten sam wyraz (*kot*) \n",
    "\n",
    "https://sjp.pl/\n",
    "    \n",
    "Słownik SJP.PL\n",
    "Słownik języka polskiego, ortograficzny, wyrazów obcych i słownik do gier w jednym.\n",
    "\n",
    "Słownik jest rozwijany z myślą o zastosowaniu do sprawdzania pisowni w programach open-source, do gier słownych (np. literaki) i do użytku online jako kilka rodzajów słowników w jednym.\n",
    "\n",
    "Redakcją słownika zajmują się hobbyści.\n",
    "\n",
    "Słownik jest udostępniany na otwartych licencjach (różnych w zależności od wersji)."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "...\n",
    "mieszkaniec, mieszkańca, mieszkańcach, mieszkańcami, mieszkańce, mieszkańcem, mieszkańcom, mieszkańcowi, mieszkańców, mieszkańcu, mieszkańcy, mieszkańcze\n",
    "mieszkanie, mieszkania, mieszkaniach, mieszkaniami, mieszkaniem, mieszkaniom, mieszkaniu, mieszkań\n",
    "mieszkaniowy, mieszkaniowa, mieszkaniową, mieszkaniowe, mieszkaniowego, mieszkaniowej, mieszkaniowemu, mieszkaniowi, mieszkaniowo, mieszkaniowych, mieszkaniowym, \n",
    "mieszkaniowymi\n",
    "...\n",
    "(219.228 wyrazów, 4.295.200 form)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import sys\n",
    "import re\n",
    "\n",
    "f = gzip.open('data/odm.txt.gz', 'rt', encoding='utf-8')\n",
    "dictionary = {}\n",
    "\n",
    "for x in f:\n",
    "    t = x.strip().split(',')\n",
    "    tt = [ x.strip().lower() for x in t]\n",
    "    for w in tt[1:]: \n",
    "        dictionary[w]=tt[0]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lematize(w):\n",
    "    return dictionary.get(w,w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpusl = [[ lematize(x) for x in l ] for l in corpus]\n",
    "for opis in corpusl[:4]:\n",
    "    print(opis)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_words = []\n",
    "for t in corpusl:\n",
    "    all_words += t\n",
    " \n",
    "print(f\"Mamy {len(all_words)} wyrazów\")\n",
    "all_words[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = {}\n",
    "\n",
    "for w in all_words:\n",
    "    counter[w] = counter.get(w,0)+1\n",
    "\n",
    "print(f\"Mamy {len(counter.keys())} RÓŻNYCH wyrazów\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from operator import itemgetter\n",
    "counted_words= [ (word,cnt) for word,cnt in counter.items() ]\n",
    "counted_words.sort(key=itemgetter(1), reverse=True)\n",
    "counted_words[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.dates as mdates\n",
    "from pandas.plotting import register_matplotlib_converters\n",
    "register_matplotlib_converters()\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "counts = [ x[1] for x in counted_words ]\n",
    "count_df = pd.DataFrame(counts[:120])\n",
    "count_df\n",
    "\n",
    "plt.figure(figsize=(24,12))\n",
    "plt.style.use(\"dark_background\")\n",
    "\n",
    "chart = sns.scatterplot(\n",
    "        color='purple', \n",
    "        data=count_df\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Metryka TF-IDF\n",
    "ile razy występuję wyraz *i* w tekście *j*\n",
    "$${n}_{ij}$$ \n",
    " ### Term Frequency (TF)\n",
    " \n",
    " $${tf}_{ij} = \\frac{{n}_{ij}}{\\sum{k}{{n}_{ik}}}$$\n",
    " \n",
    " W tekście *j* sprawdzamy ile proporcjonalnie do całości występuje wyraz *i*\n",
    "### Inverted Document Frequency (IDF)\n",
    "\n",
    " $$idf_i = log \\frac{|D|}{ \\{ d: n_i \\in d \\}}$$\n",
    " \n",
    " licznik - liczba dokumentów\n",
    " \n",
    " mianownik - liczba dokumentów w którym wystapił wyraz *i*-ty "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ile razy występuję wyraz *i* w tekście *j*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$${n}_{ij}$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Term Frequency (TF)\n",
    " \n",
    " $${tf}_{ij} = \\frac{{n}_{ij}}{\\sum{k}{{n}_{ik}}}$$\n",
    " \n",
    " W tekście *j* sprawdzamy ile proporcjonalnie do całości występuje wyraz *i*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverted Document Frequency (IDF)\n",
    "\n",
    " $$idf_i = log \\frac{|D|}{ \\{ d: n_i \\in d \\}}$$\n",
    " \n",
    " licznik - liczba dokumentów\n",
    " \n",
    " mianownik - liczba dokumentów w którym wystapił wyraz *i*-ty "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Zadanie 1\n",
    "\n",
    "Podać wyrazy z korpusu o największym `TF-IDF` z pominięciem wyrazów które wystepują _**raz**_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}