examples/python/training/italian/Train-Lemmatizer-Italian.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![JohnSnowLabs](https://sparknlp.org/assets/images/logo.png)\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/training/italian/Train-Lemmatizer-Italian.ipynb)\n",
    "\n",
    "# Training Lemmatizer Model in Italian language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only run this cell when you are using Spark NLP on Google Colab\n",
    "!wget http://setup.johnsnowlabs.com/colab.sh -O - | bash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A brief explaination about `Lemmatizer` annotator in Spark NLP:\n",
    "\n",
    "Retrieves lemmas out of words with the objective of returning a base dictionary word<br><br>\n",
    "**Type:** Token<br>\n",
    "**Requires:** None<br>\n",
    "**Input:** abduct -> abducted abducting abduct abducts<br><br>\n",
    "**Functions:**<br>\n",
    "* setDictionary(path, keyDelimiter, valueDelimiter, readAs, options): Path and options to lemma dictionary, in lemma vs possible words format. readAs can be LINE_BY_LINE or SPARK_DATASET. options contain option passed to spark reader if readAs is SPARK_DATASET.\n",
    "<br>\n",
    "\n",
    "**Example:**\n",
    "```Python\n",
    "lemmatizer = Lemmatizer() \\\n",
    "  .setInputCols([\"token\"]) \\\n",
    "  .setOutputCol(\"lemma\") \\\n",
    "  .setDictionary(\"./lemmas001.txt\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import required libraries including `SQL` and `ML` from Spark and some annotators from Spark NLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Spark ML and SQL\n",
    "from pyspark.ml import Pipeline, PipelineModel\n",
    "from pyspark.sql.functions import array_contains\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.types import StructType, StructField, IntegerType, StringType\n",
    "#Spark NLP\n",
    "import sparknlp\n",
    "from sparknlp.annotator import *\n",
    "from sparknlp.common import RegexRule\n",
    "from sparknlp.base import DocumentAssembler, Finisher"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's create a Spark Session for our app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spark NLP version:  4.3.1\n",
      "Apache Spark version:  3.3.0\n"
     ]
    }
   ],
   "source": [
    "spark = sparknlp.start()\n",
    "\n",
    "print(\"Spark NLP version: \", sparknlp.version())\n",
    "print(\"Apache Spark version: \", spark.version)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-02-20 18:14:45--  https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/it/lemma/dxc.technology/lemma_italian.txt\n",
      "Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.217.87.190, 52.217.68.38, 52.216.37.120, ...\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.87.190|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 6900964 (6,6M) [text/plain]\n",
      "Saving to: ‘/tmp/lemma_italian.txt’\n",
      "\n",
      "lemma_italian.txt   100%[===================>]   6,58M  6,52MB/s    in 1,0s    \n",
      "\n",
      "2023-02-20 18:14:47 (6,52 MB/s) - ‘/tmp/lemma_italian.txt’ saved [6900964/6900964]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget -N https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/it/lemma/dxc.technology/lemma_italian.txt -P /tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we are going to create a Spark NLP Pipeline by using Spark ML Pipeline natively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_assembler = DocumentAssembler() \\\n",
    "    .setInputCol(\"text\")\n",
    "\n",
    "sentence_detector = SentenceDetector() \\\n",
    "    .setInputCols([\"document\"]) \\\n",
    "    .setOutputCol(\"sentence\")\n",
    "\n",
    "tokenizer = Tokenizer() \\\n",
    "    .setInputCols([\"sentence\"]) \\\n",
    "    .setOutputCol(\"token\")\n",
    "\n",
    "normalizer = Normalizer() \\\n",
    "    .setInputCols([\"token\"]) \\\n",
    "    .setOutputCol(\"normal\")\n",
    "\n",
    "lemmatizer = Lemmatizer() \\\n",
    "    .setInputCols([\"normal\"]) \\\n",
    "    .setOutputCol(\"lemma\") \\\n",
    "    .setDictionary(\n",
    "          path = \"/tmp/lemma_italian.txt\",\n",
    "          read_as = \"TEXT\",\n",
    "          key_delimiter = \"\\\\s+\",\n",
    "          value_delimiter = \"->\"\n",
    "        )\n",
    "pipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, normalizer, lemmatizer])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how good our model does when it comes to prediction. We are going to create a DataFrame with Italian text for testing purposes and use `transform()` to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|result                                                                                                                                                                                                                                                 |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|[Finchè, non, avevo, la, linea, ADSL, di, fastweb, potevo, entrare, nel, router, e, configurare, quelle, pochissime, cose, configurabili, (, es, ., nome, dei, device, ),, da, ieri, che, ho, avuto, la, linea, niente, è, più, configurabile, ., ., .]|\n",
      "|[L'uomo, è, insoddisfatto, del, prodotto, .]                                                                                                                                                                                                           |\n",
      "|[La, coppia, contenta, si, abbraccia, sulla, spiaggia, .]                                                                                                                                                                                              |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|result                                                                                                                                                                                                                              |\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|[Finchè, non, avevo, la, linea, ADSL, di, fastweb, potevo, entrare, nel, router, e, configurare, quelle, pochissime, cose, configurabili, es, nome, dei, device, da, ieri, che, ho, avuto, la, linea, niente, è, più, configurabile]|\n",
      "|[Luomo, è, insoddisfatto, del, prodotto]                                                                                                                                                                                            |\n",
      "|[La, coppia, contenta, si, abbraccia, sulla, spiaggia]                                                                                                                                                                              |\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|result                                                                                                                                                                                                                              |\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|[Finchè, non, avevo, la, linea, ADSL, di, fastweb, potevo, entrare, nel, router, e, configurare, quelle, pochissime, cose, configurabili, es, nome, dei, device, da, ieri, che, ho, avuto, la, linea, niente, è, più, configurabile]|\n",
      "|[Luomo, è, insoddisfatto, del, prodotto]                                                                                                                                                                                            |\n",
      "|[La, coppia, contenta, si, abbraccia, sulla, spiaggia]                                                                                                                                                                              |\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n",
      "root\n",
      " |-- text: string (nullable = true)\n",
      " |-- document: array (nullable = true)\n",
      " |    |-- element: struct (containsNull = true)\n",
      " |    |    |-- annotatorType: string (nullable = true)\n",
      " |    |    |-- begin: integer (nullable = false)\n",
      " |    |    |-- end: integer (nullable = false)\n",
      " |    |    |-- result: string (nullable = true)\n",
      " |    |    |-- metadata: map (nullable = true)\n",
      " |    |    |    |-- key: string\n",
      " |    |    |    |-- value: string (valueContainsNull = true)\n",
      " |    |    |-- embeddings: array (nullable = true)\n",
      " |    |    |    |-- element: float (containsNull = false)\n",
      " |-- sentence: array (nullable = true)\n",
      " |    |-- element: struct (containsNull = true)\n",
      " |    |    |-- annotatorType: string (nullable = true)\n",
      " |    |    |-- begin: integer (nullable = false)\n",
      " |    |    |-- end: integer (nullable = false)\n",
      " |    |    |-- result: string (nullable = true)\n",
      " |    |    |-- metadata: map (nullable = true)\n",
      " |    |    |    |-- key: string\n",
      " |    |    |    |-- value: string (valueContainsNull = true)\n",
      " |    |    |-- embeddings: array (nullable = true)\n",
      " |    |    |    |-- element: float (containsNull = false)\n",
      " |-- token: array (nullable = true)\n",
      " |    |-- element: struct (containsNull = true)\n",
      " |    |    |-- annotatorType: string (nullable = true)\n",
      " |    |    |-- begin: integer (nullable = false)\n",
      " |    |    |-- end: integer (nullable = false)\n",
      " |    |    |-- result: string (nullable = true)\n",
      " |    |    |-- metadata: map (nullable = true)\n",
      " |    |    |    |-- key: string\n",
      " |    |    |    |-- value: string (valueContainsNull = true)\n",
      " |    |    |-- embeddings: array (nullable = true)\n",
      " |    |    |    |-- element: float (containsNull = false)\n",
      " |-- normal: array (nullable = true)\n",
      " |    |-- element: struct (containsNull = true)\n",
      " |    |    |-- annotatorType: string (nullable = true)\n",
      " |    |    |-- begin: integer (nullable = false)\n",
      " |    |    |-- end: integer (nullable = false)\n",
      " |    |    |-- result: string (nullable = true)\n",
      " |    |    |-- metadata: map (nullable = true)\n",
      " |    |    |    |-- key: string\n",
      " |    |    |    |-- value: string (valueContainsNull = true)\n",
      " |    |    |-- embeddings: array (nullable = true)\n",
      " |    |    |    |-- element: float (containsNull = false)\n",
      " |-- lemma: array (nullable = true)\n",
      " |    |-- element: struct (containsNull = true)\n",
      " |    |    |-- annotatorType: string (nullable = true)\n",
      " |    |    |-- begin: integer (nullable = false)\n",
      " |    |    |-- end: integer (nullable = false)\n",
      " |    |    |-- result: string (nullable = true)\n",
      " |    |    |-- metadata: map (nullable = true)\n",
      " |    |    |    |-- key: string\n",
      " |    |    |    |-- value: string (valueContainsNull = true)\n",
      " |    |    |-- embeddings: array (nullable = true)\n",
      " |    |    |    |-- element: float (containsNull = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's create a DataFrame with Italian text for testing our Spark NLP Pipeline\n",
    "dfTest = spark.createDataFrame([\"Finchè non avevo la linea ADSL di fastweb potevo entrare nel router e configurare quelle pochissime cose configurabili (es. nome dei device), da ieri che ho avuto la linea niente è più configurabile...\",\n",
    "    \"L'uomo è insoddisfatto del prodotto.\",\n",
    "    \"La coppia contenta si abbraccia sulla spiaggia.\"], StringType()).toDF(\"text\")\n",
    "\n",
    "# Of course you can select multiple columns at the same time however, this way we see each annotator without truncating their results\n",
    "pipeline.fit(dfTest).transform(dfTest).select(\"token.result\").show(truncate=False)\n",
    "pipeline.fit(dfTest).transform(dfTest).select(\"normal.result\").show(truncate=False)\n",
    "pipeline.fit(dfTest).transform(dfTest).select(\"lemma.result\").show(truncate=False)\n",
    "\n",
    "# Print the schema of the Pipeline\n",
    "pipeline.fit(dfTest).transform(dfTest).printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credits \n",
    "We would like to thank `DXC.Technology` for sharing their Italian datasets and models with Spark NLP community. The datasets are used to train `Lemmatizer` and `SentimentDetector` Models."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Train-Lemmatizer-Italian.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}