examples/python/reader/SparkNLP_TXT_Reader_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0o5UQ-Gy2Xvr"
   },
   "source": [
    "![JohnSnowLabs](https://sparknlp.org/assets/images/logo.png)\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/reader/SparkNLP_TXT_Reader_Demo.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c0efed73-75e9-41f1-9a2e-a2d0953b3a76",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "id": "tzcU5p2gdak9"
   },
   "source": [
    "# Introducing TXT reader in SparkNLP\n",
    "This notebook showcases the newly added  `sparknlp.read().txt()` method in Spark NLP that parses txt file content from both local files and real-time URLs into a Spark DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "356de93e-af38-4156-823b-6371d7fd825c",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "id": "RFOFhaEedalB"
   },
   "source": [
    "## Setup and Initialization\n",
    "Let's keep in mind a few things before we start 😊\n",
    "\n",
    "Support for reading html files was introduced in Spark NLP 5.6.0. Please make sure you have upgraded to the latest Spark NLP release.\n",
    "\n",
    "- Let's install and setup Spark NLP in Google Colab\n",
    "- This part is pretty easy via our simple script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "xrWTskQJ2Xv5"
   },
   "outputs": [],
   "source": [
    "! wget -q http://setup.johnsnowlabs.com/colab.sh -O - | bash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9B98jlOn2Xv8"
   },
   "source": [
    "For local files example we will download a TXT file from Spark NLP Github repo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bb622e88-2ef9-49c4-8cfb-e49209ad206a",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ya8qZe00dalC",
    "outputId": "144186be-781d-451b-894e-d9c590a93c6a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘txt-files’: File exists\n",
      "--2025-03-07 00:33:21--  https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp/feature/SPARKNLP-1113-Adding-support-to-enhance-read-TXT-files/src/test/resources/reader/txt/simple-text.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 300 [text/plain]\n",
      "Saving to: ‘txt-files/simple-text.txt’\n",
      "\n",
      "simple-text.txt     100%[===================>]     300  --.-KB/s    in 0s      \n",
      "\n",
      "2025-03-07 00:33:21 (4.67 MB/s) - ‘txt-files/simple-text.txt’ saved [300/300]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!mkdir txt-files\n",
    "!wget https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp/feature/SPARKNLP-1113-Adding-support-to-enhance-read-TXT-files/src/test/resources/reader/txt/simple-text.txt -P txt-files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "13d72e9f-04b4-4547-bc4e-35b3878a93c2",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "id": "EoFI66NAdalE"
   },
   "source": [
    "## Parsing text from Local Files\n",
    "Use the `txt()` method to parse text file content from local directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "df54ed9b-682b-4b99-891a-84c23bc5cbd0",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bAkMjJ1vdalE",
    "outputId": "74f0e218-6378-4df4-9b12-3ee6e33020e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning::Spark Session already created, some configs may not take.\n",
      "+--------------------+--------------------+\n",
      "|                path|                 txt|\n",
      "+--------------------+--------------------+\n",
      "|file:/content/txt...|[{Title, BIG DATA...|\n",
      "+--------------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import sparknlp\n",
    "\n",
    "txt_df = sparknlp.read().txt(\"./txt-files\")\n",
    "txt_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "9f5c787d-2eab-4546-8001-e34f00124670",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4iky1gvEz7Pt",
    "outputId": "ead23526-18be-4bb9-e952-38ef3d483cb0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|txt                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|[{Title, BIG DATA ANALYTICS, {paragraph -> 0}}, {NarrativeText, Apache Spark is a fast and general-purpose cluster computing system.\\nIt provides high-level APIs in Java, Scala, Python, and R., {paragraph -> 0}}, {Title, MACHINE LEARNING, {paragraph -> 1}}, {NarrativeText, Spark's MLlib provides scalable machine learning algorithms.\\nIt includes tools for classification, regression, clustering, and more., {paragraph -> 1}}]|\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "txt_df.select(\"txt\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "brto-6NX2wLT"
   },
   "source": [
    "You can also use DFS file systems like:\n",
    "- Databricks: `dbfs://`\n",
    "- HDFS: `hdfs://`\n",
    "- Microsoft Fabric OneLake: `abfss://`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CYnoVMVD211Z"
   },
   "source": [
    "### Configuration Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rJhyeem_3Gqh"
   },
   "source": [
    "- `titleLengthSize`: You can customize the font size used to identify titles that should be treated as titles. By default, the font size is set to 50. However, if your text files require a different configuration, you can adjust this parameter accordingly. The example below demonstrates how to modify and work with this setting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nLUtWTk-3jcT",
    "outputId": "60d10ba0-cf91-4706-efb4-4e640d7e6bb0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning::Spark Session already created, some configs may not take.\n",
      "+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|path                                   |txt                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|file:/content/txt-files/simple-text.txt|[{NarrativeText, BIG DATA ANALYTICS, {paragraph -> 0}}, {NarrativeText, Apache Spark is a fast and general-purpose cluster computing system.\\nIt provides high-level APIs in Java, Scala, Python, and R., {paragraph -> 0}}, {NarrativeText, MACHINE LEARNING, {paragraph -> 1}}, {NarrativeText, Spark's MLlib provides scalable machine learning algorithms.\\nIt includes tools for classification, regression, clustering, and more., {paragraph -> 1}}]|\n",
      "+---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "params = {\"titleLengthSize\": \"5\"}\n",
    "txt_df = sparknlp.read(params).txt(\"./txt-files\")\n",
    "txt_df.show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d444S-MK239M"
   },
   "source": [
    "- `storeContent`: By default, this is set to `false`. When enabled, the output will include the raw content of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "optYF_SS22TW",
    "outputId": "e21f8dab-ef69-432b-aa3e-fb0afc075bbb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning::Spark Session already created, some configs may not take.\n",
      "+--------------------+--------------------+--------------------+\n",
      "|                path|                 txt|             content|\n",
      "+--------------------+--------------------+--------------------+\n",
      "|file:/content/txt...|[{Title, BIG DATA...|BIG DATA ANALYTIC...|\n",
      "+--------------------+--------------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "params = {\"storeContent\": \"true\"}\n",
    "txt_df = sparknlp.read(params).txt(\"./txt-files\")\n",
    "txt_df.show()"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "computePreferences": null,
   "dashboards": [],
   "environmentMetadata": null,
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "SparkNLP_TXT_Reader_Demo",
   "widgets": {}
  },
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}