notebook_examples/pyspark-data_flow_studio-spark_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "2896c30e",
   "metadata": {},
   "source": [
    "@notebook{pyspark-data_flow_studio-spark_nlp.ipynb,\n",
    "    title: Spark NLP within Oracle Cloud Infrastructure Data Flow Studio,\n",
    "    summary: Demonstrates how to use Spark NLP within a long lasting Oracle Cloud Infrastructure Data Flow cluster.,\n",
    "    developed_on: pyspark32_p38_cpu_v1,\n",
    "    keywords: pyspark, data flow,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52f8981d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5095f9f",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2022 Oracle, Inc. All rights reserved. Licensed under the [Universal Permissive License v 1.0](https://oss.oracle.com/licenses/upl).\n",
    "\n",
    "***\n",
    "\n",
    "# <font color=\"red\">Spark NLP within Oracle Cloud Infrastructure Data Flow Studio</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=\"teal\">Oracle Cloud Infrastructure Data Science Service.</font></p>\n",
    "\n",
    "---\n",
    "# Overview:\n",
    "\n",
    "This notebook demonstrates how to use [Spark NLP](https://www.johnsnowlabs.com/) within a long lasting [Oracle Cloud Infrastructure Data Flow](https://docs.oracle.com/en-us/iaas/data-flow/using/home.htm) cluster.\n",
    "\n",
    "Compatible conda pack: [PySpark 3.2 and Data Flow](https://docs.oracle.com/iaas/data-science/using/conda-pyspark-fam.htm) for CPU on Python 3.8\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href='#pre-requisites'>1. Pre-requisites</a>\n",
    "    - <a href='#policies'>1.2 Policies</a>\n",
    "    - <a href='#prerequisites_helpers'>1.3 Helpers</a>\n",
    "    - <a href='#prerequisites_authentication'>1.4 Authentication</a>\n",
    "    - <a href='#prerequisites_variables'>1.5 Variables</a>    \n",
    "- <a href='#spark_nlp'>2. John Snow Labs Spark NLP</a>\n",
    "    - <a href='#load_extension'>2.1. Load Data Flow Spark Magic Extension</a>\n",
    "    - <a href='#spark_nlp_pretrained_models'>2.2. Pre-trained Spark NLP models installation</a>\n",
    "    - <a href='#custom_conda_environment_publishing'>2.3. Publishing custom PySpark 3.2 and Data Flow conda environment</a>\n",
    "    - <a href='#new_session_published_conda'>2.4 Create a Data Flow Session with a new `spark.archives` configuration</a>\n",
    "    - <a href='#simple_anotation_example'>2.5. Simple annotation task example</a>    \n",
    "- <a href='#cleanup'>3. Clean Up</a> \n",
    "- <a href='#ref'>4. References</a>   \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b770afd1",
   "metadata": {},
   "source": [
    "<a id='pre-requisites'></a>\n",
    "# 1. Pre-requisites \n",
    "\n",
    "Data Flow Sessions are accessible through the following conda environment: \n",
    "\n",
    "* **PySpark 3.2 and Data Flow 1.0 (pyspark32_p38_cpu_v1)**\n",
    "\n",
    "You can customize `pypspark32_p38_cpu_v1`, publish it, and use it as a runtime environment for a Data Flow session cluster. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6056dd3b",
   "metadata": {},
   "source": [
    "<a id='policies'></a>\n",
    "## 1.2. Policies\n",
    "This section covers the creation of dynamic groups and policies needed to use the service.\n",
    "\n",
    "* [Data Flow Policies](https://docs.oracle.com/iaas/data-flow/using/policies.htm/)\n",
    "* [Getting Started with Data Flow](https://docs.oracle.com/iaas/data-flow/using/dfs_getting_started.htm)\n",
    "* [About Data Science Policies](https://docs.oracle.com/iaas/data-science/using/policies.htm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "980557bd",
   "metadata": {},
   "source": [
    "<a id=\"prerequisites_helpers\"></a>\n",
    "## 1.3 Helpers\n",
    "This section provides a helper method that will be used across the notebook to prepare arguments for the magic commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5ceaba9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def prepare_command(command: dict) -> str:\n",
    "    \"\"\"Converts dictionary command to the string formatted commands.\"\"\"\n",
    "    return f\"'{json.dumps(command)}'\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee52aba5",
   "metadata": {},
   "source": [
    "<a id=\"prerequisites_authentication\"></a>\n",
    "## 1.4. Authentication\n",
    "The [Oracle Accelerated Data Science SDK (ADS)](https://docs.oracle.com/iaas/tools/ads-sdk/latest/index.html) controls the authentication mechanism with the Data Flow Session Spark cluster.<br> \n",
    "To setup authentication use the ```ads.set_auth(\"resource_principal\")``` or ```ads.set_auth(\"api_key\")```. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ef3a6df",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ads\n",
    "\n",
    "ads.set_auth(\"resource_principal\")  # Supported values: resource_principal, api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60ffc42e",
   "metadata": {},
   "source": [
    "<a id=\"prerequisites_variables\"></a>\n",
    "## 1.5. Variables\n",
    "To run this notebook, you must provide some information about your tenancy configuration. To create and run a Data Flow session, you must specify a `<compartment_id>` and bucket `<logs_bucket_uri>` for storing logs. These resources must be in the same compartment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91d8a2fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "compartment_id = os.environ.get(\"NB_SESSION_COMPARTMENT_OCID\")\n",
    "# Assuming you already have a dataflow-logs bucket created in the region where the cluster is running.\n",
    "# Otherwise specify the bucket where the stdout/err logs will be stored.\n",
    "logs_bucket_uri = \"oci://<bucket_name>@<namespace>/<prefix>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf129ce",
   "metadata": {},
   "source": [
    "<a id=\"spark_nlp\"></a>\n",
    "# 2. John Snow Labs Spark NLP \n",
    "By default the **PySpark 3.2 and Data Flow** conda environment includes pre-installed [Matplotlib](https://matplotlib.org/) and [Spark NLP](https://www.johnsnowlabs.com/) libraries. The examples below demonstrate how to prepare custom conda environment, publish it to the Object Storage and use within a Data Flow Spark session."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d9d27e",
   "metadata": {},
   "source": [
    "<a id=\"load_extension\"></a>\n",
    "## 2.1. Load Data Flow Spark Magic Extension\n",
    "Data Flow Spark Magic is a JupyterLab extension, that you need to activate in your notebook using the `%load_ext dataflow.magics` magic command.\n",
    "It will automatically create a SparkContext (`sc`) and HiveContext (`sqlContext`) inside any SparkMagic cell. Use the `%help` command to get the list of supported commands. If you want to access the docstrings of any magic command and figure out what arguments to provide, simply add `?` at then end of the command, for instance `%create_session?`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c2a6841",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext dataflow.magics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb9d85fa",
   "metadata": {},
   "source": [
    "<a id=\"spark_nlp_pretrained_models\"></a>\n",
    "## 2.2. Pre-trained Spark NLP models installation\n",
    "If you need any pre-trained Spark NLP models, you have to download them and unzip them in the conda environment folder. Data Flow does not support egress to the public internet. You cannot dynamically download pre-trained models from the internet in Data Flow sessions.\n",
    "However you can download pre-trained models from the model hub as zip [archives](https://nlp.johnsnowlabs.com/models) and then unzip the models in the conda environment folder. Download the example model [Explain Document DL Pipeline for English](https://nlp.johnsnowlabs.com/2021/03/23/explain_document_dl_en.html) and upload it into your notebook session. Or you can execute this cell and download a public model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aeab1c61",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "cd ~\n",
    "curl https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/explain_document_dl_en_3.0.0_3.0_1616473268265.zip --output explain_document_dl_en_3.0.0_3.0_1616473268265.zip\n",
    "mkdir /home/datascience/conda/pyspark32_p38_cpu_v1/sparknlp-models\n",
    "unzip explain_document_dl_en_3.0.0_3.0_1616473268265.zip -d /home/datascience/conda/pyspark32_p38_cpu_v1/sparknlp-models/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5967e1a8",
   "metadata": {},
   "source": [
    "<a id=\"custom_conda_environment_publishing\"></a>\n",
    "## 2.3. Publishing custom PySpark 3.2 and Data Flow conda environment\n",
    "Use the `odsc conda publish` command to publish conda environment to the Object Storage bucket.<br>\n",
    "Follow the [Publishing a Conda Environment to an Object Storage Bucket in Your Tenancy](https://docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm#:~:text=You%20can%20publish%20a%20conda%20environment%20that%20you%20have%20installed,persist%20them%20across%20notebook%20sessions.) to get more details about how to publish custom conda environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4998c29a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "odsc conda publish -s pyspark32_p38_cpu_v1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "663d23db",
   "metadata": {},
   "source": [
    "<a id=\"new_session_published_conda\"></a>\n",
    "## 2.4. Create a Data Flow Session with a new `spark.archives` configuration\n",
    "\n",
    "Now the published conda environment can be used within a Data Flow session. The path to the published conda environment can be copied from the [Environment Explorer](https://docs.oracle.com/en-us/iaas/data-science/using/conda_viewing.htm). <br>\n",
    "\n",
    "Example path : `oci://<your-bucket>@<your-tenancy-namespace>/conda_environments/cpu/PySpark 3.2 and Data Flow/1.0/pyspark32_p38_cpu_v1#conda`\n",
    "\n",
    "To create a new Data Flow session use the `%create_session` magic command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc94999b",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_conda_environment_uri = \"oci://<your-bucket>@<your-tenancy-namespace>/conda_environments/cpu/PySpark 3.2 and Data Flow/1.0/pyspark32_p38_cpu_v1#conda\"\n",
    "\n",
    "command = prepare_command(\n",
    "    {\n",
    "        \"compartmentId\": compartment_id,\n",
    "        \"displayName\": \"TestDataFlowSessionWithCustomCondaEnvironment\",\n",
    "        \"language\": \"PYTHON\",\n",
    "        \"sparkVersion\": \"3.2.1\",\n",
    "        \"numExecutors\": 2,\n",
    "        \"driverShape\": \"VM.Standard.E4.Flex\",\n",
    "        \"executorShape\": \"VM.Standard.E4.Flex\",\n",
    "        \"driverShapeConfig\": {\"ocpus\": 2, \"memoryInGBs\": 32},\n",
    "        \"executorShapeConfig\": {\"ocpus\": 2, \"memoryInGBs\": 32},\n",
    "        \"logsBucketUri\": logs_bucket_uri,\n",
    "        \"type\": \"SESSION\",\n",
    "        \"configuration\": {\n",
    "            \"spark.archives\": custom_conda_environment_uri,\n",
    "            \"spark.jars.ivy\": \"/opt/spark/work-dir/conda/.ivy2\",\n",
    "            \"spark.jars.packages\": \"com.johnsnowlabs.nlp:spark-nlp_2.12:4.1.0\",\n",
    "        },\n",
    "    }\n",
    ")\n",
    "\n",
    "%create_session -l python -c $command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a20b975",
   "metadata": {},
   "source": [
    "<a id=\"simple_anotation_example\"></a>\n",
    "## 2.5. Simple annotation task example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6181c51a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%spark\n",
    " \n",
    "from sparknlp.base import *\n",
    "from sparknlp.annotator import *\n",
    "from sparknlp.pretrained import PretrainedPipeline\n",
    "import sparknlp\n",
    " \n",
    "# Start SparkSession with Spark NLP\n",
    "# start() functions has 3 parameters: gpu, m1, and memory\n",
    "# sparknlp.start(gpu=True) will start the session with GPU support\n",
    "# sparknlp.start(m1=True) will start the session with macOS M1 support\n",
    "# sparknlp.start(memory=\"16G\") to change the default driver memory in SparkSession\n",
    "spark = sparknlp.start()\n",
    " \n",
    "# Download a pre-trained pipeline\n",
    "pipeline = PretrainedPipeline('explain_document_dl', lang='en', disk_location=\"/opt/spark/work-dir/conda/sparknlp-models/\")\n",
    " \n",
    "# Your testing dataset\n",
    "text = \"\"\"\n",
    "Lawrence Joseph Ellison (born August 17, 1944) is an American business magnate and investor who is the co-founder,\n",
    "executive chairman, chief technology officer (CTO) and former chief executive officer (CEO) of the\n",
    "American computer technology company Oracle Corporation.[2] As of September 2022, he was listed by\n",
    "Bloomberg Billionaires Index as the ninth-wealthiest person in the world, with an estimated\n",
    "fortune of $93 billion.[3] Ellison is also known for his 98% ownership stake in Lanai,\n",
    "the sixth-largest island in the Hawaiian Archipelago.[4]\n",
    "\"\"\"\n",
    " \n",
    "# Annotate your testing dataset\n",
    "result = pipeline.annotate(text)\n",
    " \n",
    "# What's in the pipeline\n",
    "print(list(result.keys()))\n",
    " \n",
    "# Check the results\n",
    "print(result['entities'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d53829",
   "metadata": {},
   "source": [
    "**Expected result:**\n",
    "<div class=\"cell border-box-sizing text_cell rendered\">\n",
    "    <div class=\"prompt input_prompt\"></div>\n",
    "    <div class=\"inner_cell\">\n",
    "        <div class=\"text_cell_render border-box-sizing rendered_html\">\n",
    "            <div class=\"alert alert-block alert-info\" style=\"background: none; border: 1px solid; padding: 10px\">\n",
    "                <b><i class=\"fa fa-info-circle\" aria-hidden=\"true\"></i>&nbsp; Info</b><br>\n",
    "<div style=\"padding:10px 0px\">\n",
    "\n",
    "```python\n",
    "['entities', 'stem', 'checked', 'lemma', 'document', 'pos', 'token', 'ner', 'embeddings', 'sentence']\n",
    "['Lawrence Joseph Ellison', 'American', 'American', 'Oracle Corporation', 'Bloomberg Billionaires Index', 'Ellison', 'Lanai', 'Hawaiian Archipelago']\n",
    "```\n",
    "</div>\n",
    "            </div>\n",
    "        </div>\n",
    "    </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5622085d",
   "metadata": {},
   "source": [
    "<a id='cleanup'></a>\n",
    "# 3. Clean Up\n",
    "Use the `%stop_session` magic command to stop your active Data Flow session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad20c778",
   "metadata": {},
   "outputs": [],
   "source": [
    "%stop_session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ed445db",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "# 4. References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
    "- [Data Flow Policies](https://docs.oracle.com/iaas/data-flow/using/policies.htm/)\n",
    "- [Getting Started with Data Flow](https://docs.oracle.com/iaas/data-flow/using/dfs_getting_started.htm)\n",
    "- [About Data Science Policies](https://docs.oracle.com/iaas/data-science/using/policies.htm)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.1 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}