v1/python-sdk/tutorials/automl-with-azureml/automl-nlp-multiclass/automl-nlp-text-classification-multiclass.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/experimental/automl-nlp-multiclass/automl-nlp-text-classification-multiclass.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "_**Multiclass Text Classification Using AutoML NLP**_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Train)\n",
    "1. [Inference](#Inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "This notebook demonstrates classification with text data using AutoML NLP.\n",
    "\n",
    "AutoML highlights here include using end to end deep learning for NLP tasks like multiclass text classification.\n",
    "\n",
    "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
    "\n",
    "Notebook synopsis:\n",
    "\n",
    "1. Creating an Experiment in an existing Workspace\n",
    "2. Configuration and remote run of AutoML for a multiclass text dataset from scikit-learn, [20 Newsgroups dataset](https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html)\n",
    "3. Evaluating the trained model on a test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480011130
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.core.dataset import Dataset\n",
    "from azureml.core.compute import AmlCompute\n",
    "from azureml.core.compute import ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "from azureml.core.script_run_config import ScriptRunConfig\n",
    "from azureml.core.run import Run\n",
    "from azureml.data.datapath import DataPath\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480011411
    }
   },
   "outputs": [],
   "source": [
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480014618
    }
   },
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# Choose an experiment name.\n",
    "experiment_name = \"automl-nlp-text-classification-multiclass\"\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace Name\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "output[\"Experiment Name\"] = experiment.name\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up a compute cluster\n",
    "This section uses a user-provided compute cluster (named \"gpu-compute\" in this example). If a cluster with this name does not exist in the user's workspace, the below code will create a new cluster. You can choose the parameters of the cluster as mentioned in the comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480040093
    }
   },
   "outputs": [],
   "source": [
    "num_nodes = 1\n",
    "\n",
    "# Choose a name for your cluster.\n",
    "amlcompute_cluster_name = \"gpu-compute\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print(\"Found existing cluster, use it.\")\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_NC6\", max_nodes=num_nodes  # use GPU Nodes\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data\n",
    "For this notebook we will use 20 Newsgroups data from scikit-learn. We filter the data to contain four classes and take a sample as training data. Please note that for accuracy improvement, more data is needed. For this notebook we provide a small-data example so that you can use this template to use with your larger sized data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480040358
    }
   },
   "outputs": [],
   "source": [
    "target_column_name = \"y\"\n",
    "feature_column_name = \"X\"\n",
    "\n",
    "\n",
    "def get_20newsgroups_data():\n",
    "    \"\"\"Fetches 20 Newsgroups data from scikit-learn\n",
    "    Returns them in form of pandas dataframes\n",
    "    \"\"\"\n",
    "    remove = (\"headers\", \"footers\", \"quotes\")\n",
    "    categories = [\n",
    "        \"rec.sport.baseball\",\n",
    "        \"rec.sport.hockey\",\n",
    "        \"comp.graphics\",\n",
    "        \"sci.space\",\n",
    "    ]\n",
    "\n",
    "    data = fetch_20newsgroups(\n",
    "        subset=\"train\",\n",
    "        categories=categories,\n",
    "        shuffle=True,\n",
    "        random_state=42,\n",
    "        remove=remove,\n",
    "    )\n",
    "    data = pd.DataFrame(\n",
    "        {feature_column_name: data.data, target_column_name: data.target}\n",
    "    )\n",
    "\n",
    "    data_train = data.loc[:200]\n",
    "    data_val = data.loc[200:300]\n",
    "    data_test = data.loc[300:400]\n",
    "\n",
    "    data_train = remove_blanks_20news(data_train)\n",
    "    data_val = remove_blanks_20news(data_val)\n",
    "    data_test = remove_blanks_20news(data_test)\n",
    "\n",
    "    return data_train, data_val, data_test\n",
    "\n",
    "\n",
    "def remove_blanks_20news(data):\n",
    "    data = data.copy()\n",
    "    data[feature_column_name] = (\n",
    "        data[feature_column_name]\n",
    "        .replace(r\"\\n\", \" \", regex=True)\n",
    "        .apply(lambda x: x.strip())\n",
    "    )\n",
    "    data = data[data[feature_column_name] != \"\"]\n",
    "\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetch data and upload to datastore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480043071
    }
   },
   "outputs": [],
   "source": [
    "data_train, data_val, data_test = get_20newsgroups_data()\n",
    "\n",
    "data_dir = \"data\"  # Local directory to store data\n",
    "blobstore_datadir = data_dir  # Blob store directory to store data in\n",
    "if not os.path.isdir(data_dir):\n",
    "    os.mkdir(data_dir)\n",
    "\n",
    "train_data_fname = data_dir + \"/train_data.csv\"\n",
    "val_data_fname = data_dir + \"/val_data.csv\"\n",
    "test_data_fname = data_dir + \"/test_data.csv\"\n",
    "\n",
    "data_train.to_csv(train_data_fname, index=False)\n",
    "data_val.to_csv(val_data_fname, index=False)\n",
    "data_test.to_csv(test_data_fname, index=False)\n",
    "\n",
    "datastore = ws.get_default_datastore()\n",
    "target = DataPath(\n",
    "    datastore=datastore, path_on_datastore=blobstore_datadir, name=\"news_group_data\"\n",
    ")\n",
    "Dataset.File.upload_directory(\n",
    "    src_dir=data_dir, target=target, overwrite=True, show_progress=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480048150
    }
   },
   "outputs": [],
   "source": [
    "train_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=[(datastore, blobstore_datadir + \"/train_data.csv\")]\n",
    ")\n",
    "val_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=[(datastore, blobstore_datadir + \"/val_data.csv\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480049084
    }
   },
   "outputs": [],
   "source": [
    "train_dataset = train_dataset.register(\n",
    "    workspace=ws,\n",
    "    name=\"20newsgroups_data_train\",\n",
    "    description=\"20newsgroups_data_train\",\n",
    "    create_new_version=True,\n",
    ")\n",
    "\n",
    "val_dataset = val_dataset.register(\n",
    "    workspace=ws,\n",
    "    name=\"20newsgroups_data_val\",\n",
    "    description=\"20newsgroups_data_val\",\n",
    "    create_new_version=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train\n",
    "\n",
    "## Submit AutoML run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the run with the prepared compute resource and datasets. This should only take a few minutes.\n",
    "\n",
    "Here we do not set `primary_metric` parameter as we only train one model and we do not need to rank trained models. The run will use default primary metrics, `accuracy`. But it is only for reporting purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480049334
    }
   },
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"text-classification\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=train_dataset,\n",
    "    validation_data=val_dataset,\n",
    "    label_column_name=target_column_name,\n",
    "    **automl_settings\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit AutoML Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480498717
    }
   },
   "outputs": [],
   "source": [
    "automl_run = experiment.submit(automl_config, show_output=False)\n",
    "automl_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Metrics\n",
    "\n",
    "These metrics logged with the training run are computed with the trained model on validation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_metrics = automl_run.get_metrics()\n",
    "pd.DataFrame(\n",
    "    {\"metric_name\": validation_metrics.keys(), \"value\": validation_metrics.values()}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get the best run id and the best model with `get_output` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run = automl_run.get_best_child()\n",
    "best_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference\n",
    "\n",
    "Now you can use the trained model to do inference on unseen data. We use a `ScriptRun` to do this, with script that we provide. The following blocks will register the test dataset, download the inference script and trigger the inference run. Our inference run do not directly log the metrics. So we need to download the results and calculate the metrics offline\n",
    "\n",
    "## Submit Inference Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480502155
    }
   },
   "outputs": [],
   "source": [
    "test_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=[(datastore, blobstore_datadir + \"/test_data.csv\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480502601
    }
   },
   "outputs": [],
   "source": [
    "test_dataset = test_dataset.register(\n",
    "    workspace=ws,\n",
    "    name=\"20newsgroups_data_test\",\n",
    "    description=\"20newsgroups_data_test\",\n",
    "    create_new_version=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480502834
    }
   },
   "outputs": [],
   "source": [
    "training_run_id = best_run.id\n",
    "training_run = Run(experiment, training_run_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480503046
    }
   },
   "outputs": [],
   "source": [
    "# Inference script run arguments\n",
    "arguments = [\n",
    "    \"--run_id\",\n",
    "    training_run_id,\n",
    "    \"--experiment_name\",\n",
    "    experiment.name,\n",
    "    \"--input_dataset_id\",\n",
    "    test_dataset.as_named_input(\"test_data\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480504777
    }
   },
   "outputs": [],
   "source": [
    "scoring_args = arguments\n",
    "with tempfile.TemporaryDirectory() as tmpdir:\n",
    "    # Download required files from training run into temp folder.\n",
    "    entry_script_name = \"score_script.py\"\n",
    "    output_path = os.path.join(tmpdir, entry_script_name)\n",
    "    training_run.download_file(\n",
    "        \"outputs/\" + entry_script_name, os.path.join(tmpdir, entry_script_name)\n",
    "    )\n",
    "\n",
    "    script_run_config = ScriptRunConfig(\n",
    "        source_directory=tmpdir,\n",
    "        script=entry_script_name,\n",
    "        compute_target=compute_target,\n",
    "        environment=training_run.get_environment(),\n",
    "        arguments=scoring_args,\n",
    "    )\n",
    "    scoring_run = experiment.submit(script_run_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480505025
    }
   },
   "outputs": [],
   "source": [
    "scoring_run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480563276
    }
   },
   "outputs": [],
   "source": [
    "_ = scoring_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480563581
    }
   },
   "outputs": [],
   "source": [
    "output_prediction_file = \"./preds_multiclass.csv\"\n",
    "scoring_run.download_file(\n",
    "    \"outputs/predictions.csv\", output_file_path=output_prediction_file\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480563775
    }
   },
   "outputs": [],
   "source": [
    "test_set_predictions_df = pd.read_csv(\"preds_multiclass.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480564083
    }
   },
   "outputs": [],
   "source": [
    "test_data_df = test_dataset.to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Offline Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633480564357
    }
   },
   "outputs": [],
   "source": [
    "print(\n",
    "    classification_report(\n",
    "        test_data_df[target_column_name], test_set_predictions_df[target_column_name]\n",
    "    )\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "anshirga"
   }
  ],
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "None"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "None"
  ],
  "friendly_name": "DNN Text Featurization",
  "index_order": 2,
  "interpreter": {
   "hash": "cc0892e042a269bcf4aec58f0c86eb5e2be478ff7be4e5f6b2680e2af1718f2e"
  },
  "kernel_info": {
   "name": "python3-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "tags": [
   "None"
  ],
  "task": "Text featurization using DNNs for classification"
 },
 "nbformat": 4,
 "nbformat_minor": 0
}