sdk/python/using-mltable/delimited-files-example/delimited-files-example.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 📄 Create a Table from delimited files\n",
    "\n",
    "In this example notebook you will create a Table using the famous Titanic dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📦 Install dependencies\n",
    "\n",
    "Ensure you have the latest MLTable library and dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r ../mltable-requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🐍 Create an MLTable using the Python SDK\n",
    "\n",
    "Here you build your data loading steps using the `mltable` Python SDK. The `show()` method allows you to see the effect of the data loading transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "from mltable import MLTableHeaders, MLTableFileEncoding, DataType\n",
    "\n",
    "# create paths to the data files\n",
    "paths = [{\"file\": \"wasbs://data@azuremlexampledata.blob.core.windows.net/titanic.csv\"}]\n",
    "\n",
    "# create an MLTable from the data files\n",
    "tbl = mltable.from_delimited_files(\n",
    "    paths=paths,\n",
    "    delimiter=\",\",\n",
    "    header=MLTableHeaders.all_files_same_headers,\n",
    "    infer_column_types=True,\n",
    "    include_path_column=False,\n",
    "    encoding=MLTableFileEncoding.utf8,\n",
    ")\n",
    "\n",
    "# filter out rows undefined ages\n",
    "tbl = tbl.filter(\"col('Age') > 0\")\n",
    "\n",
    "# drop PassengerId\n",
    "tbl = tbl.drop_columns([\"PassengerId\"])\n",
    "\n",
    "# ensure survived column is treated as boolean\n",
    "data_types = {\n",
    "    \"Survived\": DataType.to_bool(\n",
    "        true_values=[\"True\", \"true\", \"1\"], false_values=[\"False\", \"false\", \"0\"]\n",
    "    )\n",
    "}\n",
    "tbl = tbl.convert_column_types(data_types)\n",
    "\n",
    "# show the first 5 records\n",
    "tbl.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐼 Load into a Pandas data frame\n",
    "\n",
    "You can load your Azure ML Table into Pandas using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = tbl.to_pandas_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💾 Save data loading steps \n",
    "Next, you'll save all your data loading steps into an `MLTable` file. This allows you to *reproduce* your Pandas data frame at a later point in time without having to redefine the data loading steps in your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the data loading steps in an MLTable file\n",
    "tbl.save(\"./titanic\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 🔍 View the saved file\n",
    "\n",
    "In the next code cell, we show you the `MLTable` file so you can understand how the data loading steps are serialized into a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./titanic/MLTable\", \"r\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ♻️ Reproduce data loading steps\n",
    "\n",
    "Now that the data loading steps have been serialized into a file, you can reproduce them at any point in time using the `load()` method. This means you do not need to redefine your data loading steps in code and makes it easier to share with others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "\n",
    "# load the previously saved MLTable file\n",
    "tbl = mltable.load(\"./titanic/\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🤝 Create a data asset to aid sharing and reproducibility\n",
    "\n",
    "You'll now create a data asset, which will automatically upload the `MLTable` to cloud storage (the default AzureML datastore) so that others can use it easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# set the version number of the data asset to the current UTC time\n",
    "VERSION = time.strftime(\"%Y.%m.%d.%H%M%S\", time.gmtime())\n",
    "\n",
    "# connect to the AzureML workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")\n",
    "\n",
    "my_data = Data(\n",
    "    path=\"./titanic\",\n",
    "    type=AssetTypes.MLTABLE,\n",
    "    description=\"The titanic dataset.\",\n",
    "    name=\"titanic-cloud-example\",\n",
    "    version=VERSION,\n",
    ")\n",
    "\n",
    "ml_client.data.create_or_update(my_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Read the data asset in an interactive session\n",
    "\n",
    "Now you have your MLTable stored in the cloud, you and Team members can access it using a friendly name in an interactive session (for example, a notebook)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# connect to the AzureML workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")\n",
    "\n",
    "# get the latest version of the data asset\n",
    "# Note: The version was set in the previous code cell.\n",
    "data_asset = ml_client.data.get(name=\"titanic-cloud-example\", version=VERSION)\n",
    "\n",
    "# create a table\n",
    "tbl = mltable.load(f\"azureml:/{data_asset.id}\")\n",
    "\n",
    "# load into pandas\n",
    "df = tbl.to_pandas_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Read the data asset in a job\n",
    "\n",
    "You can also access your Table in a job, using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient, command, Input\n",
    "from azure.ai.ml.entities import Environment\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# connect to the AzureML workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")\n",
    "\n",
    "# get the latest version of the data asset\n",
    "# Note: the VERSION was set in a previous cell.\n",
    "data_asset = ml_client.data.get(name=\"titanic-cloud-example\", version=VERSION)\n",
    "\n",
    "job = command(\n",
    "    command=\"python train.py --input ${{inputs.titanic}}\",\n",
    "    inputs={\"titanic\": Input(type=\"mltable\", path=data_asset.id)},\n",
    "    compute=\"cpu-cluster\",\n",
    "    environment=Environment(\n",
    "        image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04\",\n",
    "        conda_file=\"./job-env/conda_dependencies.yml\",\n",
    "    ),\n",
    "    code=\"./src\",\n",
    ")\n",
    "\n",
    "ml_client.jobs.create_or_update(job)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}